Robotic dataset distributed pre-computation coordination method and related products

Abstract

The application provides a robot dataset distributed pre-computation coordination method and related products. The method comprises: obtaining a pre-computation task of a robot task trajectory data unit, determining a task key, a task claim object key and a bypass list object key; querying a bypass list, when no bypass list with a complete state of complete is queried, performing a conditional operation on the task claim object key, enabling a work node with a successful operation to obtain a processing qualification and claim a scheduling generation; performing pre-computation by the work node, writing a result into an immutable result object and generating a to-be-submitted bypass list; verifying consistency of the claim scheduling generation and a current scheduling generation, and when the consistency is consistent, conditionally submitting the bypass list; and after successful submission, determining the bypass list and a referenced result as a formal release bypass list and a formal release result. Thus, invalid work node coverage results can be avoided, and coordination efficiency and release consistency of distributed pre-computation can be improved.

Description

Technical Field

[0001] This application relates to the field of distributed computing technology, and in particular to a distributed pre-computation coordination method for robot datasets and related products. Background Technology

[0002] Robot datasets typically consist of multiple robot task trajectory data units. To improve the efficiency of model training, evaluation, or data analysis, computations can be performed on the robot task trajectory data units in advance to obtain video latent variables, data statistics, visual features, or other derived data.

[0003] As data volumes increase, pre-computation tasks typically need to be executed in parallel by multiple worker nodes, with the results and task processing information stored in object storage. Object storage usually uses objects as the unit of data access, and it differs from traditional shared file systems in terms of file locking, controlled appending, and collaborative updates among multiple objects. Therefore, the task coordination methods based on file locks or centralized file updates in traditional shared file systems are difficult to directly apply to object storage environments.

[0004] In one processing method, multiple worker nodes jointly update a centralized index file or progress object to record completed pre-computation tasks, the storage location of pre-computation results, and task processing status. However, when multiple worker nodes concurrently perform read, modify, and write operations, different worker nodes may generate updated results based on previously read content, causing the task status or result reference information written by one worker node to be overwritten by another worker node.

[0005] In distributed precomputation, the same precomputation task may be executed simultaneously or sequentially by multiple worker nodes due to duplicate scheduling, execution timeouts, network anomalies, or worker node failures. After a worker node that executed the precomputation task first recovers, it may continue to submit its generated precomputation results, while subsequent worker nodes may have already generated new precomputation results. This can lead to issues such as precomputation results overwriting each other, expired results being submitted, or the same precomputation task being published repeatedly. Furthermore, precomputation results typically have a large data volume, while task completion status information has a smaller data volume. The write duration and failure point of the two may differ, and the reader may access precomputation results that have not yet been fully written based on the task completion status. Summary of the Invention

[0006] Embodiments of this application propose a distributed pre-computation coordination method for robot datasets and related products.

[0007] Firstly, this application provides a distributed pre-computation coordination method for robot datasets. The method includes: Acquire pre-calculated tasks for robot task trajectory data units; Based on the pre-computation task, a task key, a task claiming object key, and a bypass list object key are determined, wherein the task key is used to identify the pre-computation task, the task claiming object key is used to determine the task claiming object that records the current scheduling generation, and the bypass list object key is used to determine the bypass list that records the reference information and completion status of the pre-computation results. Query the bypass list corresponding to the bypass list object key; If the bypass list is not found, or if the bypass list is found but the completion status of the bypass list is not indicated as complete, a conditional operation is performed on the task claiming object key, so that the work node that successfully performs the conditional operation obtains the qualification to process the pre-computed task and obtains the claiming scheduling generation. The pre-computation task is executed by the work node that has obtained the processing qualification, and the pre-computation result of the pre-computation task is obtained; The pre-calculated results are written to object storage to form one or more immutable result objects, and a list of pending bypasses that reference the one or more immutable result objects and have a completion status of complete is generated. Perform a consistency check between the claimed scheduling generation and the current scheduling generation recorded by the task claiming object; When the consistency check passes, a conditional commit is performed on the list of bypasses to be submitted; When the conditions are successfully submitted, the successfully submitted bypass list is used as the official release bypass list of the pre-computation task, and the pre-computation results referenced by the official release bypass list are determined as the official release results of the pre-computation task.

[0008] In some implementations, determining the task key, task claiming object key, and bypass list object key based on the pre-computed task includes: Obtain the dataset identifier, robot task trajectory data unit identifier, optional time segment identifier or target time range, and configuration information corresponding to the pre-calculation task; The task key is determined based on the dataset identifier, the robot task trajectory data unit identifier, the optional time segment identifier or the target time range, and the configuration information. The task claiming object key and the bypass list object key are determined according to the task key and the preset mapping relationship, respectively. The preset mapping relationship is used to characterize the deterministic correspondence between the task key of the same pre-computation task and the corresponding task claiming object key and bypass list object key.

[0009] In some implementations, determining the task key based on the dataset identifier, the robot task trajectory data unit identifier, the optional time segment identifier or the target time range, and the configuration information includes: The configuration information is deterministically serialized to obtain a configuration byte sequence; Perform a digest operation on the configuration byte sequence to obtain the configuration fingerprint; The task key is determined based on the dataset identifier, the robot task trajectory data unit identifier, the optional time segment identifier or the target time range, and the configuration fingerprint.

[0010] In some implementations, when the task claiming object corresponding to the task claiming object key does not exist in the object storage, the conditional operation performed on the task claiming object key includes: If there is no task claim object identified by the task claim object key in the object storage, a conditional creation operation is performed based on the task claim object key to create the task claim object. The created task claim object records the worker node identifier, initial scheduling generation, claim time and lease expiration time of the worker node that successfully created the task claim object. The claim scheduling generation is the initial scheduling generation.

[0011] In some implementations, when a task claiming object corresponding to the task claiming object key already exists in the object storage, the conditional operation performed on the task claiming object key includes: Read the task claiming object and its object metadata, obtain the current scheduling generation and lease expiration time from the task claiming object, and obtain the entity tag or object version identifier returned by the object storage from the object metadata, wherein the entity tag or the object version identifier is used to identify the storage version of the task claiming object at the time of this read; The current time is compared with the lease expiration time, and if the current time is equal to or later than the lease expiration time, it is determined that the lease corresponding to the task claiming object has expired; Determine a target scheduling generation higher than the current scheduling generation based on the current scheduling generation; If the current entity tag or object version identifier of the task claiming object is consistent with the entity tag or object version identifier obtained during this reading, a condition update is performed on the task claiming object. When the condition update is successful, the current scheduling generation recorded by the task claiming object is updated to the target scheduling generation, the work node identifier recorded by the task claiming object is updated to the work node identifier of the work node that reclaims the pre-computation task, and the lease expiration time recorded by the task claiming object is updated to the new lease expiration time, wherein the claimed scheduling generation is the target scheduling generation.

[0012] In some implementations, after querying the bypass list corresponding to the bypass list object key, the method further includes: When a bypass list with a completion status of "complete" is found, the corresponding one or more immutable result objects are accessed based on the object key of one or more immutable result objects recorded in the bypass list. Availability verification is performed on the one or more immutable result objects, wherein the availability verification includes determining that all one or more immutable result objects exist, and determining that one or more of the actual byte count, content digest, output shape, data type, and original data corresponding range of each of the one or more immutable result objects are consistent with the corresponding information recorded in the bypass list; When the availability check passes, the pre-calculation result referenced by the bypass list is determined as the pre-calculation result of the pre-calculation task, and the conditional operation is no longer performed on the task claiming object key.

[0013] In some implementations, writing the pre-computed results into object storage to form one or more immutable result objects includes: Based on the task key and the claimed scheduling generation, determine the object key of each of the one or more immutable result objects; Using the object key of each of the one or more immutable result objects as a storage identifier, the pre-computed result is written into the object storage to form the one or more immutable result objects, wherein the immutable result objects are result objects that are not modified or overwritten in place after being written; The method further includes: validating one or more of the following for each of the one or more immutable result objects written: byte count, content digest, output shape, data type, and corresponding range of the original data.

[0014] In some implementations, generating a list of pending bypasses that references the one or more immutable result objects and has a completion status of "complete" includes: When all one or more immutable result objects pass the verification, the object key and the claim scheduling generation of each of the one or more immutable result objects, as well as one or more of their byte count, content digest, output shape, data type, and original data corresponding range, are written into the bypass list, and the completion status of the bypass list is set to complete, generating a pending bypass list that references the one or more immutable result objects and has a completion status of complete.

[0015] In some implementations, performing the conditional commit on the list of bypasses to be committed includes: If the bypass list is not found, conditional write is performed on the bypass list to be submitted, based on the condition that no bypass list identified by the bypass list object key exists in the object storage; or... When the bypass list is found, the object is stored as the entity tag or object version identifier returned by the bypass list, and the condition is updated on the bypass list to be submitted, based on the condition that the current entity tag or object version identifier of the bypass list is consistent with the entity tag or object version identifier obtained during the query. If the condition writing or condition update fails, the task claiming object is read again, and the bypass list is queried again. If the completion status of the bypass list retrieved again is complete and the pre-calculation result referenced by the bypass list passes the verification, the submission of the bypass list to be submitted generated by the current working node is abandoned. When the claimed scheduling generation matches the current scheduling generation of the reread task claiming object record, the conditional submission is re-executed based on the result of the re-query of the bypass list. Specifically, if the bypass list is not found, the conditional write is re-executed on the condition that there is no bypass list identified by the bypass list object key in the object storage. If a bypass list with a completion status not indicated as complete is found, the conditional update is re-executed based on the entity tag or object version identifier obtained when re-querying the bypass list.

[0016] In some implementations, when the robot task trajectory data unit includes multiple time-series segments, the execution of the pre-calculation task by the worker node that has obtained the processing qualification includes: The working node that has obtained the processing qualification determines each of the time segments as a pre-computation subtask of the pre-computation task, and determines the time segment-level task claiming object key and the corresponding immutable result object object key for each of the pre-computation subtasks. Conditional operations are performed on each of the time-series segmented task claiming object keys, so that the work nodes that successfully execute the corresponding conditional operations acquire the processing qualification for the corresponding pre-computation sub-tasks, and the multiple work nodes that have acquired the processing qualification for different pre-computation sub-tasks execute their respective pre-computation sub-tasks in parallel. After a work node that has obtained the qualification to process the corresponding pre-computation subtask completes the corresponding pre-computation subtask, the pre-computation result corresponding to each pre-computation subtask is written into an immutable result object identified by the object key of the corresponding immutable result object.

[0017] In some implementations, the bypass list corresponding to the robot task trajectory data unit records the total number of time-series segments, reference information of immutable result objects corresponding to each time-series segment, and the time-series segment completion set. The method further includes: After the corresponding pre-calculated subtask is completed at the work node, the bypass list corresponding to the robot task trajectory data unit is read, and the entity tag or object version identifier corresponding to the bypass list returned by the object storage is obtained. Based on the time segment completed by the work node and its corresponding immutable result object, merge and update the reference information of the immutable result object corresponding to the time segment in the bypass list and the time segment completion set. When the merged and updated time series segment completion set indicates that all time series segments have been completed, the completion status of the bypass list is set to complete; The conditional update is performed on the merged and updated bypass list, based on the condition that the current entity tag or object version identifier of the bypass list is consistent with the entity tag or object version identifier obtained during this reading. If the conditional update fails, the bypass list is reread, and based on the reread bypass list and the newly obtained entity tag or object version identifier, the merge update and the conditional update are executed again.

[0018] In some implementations, determining the bypass list object key based on the pre-computation task includes: According to the preset bucketing rules, the target storage bucket is determined from multiple storage buckets based on the task key, and the bypass list object key is determined based on the target storage bucket and the task key; The method further includes: During task recovery, the bypass lists in the multiple storage buckets are scanned in parallel to obtain the pre-computation task set corresponding to the bypass list with a completion status of "complete". Based on the difference between the expected pre-computation task set and the pre-computation task set with a completion status of "complete", the incomplete pre-computation tasks are determined.

[0019] In some embodiments, in a second aspect, this application provides a distributed pre-computation coordination device for robot datasets, comprising: The task acquisition module is used to acquire pre-calculated tasks for robot task trajectory data units; The object key determination module is used to determine a task key, a task claiming object key, and a bypass list object key based on the pre-computation task. The task key is used to identify the pre-computation task, the task claiming object key is used to determine the task claiming object that records the current scheduling generation, and the bypass list object key is used to determine the bypass list that records the reference information and completion status of the pre-computation results. The bypass list query module is used to query the bypass list corresponding to the bypass list object key; The task assignment module is used to perform a conditional operation on the task assignment object key if the bypass list is not found, or if the bypass list is found but the completion status of the bypass list is not indicated as complete. This operation enables the work node that successfully executes the conditional operation to obtain the qualification to process the pre-computed task and to obtain the assignment scheduling generation. A pre-computation execution module is used to execute the pre-computation task by the work node that has obtained the processing qualification, and to obtain the pre-computation result of the pre-computation task; The module for generating a list of bypasses to be submitted is used to write the pre-calculated results into object storage to form one or more immutable result objects, and to generate a list of bypasses to be submitted that references the one or more immutable result objects and has a completion status of complete. The consistency verification module is used to perform consistency verification between the claimed scheduling generation and the current scheduling generation recorded by the task claiming object; The conditional commit module is used to perform a conditional commit on the list of bypasses to be committed when the consistency check passes. The result publishing module is used to, when the conditions are successfully submitted, use the successfully submitted bypass list as the official publishing bypass list of the pre-computation task, and determine the pre-computation results referenced by the official publishing bypass list as the official publishing results of the pre-computation task.

[0020] In some implementations, the object key determination module is further configured to: Obtain the dataset identifier, robot task trajectory data unit identifier, optional time segment identifier or target time range, and configuration information corresponding to the pre-calculation task; The task key is determined based on the dataset identifier, the robot task trajectory data unit identifier, the optional time segment identifier or the target time range, and the configuration information. The task claiming object key and the bypass list object key are determined according to the task key and the preset mapping relationship, respectively. The preset mapping relationship is used to characterize the deterministic correspondence between the task key of the same pre-computation task and the corresponding task claiming object key and bypass list object key.

[0021] In some implementations, the object key determination module is further configured to: The configuration information is deterministically serialized to obtain a configuration byte sequence; Perform a digest operation on the configuration byte sequence to obtain the configuration fingerprint; The task key is determined based on the dataset identifier, the robot task trajectory data unit identifier, the optional time segment identifier or the target time range, and the configuration fingerprint.

[0022] In some implementations, when the task claiming object corresponding to the task claiming object key does not exist in the object storage, the task claiming module is further configured to: If there is no task claim object identified by the task claim object key in the object storage, a conditional creation operation is performed based on the task claim object key to create the task claim object. The created task claim object records the worker node identifier, initial scheduling generation, claim time and lease expiration time of the worker node that successfully created the task claim object. The claim scheduling generation is the initial scheduling generation.

[0023] In some implementations, when a task claiming object corresponding to the task claiming object key already exists in the object storage, the task claiming module is further configured to: Read the task claiming object and its object metadata, obtain the current scheduling generation and lease expiration time from the task claiming object, and obtain the entity tag or object version identifier returned by the object storage from the object metadata, wherein the entity tag or the object version identifier is used to identify the storage version of the task claiming object at the time of this read; The current time is compared with the lease expiration time, and if the current time is equal to or later than the lease expiration time, it is determined that the lease corresponding to the task claiming object has expired; Determine a target scheduling generation higher than the current scheduling generation based on the current scheduling generation; If the current entity tag or object version identifier of the task claiming object is consistent with the entity tag or object version identifier obtained during this reading, a condition update is performed on the task claiming object. When the condition update is successful, the current scheduling generation recorded by the task claiming object is updated to the target scheduling generation, the work node identifier recorded by the task claiming object is updated to the work node identifier of the work node that reclaims the pre-computation task, and the lease expiration time recorded by the task claiming object is updated to the new lease expiration time, wherein the claimed scheduling generation is the target scheduling generation.

[0024] In some implementations, the bypass list query module is further used for: When a bypass list with a completion status of "complete" is found, the corresponding one or more immutable result objects are accessed based on the object key of one or more immutable result objects recorded in the bypass list. Availability verification is performed on the one or more immutable result objects, wherein the availability verification includes determining that all one or more immutable result objects exist, and determining that one or more of the actual byte count, content digest, output shape, data type, and original data corresponding range of each of the one or more immutable result objects are consistent with the corresponding information recorded in the bypass list; When the availability check passes, the pre-calculation result referenced by the bypass list is determined as the pre-calculation result of the pre-calculation task, and the conditional operation is no longer performed on the task claiming object key.

[0025] In some implementations, the bypass list generation module is further used for: Based on the task key and the claimed scheduling generation, determine the object key of each of the one or more immutable result objects; Using the object key of each of the one or more immutable result objects as a storage identifier, the pre-computed result is written into the object storage to form the one or more immutable result objects, wherein the immutable result objects are result objects that are not modified or overwritten in place after being written; The method further includes: validating one or more of the following for each of the one or more immutable result objects written: byte count, content digest, output shape, data type, and corresponding range of the original data.

[0026] In some implementations, the bypass list generation module is further used for: When all one or more immutable result objects pass the verification, the object key and the claim scheduling generation of each of the one or more immutable result objects, as well as one or more of their byte count, content digest, output shape, data type, and original data corresponding range, are written into the bypass list, and the completion status of the bypass list is set to complete, generating a pending bypass list that references the one or more immutable result objects and has a completion status of complete.

[0027] In some implementations, the condition submission module is further configured to: If the bypass list is not found, conditional write is performed on the bypass list to be submitted, based on the condition that no bypass list identified by the bypass list object key exists in the object storage; or... When the bypass list is found, the object is stored as the entity tag or object version identifier returned by the bypass list, and the condition is updated on the bypass list to be submitted, based on the condition that the current entity tag or object version identifier of the bypass list is consistent with the entity tag or object version identifier obtained during the query. If the condition writing or condition update fails, the task claiming object is read again, and the bypass list is queried again. If the completion status of the bypass list retrieved again is complete and the pre-calculation result referenced by the bypass list passes the verification, the submission of the bypass list to be submitted generated by the current working node is abandoned. When the claimed scheduling generation matches the current scheduling generation of the reread task claiming object record, the conditional submission is re-executed based on the result of the re-query of the bypass list. Specifically, if the bypass list is not found, the conditional write is re-executed on the condition that there is no bypass list identified by the bypass list object key in the object storage. If a bypass list with a completion status not indicated as complete is found, the conditional update is re-executed based on the entity tag or object version identifier obtained when re-querying the bypass list.

[0028] In some implementations, when the robot task trajectory data unit includes multiple time-series segments, the pre-computation execution module is further configured to: The working node that has obtained the processing qualification determines each of the time segments as a pre-computation subtask of the pre-computation task, and determines the time segment-level task claiming object key and the corresponding immutable result object object key for each of the pre-computation subtasks. Conditional operations are performed on each of the time-series segmented task claiming object keys, so that the work nodes that successfully execute the corresponding conditional operations acquire the processing qualification for the corresponding pre-computation sub-tasks, and the multiple work nodes that have acquired the processing qualification for different pre-computation sub-tasks execute their respective pre-computation sub-tasks in parallel. After a work node that has obtained the qualification to process the corresponding pre-computation subtask completes the corresponding pre-computation subtask, the pre-computation result corresponding to each pre-computation subtask is written into an immutable result object identified by the object key of the corresponding immutable result object.

[0029] In some implementations, the bypass list corresponding to the robot task trajectory data unit records the total number of time-series segments, reference information of immutable result objects corresponding to each time-series segment, and the time-series segment completion set. The device also includes a bypass list update module, which is used to read the bypass list corresponding to the robot task trajectory data unit after the work node completes the corresponding pre-calculated subtask, and obtain the entity tag or object version identifier corresponding to the bypass list returned by the object storage; according to the time-series segments completed by the work node and their corresponding immutable result objects, merge and update the reference information of the immutable result objects corresponding to the time-series segments and the time-series segment completion set in the bypass list; when the merged and updated time-series segment completion set indicates that all time-series segments have been completed, set the completion status of the bypass list to complete; perform a conditional update on the merged and updated bypass list based on the condition that the current entity tag or object version identifier of the bypass list is consistent with the entity tag or object version identifier obtained during this reading; if the conditional update fails, reread the bypass list, and perform the merge update and the conditional update again based on the reread bypass list and the re-obtained entity tag or object version identifier.

[0030] In some implementations, the object key determination module is further configured to: According to the preset bucketing rules, the target storage bucket is determined from multiple storage buckets based on the task key, and the bypass list object key is determined based on the target storage bucket and the task key; The method further includes: During task recovery, the bypass lists in the multiple storage buckets are scanned in parallel to obtain the pre-computation task set corresponding to the bypass list with a completion status of "complete". Based on the difference between the expected pre-computation task set and the pre-computation task set with a completion status of "complete", the incomplete pre-computation tasks are determined.

[0031] Thirdly, this application provides an electronic device, including: one or more processors; a storage device having one or more programs stored thereon; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method described in any embodiment of the first aspect of this application.

[0032] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the first aspect of this application.

[0033] Fifthly, this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method described in any embodiment of the first aspect of this application.

[0034] This application defines a task key, a task claiming object key, and a bypass list object key for a pre-computation task. When the bypass list is not found or its completion status is not indicated as "complete," a conditional operation is performed on the task claiming object key. This allows the worker node that successfully executes the conditional operation to acquire the processing qualification for the pre-computation task and obtain a claiming scheduling generation. This enables coordination of task processing qualifications among multiple worker nodes executing concurrently or sequentially taking over the same pre-computation task. The worker node that acquires processing qualification writes the pre-computation results to one or more immutable result objects. After the immutable result objects are written, a pending bypass list with a completion status of "complete" is generated. This aligns the writing status of the pre-computation results with the task completion status, reducing the possibility of pre-computation results overwriting each other and the reader accessing incompletely written pre-computation results based on the task completion status. Before submitting the pending bypass list, a consistency check is performed between the claimed scheduling generation and the current scheduling generation recorded in the task claiming object. If they match, a conditional commit is executed on the pending bypass list. This prevents work nodes whose claimed scheduling generations have expired from submitting their generated pending bypass lists and reduces the possibility of task completion status or pre-computation result reference information being overwritten due to concurrent commits from multiple work nodes. After a successful conditional commit, the successfully submitted bypass list is used as the official release bypass list for the pre-computation task, and the pre-computation results referenced in the official release bypass list are determined as the official release results of the pre-computation task. This enables coordination of the processing and result submission of the same pre-computation task by multiple work nodes, avoids the submission of expired results or the duplicate release of the same pre-computation task, and confirms the currently officially valid pre-computation results. Attached Figure Description

[0035] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings: Figure 1 This is a system architecture diagram of one embodiment of the distributed pre-computation coordination method for robot datasets according to this application; Figure 2 This is a flowchart of an embodiment of the distributed pre-computation coordination method for robot datasets according to this application; Figure 3 This is an exploded flowchart of an embodiment of step S202 of this application; Figure 4 This is an exploded flowchart of an embodiment of step S204 of this application; Figure 5 This is an exploded flowchart of an embodiment of step S205 of this application; Figure 6 This is an exploded flowchart of an embodiment of step S208 of this application; Figure 7 This is a schematic diagram of a structure of an embodiment of the robot dataset distributed pre-computation coordination device according to this application; Figure 8 This is a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of this application. Detailed Implementation

[0036] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of the present application. For ease of description, only the parts relevant to the present application are shown in the accompanying drawings.

[0037] It should be noted that, unless otherwise expressly stated, the embodiments and technical features described in this application can be combined with each other. Unless otherwise expressly stated, the ordinal numbers such as "first" and "second" used in this application are only used to distinguish different objects, and are not used to limit the order, quantity, or importance of the corresponding objects.

[0038] Figure 1 The system architecture 100 of one embodiment of the robot dataset distributed pre-computation coordination method of this application is shown.

[0039] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 provides a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired communication links, wireless communication links, or fiber optic communication links.

[0040] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send data. Terminal devices 101, 102, and 103 can be equipped with applications related to robot dataset processing, such as robot dataset pre-computation applications, pre-computation task scheduling applications, pre-computation result reading applications, data management applications, or other applications related to robot dataset processing.

[0041] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be desktop computers, portable computers, workstations, computing servers, graphics processing nodes, or other electronic devices with data processing capabilities. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. Terminal devices 101, 102, and 103 can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0042] Server 105 can be a server providing various services, such as a backend server processing pre-computation tasks sent by terminal devices 101, 102, and 103. The backend server can determine the task key, task assignment object key, and bypass list object key based on the pre-computation task, query the bypass list, and coordinate the assignment, execution, and result submission of the pre-computation task based on the task assignment object key if the bypass list does not exist or its completion status is not indicated as "complete." Server 105 can also provide object storage services to store task assignment objects, immutable result objects, and bypass lists, and perform condition creation or condition update on the corresponding objects.

[0043] In some cases, the distributed pre-computation coordination method for robot datasets provided in this application can be jointly executed by terminal devices 101, 102, and 103 and server 105. For example, terminal devices 101, 102, and 103 can act as working nodes, acquiring and executing pre-computation tasks for robot task trajectory data units, writing the pre-computation results into immutable result objects, and generating a list of bypasses to be submitted. Server 105 can provide task scheduling services and object storage services, and, based on object operation requests sent by terminal devices 101, 102, and 103, perform condition creation, condition writing, or condition updating on task claiming objects and bypass lists. This application does not limit this. Correspondingly, the distributed pre-computation coordination device for robot datasets can also be respectively set in terminal devices 101, 102, and 103 and server 105.

[0044] In some cases, the distributed pre-computation coordination method for robot datasets provided in this application can be executed by server 105, and correspondingly, the distributed pre-computation coordination device for robot datasets can also be set in server 105. For example, different computing nodes in server 105 can each act as worker nodes, and coordinate the assignment, execution, and result publication of pre-computation tasks through the object storage service provided by server 105. In this case, system architecture 100 may not include terminal devices 101, 102, and 103.

[0045] In some cases, the distributed pre-computation coordination method for robot datasets provided in this application can be executed by terminal devices 101, 102, and 103. Correspondingly, the distributed pre-computation coordination device for robot datasets can also be located in terminal devices 101, 102, and 103. For example, terminal devices 101, 102, and 103 can access independently deployed object storage services via network 104 and coordinate pre-computation tasks based on the condition creation and condition update capabilities provided by the object storage service. In this case, system architecture 100 may not include server 105.

[0046] Server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown in the diagram is for illustrative purposes only. Depending on the specific implementation requirements, system architecture 100 may include any number of terminal devices, networks, and servers.

[0048] Continue to refer to Figure 2 , Figure 2The flowchart S200 of one embodiment of the distributed pre-computation coordination method for robot datasets according to this application is shown.

[0049] Process S200 can be generated by Figure 1 The process can be executed by terminal devices 101, 102, 103 or server 105, or it can be executed collaboratively by terminal devices 101, 102, 103 and server 105. Process S200 may include the following steps S201 to S209.

[0050] Step S201: Obtain the pre-calculated task for the robot task trajectory data unit.

[0051] In this embodiment, the robot task trajectory data unit can refer to a data unit organized in units of a robot task execution process or its corresponding data record. It can include data such as video frame sequence, action sequence, timestamp, robot status and data unit identifier. In some engineering implementations, the robot task trajectory data unit can also be called an episode.

[0052] Pre-computation refers to the processing of robot task trajectory data units, such as encoding, feature extraction, statistics, or index generation, before using the robot dataset for model training, evaluation, or data analysis.

[0053] Accordingly, the pre-computation task can be a task used to instruct the working nodes to perform the above-described processing. The pre-computation results can include video latent variables.

[0054] Pre-computed tasks can be generated by the task generation component based on the task list of the robot dataset, can be issued by the scheduling service, or can be regenerated by the task recovery component based on the set of incomplete pre-computed tasks.

[0055] In some implementations, multiple worker nodes can obtain different pre-computed tasks from a pre-computed task list, a scheduling service, or a task recovery component, and access object storage separately. When multiple worker nodes obtain the same pre-computed task, subsequent steps allow each worker node to perform storage condition-protected contention for the same coordination object, without requiring each worker node to jointly hold a traditional blocking mutex lock.

[0056] In some implementations, the scheduling service can select candidate worker nodes for pre-computation tasks based on the region where the pre-computation task is located and the running status of each worker node.

[0057] Specifically, for candidate working nodes and pre-computation tasks The pre-calculation task can be determined according to formula (1). Assigned to candidate worker nodes The overall scheduling cost: (1) in, This indicates that the pre-computation task will be performed. Assigned to candidate worker nodes The overall scheduling cost; Indicates the expected network latency; This indicates queue load, occupied computing resources, or estimated waiting time. Indicates the cost of reading across regions; Indicates the estimated calculation time; This indicates normalization or equivalent cost conversion; to The weights are non-negative.

[0058] For example, when the normalization costs are 0.2, 0.4, 0.1 and 0.3 respectively, and the weights are 0.4, 0.2, 0.2 and 0.2 respectively, the overall scheduling cost is 0.24.

[0059] In some implementations, the scheduling service may prioritize working nodes with lower overall scheduling costs.

[0060] The above scheduling method only affects the priority of task allocation and does not change the conditions for task assignment and result publication described later.

[0061] After a worker node acquires a pre-computation task, it can check whether the necessary fields in the pre-computation task are complete and whether the original data object of the pre-computation task is accessible. If a necessary field is missing or the original data object is inaccessible, the worker node can record the task input error and stop further processing of the current pre-computation task.

[0062] Step S202: Determine the task key, task claiming object key, and bypass list object key based on the pre-calculated task.

[0063] Among them, the task key is used to identify the pre-computation task, the task claiming object key is used to determine the task claiming object that records the current scheduling generation, and the bypass list object key is used to determine the bypass list that records the reference information and completion status of the pre-computation results.

[0064] In this embodiment, the task assignment object is used to record the current assignment status of the pre-computed task. In addition to the current scheduling generation, the task assignment object may also record one or more of the following: worker node identifier, assignment time, and assignment status. The current scheduling generation indicates the currently valid task scheduling round for the task assignment object.

[0065] The bypass list is a metadata object stored separately from the original data object. Pre-computed result reference information may include the object key, storage location, or access address of one or more immutable result objects.

[0066] The bypass list can also record the claimed scheduling generation and result verification information.

[0067] The assignment scheduling cycle indicates the task scheduling round at which the corresponding work node acquires the qualification to process the pre-computed task.

[0068] Immutable result objects are used to store pre-computed results, and the content corresponding to the same object key is not modified or overwritten in place after writing.

[0069] refer to Figure 3 , Figure 3 This is an exploded flowchart of an embodiment of step S202 of this application.

[0070] In some implementations, step S202 may include steps S2021 to S2023.

[0071] Step S2021: Obtain the dataset identifier, robot task trajectory data unit identifier, optional time segment identifier or target time range, and configuration information corresponding to the pre-calculation task.

[0072] In this embodiment, the dataset identifier is used to distinguish different robot datasets.

[0073] Robot task trajectory data unit identifiers are used to identify the robot task trajectory data units to be processed in the corresponding robot dataset.

[0074] Optional time-series segment identifiers are used to identify target time-series segments obtained from robot task trajectory data units according to preset time-series segmentation rules. When the pre-calculation task uses pre-divided time-series segments as its processing object, the pre-calculation task can carry the corresponding time-series segment identifiers. When no time-series segment identifiers are set or not pre-generated, the time-series data interval to be processed can be determined by the target time range. The target time range is used to represent the corresponding interval of the target time-series segment in the original time-series data.

[0075] Configuration information may include one or more of the following: model identifier and version, temporal downsampling factor, target resolution, camera set arranged in a fixed order, preprocessing rule version, temporal segmentation boundary rule version, and output data type.

[0076] Step S2022: Determine the task key based on the dataset identifier, robot task trajectory data unit identifier, optional time segment identifier or target time range, and configuration information.

[0077] In some implementations, step S2022 can directly determine the task key by arranging the fields in a preset order. For example, the corresponding fields can be arranged in one or more of the following order: dataset identifier, robot task trajectory data unit identifier, temporal segment identifier or target time range, model identifier and version, temporal downsampling factor, target resolution, camera set, preprocessing rule version, temporal segment boundary rule version and output data type in the configuration information. Adjacent fields are connected using a preset separator to obtain a field combination string, and this field combination string is determined as the task key.

[0078] When a time-series segmentation identifier is used, the time-series segmentation identifier is written to the corresponding position in the field combination string; when a time-series segmentation identifier is not used, the target time range is written to that position.

[0079] In some implementations, the task key may also be determined using (A1)-(A3) as described later.

[0080] A1 performs deterministic serialization of the configuration information to obtain a configuration byte sequence.

[0081] In this embodiment, deterministic serialization can refer to converting configuration information into a byte sequence using a fixed field order, character encoding, default value representation, and numerical format, so that configuration information with the same content obtains the same configuration byte sequence on different working nodes.

[0082] A2 performs a digest operation on the configuration byte sequence to obtain the configuration fingerprint.

[0083] In this embodiment, the configuration fingerprint can refer to a fixed-length digest value obtained by performing a digest operation on the configuration byte sequence. It is used to characterize the configuration information used when generating the corresponding pre-computation results and to distinguish pre-computation tasks that use different configuration information.

[0084] When using the same digest algorithm, the same configuration byte sequence corresponds to the same configuration fingerprint. If any configuration field in the configuration byte sequence changes, the resulting configuration fingerprint will typically change accordingly.

[0085] In an example of generating video latent variables using a video variational autoencoder, the configuration fingerprint can be generated according to formula (2): (2) in, Indicates fingerprint configuration; Indicates the model identifier and version; Indicates the time-series downsampling factor; Indicates the target resolution; This refers to a set of cameras arranged in a fixed order. Indicates the version of the preprocessing rules; Indicates the version of the time-series segmentation boundary rule; Indicate the data type of the output; Indicates deterministic serialization; This represents the deterministic summary operation.

[0086] For example, when two worker nodes receive model version M1, temporal downsampling factor 4, target resolution 256×256, camera set {forward camera, wrist camera}, preprocessing rule version P2, temporal segmentation boundary rule version S1, and output data type float16 respectively, the two worker nodes obtain the same configuration byte sequence and configuration fingerprint. If the model version or output data type changes, different configuration fingerprints are obtained.

[0087] Where M1 represents the model version identifier of the video variational autoencoder used in this example; the temporal downsampling factor of 4 indicates that the sequence length of the video latent variables in the time dimension is approximately one-quarter of the input temporal data; the target resolution of 256×256 indicates that the image participating in the pre-computation includes 256 pixels in both the width and height directions; P2 represents the preprocessing rule version identifier used in this example; and S1 represents the temporal segmentation boundary rule version identifier used in this example. M1, 4, 256×256, P2, and S1 are all exemplary parameters and do not constitute a limitation on the model version, temporal downsampling factor, target resolution, preprocessing rule version, or temporal segmentation boundary rule version.

[0088] A3 determines the task key based on the dataset identifier, robot task trajectory data unit identifier, optional time sequence segment identifier or target time range, and configured fingerprint.

[0089] In this embodiment, the corresponding fields can be arranged in the order of dataset identifier, robot task trajectory data unit identifier, optional time segment identifier or target time range and configuration fingerprint, and adjacent fields can be connected by a preset separator to obtain a field combination string, which is then determined as the task key.

[0090] In other implementations, a digest operation can be performed on the combination of fields arranged and concatenated in the above order, and the resulting digest value can be used as the task key.

[0091] When using the same field order, encoding rules, and connection rules, different working nodes can determine the same task key based on the same dataset identifier, robot task trajectory data unit identifier, time sequence segment identifier or target time range, and configuration fingerprint. When at least one of the dataset identifier, robot task trajectory data unit identifier, time sequence segment determination information, or configuration fingerprint changes, the determined task key changes accordingly. Therefore, the task key can distinguish pre-computation tasks corresponding to different robot datasets, different robot task trajectory data units, different target time sequence segments, or different pre-computation configurations.

[0092] Step S2023: Determine the task claim object key and bypass list object key according to the task key and the preset mapping relationship respectively.

[0093] Among them, the preset mapping relationship is used to characterize the deterministic correspondence between the task key of the same pre-computation task and the corresponding task claiming object key and bypass list object key.

[0094] A deterministic mapping relationship can refer to the fact that, under the same preset mapping relationship, different working nodes can determine the same task claim object key and the same bypass list object key based on the same task key.

[0095] In this embodiment, the preset mapping relationship can be a path prefix mapping, a path suffix mapping, or an object key template mapping.

[0096] For example, a first path prefix representing the task assignment object can be appended to the task key to obtain the task assignment object key, and a second path prefix representing the bypass list can be appended to the same task key to obtain the bypass list object key. The first path prefix and the second path prefix are different from each other.

[0097] The first and second path prefixes can be used to distinguish the object category or storage namespace to which the task assignment object and the bypass list belong. They do not introduce variable information related to the pre-computation task, and therefore do not change the deterministic correspondence between the task key, the task assignment object key, and the bypass list object key. Thus, different working nodes acquiring the same pre-computation task can perform task assignment operations on the same task assignment object key, and can read or publish the completion status and pre-computation result reference information of the pre-computation task through the same bypass list object key.

[0098] In some implementations, the target storage bucket can be determined from multiple storage buckets according to a preset bucketing rule and the task key, and the bypass list object key can be determined according to the target storage bucket and the task key.

[0099] Storage buckets are storage units obtained by logically or physically dividing the storage space of bypass list objects.

[0100] Preset bucketing rules can include the number of storage buckets, hash algorithm, and the correspondence between bucket number and storage location.

[0101] By distributing the bypass list objects corresponding to different pre-computed tasks across multiple storage buckets, the access pressure generated when bypass list objects are stored centrally can be reduced.

[0102] Let the number of storage buckets be The task key can be determined according to formula (3). Corresponding target storage bucket number : (3) in, This represents a deterministic hash operation. This represents the modulo operation. This represents the number of storage buckets and is a positive integer. Indicates the target storage bucket number.

[0103] With the number of storage buckets and hash rules remaining constant, the same task key can be mapped to the same target storage bucket.

[0104] For example, when If the hash value is 1024 and the remainder after taking the hash integer corresponding to the task key modulo 1024 is 317, then the storage bucket numbered 317 is determined as the target storage bucket.

[0105] After determining the target storage bucket, the target storage bucket number or the corresponding storage location identifier can be used as the bucket path field in the bypass list object key, and the task key can be used as the task path field, thereby determining the bypass list object key.

[0106] For example, bypass list object keys can be constructed according to the field order of the target storage bucket number, bypass list category prefix, or task key.

[0107] Therefore, with the preset bucketing rules unchanged, different worker nodes can determine the same target storage bucket and the same bypass list object key based on the same task key.

[0108] Step S203: Query the bypass list corresponding to the bypass list object key.

[0109] In this embodiment, a worker node can send a read request containing a bypass list object key to the object storage to determine whether a corresponding bypass list exists in the object storage. If the bypass list exists, the worker node can read one or more of the following from the bypass list: completion status, pre-computation result reference information, claimed scheduling generation, and result verification information. If the bypass list does not exist, it can be determined that the pre-computation results of the corresponding pre-computation task have not yet been published through the bypass list.

[0110] A completion status of "complete" indicates that the pre-computed results referenced by the bypass manifest have been written. This completion status is the publication status recorded by the bypass manifest and does not alone indicate that the referenced pre-computed results can be used as official results. The worker node also needs to access the corresponding immutable result object based on the pre-computed result reference information and, in conjunction with the result validation information, validate one or more of the following for the immutable result object: existence, readability, object size, content digest, or data format.

[0111] When a bypass list has a completion status of "complete" and the immutable result object it references passes availability checks, the pre-computed results stored in that immutable result object can be considered as officially usable results. If the bypass list's completion status is not "complete," or the immutable result object it references fails availability checks, it cannot be determined solely from the bypass list that the corresponding pre-computed task has been completed.

[0112] In some implementations, after querying the bypass list corresponding to the bypass list object key, the following (B1)-(B3) may be included.

[0113] B1, when a bypass list with a completion status of "complete" is found, access one or more immutable result objects corresponding to the one or more immutable result objects recorded in the bypass list.

[0114] The object key of an immutable result object is used to determine the storage location of the corresponding immutable result object in object storage. A bypass list can record the object key of one immutable result object, or it can record the object keys of multiple immutable result objects.

[0115] When pre-computed results are saved separately according to different cameras, different data types, different time intervals, or different data slices, the bypass list can record the object keys of the immutable result objects corresponding to each part of the pre-computed results.

[0116] Worker nodes can send object read requests or object attribute query requests to the object store based on the object keys of each immutable result object to obtain one or more of the object content and object attributes of the corresponding immutable result object. B2 performs availability checks on one or more immutable result objects.

[0117] After retrieving the bypass list with a completion status of "complete", the worker node can access one or more immutable result objects based on the reference information of one or more immutable result objects recorded in the bypass list, and perform availability checks on one or more immutable result objects.

[0118] The availability verification may include determining that one or more immutable result objects exist, and determining that one or more of the following information of the one or more immutable result objects—the actual number of bytes, content digest, output shape, data type, and the corresponding range of the original data—are consistent with the corresponding information recorded in the bypass list.

[0119] Furthermore, availability verification may also include verifying whether the task key or configuration fingerprint recorded in the bypass list matches the currently pre-computed task.

[0120] When the bypass list records reference information for multiple immutable result objects, the availability check can be considered passed only after each immutable result object has passed its corresponding availability check.

[0121] B3, when the availability check passes, determines the pre-computation result referenced by the bypass list as the pre-computation result of the pre-computation task, and no longer performs conditional operations on the task claiming object key.

[0122] Conditional operations can refer to object creation or update operations that are performed by the object storage only when preset storage conditions are met. Preset storage conditions can include one or more of the following: the object corresponding to the object key does not exist; the entity tag or object version identifier returned by the object storage matches the specified entity tag or object version identifier; or the object content digest matches the specified content digest.

[0123] Here, no longer performing conditional operations on the task claiming object key means that when it has been confirmed that there is a formally usable pre-computation result, the worker node will no longer enter the task claiming competition described later, nor will it create or update the task claiming object for the current pre-computation task.

[0124] If no bypass list is found, or if a bypass list is found but its completion status is not indicated as complete, the process proceeds to step S204. If the bypass list structure is incomplete, the task key or configuration fingerprint recorded in the bypass list does not match the current pre-computation task, the immutable result object referenced by the bypass list does not exist, or the result verification information is inconsistent, the pre-computation result referenced by the bypass list is not used, and the current pre-computation task can continue to execute step S204 as an incomplete task.

[0125] For example, the bypass list records two result object keys, along with the number of bytes and content digest for each result object. Worker nodes access the corresponding immutable result objects based on the two result object keys. If both immutable result objects exist, and the actual number of bytes and the recalculated content digest for each immutable result object match the corresponding information recorded in the bypass list, the availability check passes. If either immutable result object is missing, the actual number of bytes is inconsistent, or the content digest is inconsistent, the availability check fails.

[0126] In step S204, if no bypass list is found, or if a bypass list is found but the completion status of the bypass list is not indicated as complete, a conditional operation is performed on the task claiming object key, so that the work node that successfully performs the conditional operation obtains the qualification to process the pre-computed task and obtains the claiming scheduling generation.

[0127] In this context, "processing eligibility" indicates that the corresponding worker node is allowed to continue executing the current pre-computation task and attempt to publish the generated pre-computation results as the official, usable results of that pre-computation task. "Claim scheduling generation" identifies the task scheduling round corresponding to when a worker node acquires processing eligibility and is used to determine whether the worker node that generated the pre-computation results still has valid publishing eligibility during subsequent result publishing.

[0128] In this embodiment, the conditional operation in step S204 may include conditional creation based on the condition that the task claim object does not exist, and conditional update based on the condition that the storage version of the task claim object is consistent with a specified storage version. Conditional creation is used to create a new task claim object when the task claim object key does not yet correspond to a task claim object. Conditional update is used to update the task claim object when it already exists and a preset update condition is met.

[0129] Object storage allows for atomic evaluation and execution of conditional operations on the same task claiming object key. When multiple worker nodes perform conditional operations on the same task claiming object key, the worker node that meets the preset storage conditions and whose conditional operation succeeds is granted processing eligibility. Worker nodes whose conditional operation fails are not granted processing eligibility. Successfully created or updated task claiming objects can record one or more of the following: worker node identifier, claiming scheduling generation, claiming time, lease expiration time, and claiming status. Worker nodes can determine their processing eligibility and corresponding claiming scheduling generation based on the successful operation response returned by object storage and the claiming scheduling generation recorded in the task claiming object.

[0130] Work nodes that fail a conditional operation can reread the task claim object and bypass list to determine the current task status, or select other pre-computed tasks, or re-execute the conditional operation on the current task claim object key when the re-claim conditions described below are met.

[0131] refer to Figure 4 , Figure 4 This is an exploded flowchart of an embodiment of step S204 of this application.

[0132] In one implementation, if there is no task claim object corresponding to the task claim object key in the object storage, step S2041 can be executed; if there is already a task claim object corresponding to the task claim object key in the object storage, steps S2042 to S2045 can be executed.

[0133] Step S2041: If there is no task claim object identified by the task claim object key in the object storage, perform a conditional creation operation based on the task claim object key to create a task claim object.

[0134] The created task claiming object records the worker node identifier, initial scheduling generation, claiming time, and lease expiration time of the worker node that successfully created the task claiming object. The claiming scheduling generation is the same as the initial scheduling generation.

[0135] The initial scheduling generation is the generation used when a task assignment object is first created. The scheduling generation represents the task scheduling round in which the pre-calculated task is positioned during task assignment and re-assignment, used to distinguish the processing eligibility acquired by different work nodes at different times. It can be a preset starting value. The assignment time represents the time when the corresponding work node acquires processing eligibility. The lease is status information in the task assignment object used to limit the valid time range of the corresponding processing eligibility. The valid time range of the lease can be determined by the assignment time and the lease expiration time. The lease expiration time can be determined based on the assignment time and the preset lease duration. The assignment time can be the operation time returned by object storage or the time of the time-synchronized work node.

[0136] In this embodiment, the worker node can send a conditional creation request to the object storage, carrying the task claim object key, the content of the task claim object to be written, and a condition that the object does not exist. When the object storage executes the request, it determines whether the task claim object key already corresponds to a task claim object; if no corresponding task claim object exists, it creates the task claim object and returns a creation success response to the worker node; if a corresponding task claim object already exists, it rejects the conditional creation request.

[0137] When multiple worker nodes concurrently perform conditional creation operations on the same task claim object key, the object storage only accepts creation requests that meet the condition that the object does not exist. A worker node that successfully creates a task claim object acquires the eligibility to process the current pre-computation task and uses the initial scheduling generation recorded in the task claim object as its claim scheduling generation. A worker node that fails to create the task claim object does not acquire the eligibility to process the task; it can reread the task claim object and bypass list, select another pre-computation task, or retry claiming the current pre-computation task after its lease expires.

[0138] For example, the initial scheduling generation can be set to 1, and the preset lease duration can be set to 10 minutes. Here, 1 indicates the first task scheduling round after the task claim object is created, distinguishing it from the state before the task claim object is created, where no task claim has occurred. The value 1 is merely an exemplary initial value and does not constitute a limitation on the initial scheduling generation. In other embodiments, other values ​​can be used as the initial scheduling generation according to preset scheduling generation initialization rules. When worker node W successfully creates a task claim object at 10:00, the task claim object can record the worker node identifier W, the initial scheduling generation 1, the claim time 10:00, the lease expiration time 10:10, and the claim scheduling generation as 1. Here, W is an exemplary worker node identifier used to distinguish different worker nodes. Subsequently, when other worker nodes perform conditional creation operations on the same task claim object key, the corresponding conditional creation operation fails because the task claim object already exists.

[0139] Step S2042: Read the task claiming object and its object metadata, obtain the current scheduling generation and lease expiration time from the task claiming object, and obtain the entity tag or object version identifier returned by the object storage from the object metadata.

[0140] Among them, the entity tag (ETag) or object version identifier is used to identify the storage version of the task-claimed object during this read.

[0141] Work nodes can also obtain one or more of the following from the task assignment object: the currently recorded work node identifier, assignment time, and assignment status. After reading the task assignment object and its object metadata, the work node associates the entity tag or object version identifier obtained in this read with the current scheduling generation and lease expiration time, and saves it for subsequent condition updates.

[0142] Step S2043: Compare the current time with the lease expiration time, and determine that the lease corresponding to the task claiming object has expired when the current time is at or after the lease expiration time.

[0143] In this embodiment, the current time and the lease expiration time use the same time base and time unit. The current time can be the server time returned by the object storage or the time of the synchronized worker node. When the current time is earlier than the lease expiration time, it indicates that the original worker node's lease has not yet expired, and the currently competing worker node does not cover the task claiming object, nor does it acquire the processing qualification for the current pre-computation task. When the current time reaches or is later than the lease expiration time, it indicates that the currently competing worker node is allowed to initiate a reclaim, but the lease expiration itself does not automatically grant the worker node the processing qualification; the worker node still needs to update its competitive processing qualification through step S2045.

[0144] Step S2044: Determine a target scheduling generation higher than the current scheduling generation based on the current scheduling generation.

[0145] In this embodiment, the target scheduling generation can be obtained by adding a preset positive integer to the current scheduling generation. For example, the preset positive integer can be 1. The target scheduling generation is higher than the current scheduling generation, so that the reclaimed scheduling round can be distinguished from the previous scheduling round, and the subsequent result publishing process can identify the results generated by the previous scheduling round according to the scheduling generation, avoiding work nodes whose leases have expired from publishing results with a lower scheduling generation.

[0146] Step S2045: Based on the condition that the current entity label or object version identifier of the task claiming object is consistent with the entity label or object version identifier obtained during this reading, perform condition update on the task claiming object. When the condition update is successful, update the current scheduling generation recorded by the task claiming object to the target scheduling generation, update the work node identifier recorded by the task claiming object to the work node identifier of the work node that reclaims the pre-computation task, and update the lease expiration time recorded by the task claiming object to the new lease expiration time.

[0147] The assigned scheduling generation is the target scheduling generation. The new lease expiration time can be determined based on the current time at the time of execution condition update and the preset lease duration. The task assignment object can also record the re-assignment time and update the assignment status to indicate that the task is being processed.

[0148] The conditional update in step S2045 is equivalent to performing a comparison and update operation based on the stored version information. Object storage is updated only if the current entity label or object version identifier of the task claiming object matches the entity label or object version identifier read by the worker node in step S2042. A worker node that successfully updates its condition acquires the eligibility to process the pre-computed task and obtains the target scheduling generation as the claiming scheduling generation. A failed conditional update indicates that the task claiming object has been updated by another worker node after this read; the current worker node does not acquire the eligibility to process the task and rereads the task claiming object and bypass list without overwriting the task claiming object with the old version information.

[0149] For example, worker node A obtains scheduling generation 5 through conditional creation. Scheduling generation 5 indicates that when worker node A successfully creates a task claiming object, the current scheduling generation recorded for that task claiming object is 5. Worker node A acquires the processing qualification corresponding to scheduling generation 5 and records this claimed scheduling generation locally. The value 5 is for illustrative purposes only and does not constitute a limitation on the initial scheduling generation value. When worker node A's lease expires, both worker nodes B and C read scheduling generation 5 and the same entity tag, and respectively determine the target scheduling generation as 6. The target scheduling generation 6 is obtained by adding a preset positive integer 1 to the current scheduling generation 5. This is used to identify the new task scheduling round corresponding to worker node B's re-claiming of the pre-computation task, and to distinguish this new task scheduling round from the previous task scheduling round corresponding to worker node A. The value 6 is for illustrative purposes only and does not constitute a limitation on the target scheduling generation value. When worker nodes B and C perform conditional updates based on the read entity tag, only the conditional update that first satisfies the version consistency condition can succeed. If worker node B's condition update succeeds, worker node B acquires processing eligibility and claims scheduling generation 6, and the entity label of the task claiming object changes accordingly. Worker node C subsequently fails to update its condition based on the old entity label and therefore does not acquire processing eligibility. Here, A, B, and C are exemplary worker node identifiers used to distinguish different worker nodes, only used to indicate that the corresponding operations are performed by different worker nodes.

[0150] Step S205: The working node that has obtained processing qualification executes the pre-computation task and obtains the pre-computation result of the pre-computation task.

[0151] In this embodiment, the working node that has obtained processing qualification can read all the original data corresponding to the robot task trajectory data unit from the object storage according to the original data access location recorded in the pre-calculation task, or read part of the original data corresponding to the target time range, and perform pre-calculation according to the configuration information to obtain the pre-calculation result of the pre-calculation task.

[0152] Step S206: Write the pre-computed results into object storage to form one or more immutable result objects, and generate a list of pending bypasses that reference one or more immutable result objects and have a completion status of complete.

[0153] The pending-commit bypass list consists of bypass lists that have been generated but not yet written to the storage location corresponding to the bypass list object key via the conditional commit operation described later. It carries reference information and completion status of candidate pre-computation results. A completion status of "complete" in the pending-commit bypass list indicates that one or more immutable result objects it references have been written, but does not mean that the corresponding candidate pre-computation result has become a formally published pre-computation result. Only after the conditional commit operation described later is successful does the pending-commit bypass list become a formally published bypass list, and only then are the candidate pre-computation results it references published as formal pre-computation results for the corresponding pre-computation task.

[0154] Worker nodes can write the pre-computed results to object storage to form one or more immutable result objects. Each immutable result object can correspond to an independent object key. After object storage confirms that the immutable result object has been successfully written, the worker node can obtain or calculate one or more of the following: the actual number of bytes, content digest, output shape, data type, and the range corresponding to the original data, as result verification information.

[0155] After one or more immutable result objects have been written, the worker node generates a list of bypasses to be committed. The list of bypasses to be committed can record one or more of the following: task key, configuration fingerprint, object key of one or more immutable result objects, result verification information corresponding to each immutable result object, claimed scheduling generation, and completion status.

[0156] When a precomputation task fails, or when any immutable result object has not yet been fully written, the worker node does not generate a pending bypass list with a completion status of "complete," nor does it publish the precomputation results that have not yet been fully written through the bypass list object key. Therefore, the bypass list is generated later than the order in which all the immutable result objects it references are written, avoiding the formal publication of bypass list references to result objects that have not yet been fully written.

[0157] refer to Figure 5 , Figure 5This is an exploded flowchart of an embodiment of step S205 of this application.

[0158] In some implementations, writing the pre-calculated results into one or more immutable result objects can be achieved by including steps S2051 to S2053.

[0159] Step S2051: Determine the object key of one or more immutable result objects based on the task key and the claimed scheduling generation.

[0160] In this embodiment, the corresponding fields can be combined according to the preset arrangement order of fields such as task key, claim scheduling generation, result object category or result fragment number, and adjacent fields can be connected by preset separators, fixed-length fields or length prefixes to obtain the object key of the immutable result object.

[0161] When the pre-computed result is carried by multiple immutable result objects, one or more of the following can be added to the object key of each immutable result object to distinguish different immutable result objects: camera identifier, data type identifier, time sequence identifier, or result slice number.

[0162] By adding the claimed scheduling generation to the object key of the immutable result object, the pre-computation results generated by different scheduling generations can correspond to different storage locations. Therefore, when re-executing the pre-computation task or generating new pre-computation results, the new pre-computation results can be written to the new immutable result object by determining the object key of the new immutable result object, without having to modify or overwrite the previously written immutable result object in situ.

[0163] Step S2052: Using the object key of each of the one or more immutable result objects as a storage identifier, the pre-computed result is written to the object storage to form one or more immutable result objects. In this embodiment, the worker node can send an object write request to the object storage for each immutable result object's object key. When the pre-computed result consists of multiple result parts, each result part can be written to the corresponding immutable result object, and the corresponding immutable result object is determined to have been written after the object storage returns a write success response.

[0164] An immutable result object is a result object that is not modified or overwritten in-place after being written.

[0165] Here, when it is necessary to write new versions of results corresponding to different scheduling generations, results with different content, or pre-computed results, a new immutable result object is written using a new object key. If the writing process of any immutable result object is interrupted, fails, or does not receive a successful write response, the worker node does not generate a list of pending commits with a completion status of "complete".

[0166] Step S2053: Verify one or more of the following for each of the one or more immutable result objects written: number of bytes, content digest, output shape, data type, and corresponding range of the original data.

[0167] In this embodiment, the worker node can read the object attributes or object content of the immutable result object to obtain one or more of the following: actual byte count, recalculated content summary, parsed output shape, data type, and the corresponding range of the original data, and compare it with the expected information generated during the pre-computation process. When a pre-computation result is carried by multiple immutable result objects, the pre-computation result is determined to have passed the verification after each immutable result object has passed the corresponding verification.

[0168] For video latent variables, it is also possible to verify whether the time length of the latent variable matches the target time range and the corresponding temporal downsampling factor, and whether the camera set, resolution, and preprocessing rule version corresponding to the latent variable match the configuration information. If any immutable result object fails the verification, the corresponding pre-calculated result will not be published as a complete candidate pre-calculated result.

[0169] Based on the above steps S2051 to S2053, generating a list of bypasses to be committed that references one or more immutable result objects and has a completion status of complete may include step S2054.

[0170] Step S2054: When one or more immutable result objects pass the verification, write one or more of the following into the bypass list: the object key and claim scheduling generation of each of the one or more immutable result objects, as well as their byte count, content digest, output shape, data type and original data corresponding range. Set the completion status of the bypass list to complete and generate a bypass list to be committed that references one or more immutable result objects and has a completion status of complete.

[0171] In this embodiment, candidate content for the bypass list can be generated in memory or temporary storage space, and the object key of each immutable result object can be written into the candidate content as a pre-computation result reference information. The bypass list can also record task keys, configuration fingerprints, and one or more of the following: the number of bytes, content summary, output shape, data type, and the corresponding range of the original data for each immutable result object. The claimed scheduling generation written into the bypass list is the claimed scheduling generation obtained when the current working node acquires the qualification to process the pre-computation task, which is used to subsequently determine whether the candidate content was generated by the current valid scheduling round.

[0172] After one or more immutable result objects have been written and validated, and the aforementioned information has been written to the bypass list, the worker node sets the completion status of the bypass list to "complete," thus obtaining the bypass list to be committed. The bypass list to be committed has not yet been committed according to the conditions described later; therefore, its completion status of "complete" only indicates that one or more immutable result objects referenced by the bypass list to be committed have been written and validated, and does not indicate that the bypass list to be committed has become a formally released bypass list.

[0173] The steps S2051 to S2054 above form a two-stage release sequence: first, one or more immutable result objects are written, and then a list of bypasses to be committed is generated. Therefore, before the list of bypasses to be committed is officially released, the immutable result objects it references already exist and can undergo availability checks.

[0174] In some alternative implementations, when the robot task trajectory data unit includes multiple time-series segments, the pre-calculation task performed by the qualified working node may also include steps S2055 to S2057.

[0175] In step S2055, the working node that has obtained the processing qualification determines each time segment as a pre-computation subtask of the pre-computation task, and determines the time segment-level task claiming object key and the corresponding immutable result object object key for each pre-computation subtask.

[0176] Among them, the pre-computation subtask is a task whose processing scope is a time segment of the robot task trajectory data unit. The working node can determine the subtask key of each pre-computation subtask according to the task key of the pre-computation task and the time segment identifier of each time segment.

[0177] Based on the path prefix corresponding to each subtask key and the task claiming object, determine the task claiming object key for each time-series segment.

[0178] The object key for each immutable result object is determined based on at least one of the following: the subtask key, the claimed scheduling generation, and the content summary of the corresponding pre-computed result. Different time segments have different subtask keys and time segment-level task claiming object keys.

[0179] Step S2056: Perform conditional operations on the task claiming object keys of each time segment level, so that the work nodes that successfully execute the corresponding conditional operations obtain the processing qualification for the corresponding pre-computation sub-tasks, and the multiple work nodes that have obtained the processing qualification for different pre-computation sub-tasks execute their respective pre-computation sub-tasks in parallel.

[0180] Specifically, the eligibility to process a pre-computation subtask is associated with the corresponding time-series segment-level task assignment object key, and its validity is limited to the pre-computation subtasks corresponding to that time-series segment. A worker node acquiring eligibility to process one pre-computation subtask does not mean that the worker node simultaneously acquires eligibility to process other pre-computation subtasks. A worker node that acquires eligibility to process the entire pre-computation task can also participate in the assignment and execution of one or more pre-computation subtasks.

[0181] In this embodiment, the conditional operation performed for each time-series segmented task claiming object key can adopt the same method as the conditional creation operation or conditional update operation in step S204. For pre-computation subtasks that do not yet have a time-series segmented task claiming object, a conditional creation operation can be performed based on the condition that the corresponding object does not exist. For pre-computation subtasks that already have a time-series segmented task claiming object and whose lease has expired, a conditional update operation can be performed based on the object's entity tag or object version identifier.

[0182] Each time-series segment-level task claiming object key corresponds to a different time-series segment. Therefore, the conditional operations performed on different time-series segment-level task claiming object keys are independent of each other. Multiple worker nodes can perform conditional operations on different time-series segment-level task claiming object keys and obtain processing eligibility for different pre-computed subtasks. For the same time-series segment-level task claiming object key, object storage only accepts conditional operations that meet preset storage conditions. Worker nodes that fail to successfully execute the conditional operation do not obtain processing eligibility for the corresponding pre-computed subtask.

[0183] Work nodes that have obtained the qualification to process the corresponding pre-computation subtasks read the raw data according to the time segment identifier or target time range corresponding to the pre-computation subtask, and perform pre-computation according to the configuration information corresponding to the pre-computation task.

[0184] Step S2057: After the working node that has obtained the qualification to process the corresponding pre-computation subtask completes the corresponding pre-computation subtask, the pre-computation result corresponding to each pre-computation subtask is written into the immutable result object identified by the object key of the corresponding immutable result object.

[0185] After a worker node that has qualified for processing a pre-computation subtask completes the corresponding pre-computation subtask, it writes the pre-computation result of the subtask into the immutable result object identified by the object key of the immutable result object determined in step S2056. Each worker node reads and calculates for different original data ranges and writes them into the corresponding immutable result objects. Therefore, different pre-computation subtasks do not need to be executed sequentially according to the time sequence.

[0186] Work nodes that fail to acquire processing rights for a pre-computation subtask will not execute that subtask. They can re-read the corresponding time-series segmented task claiming object, select other unclaimed pre-computation subtasks, or re-attempt to claim the subtask after the corresponding lease expires. The failure or needing to reclaim a pre-computation subtask does not affect the continued execution of pre-computation subtasks already acquired processing rights by other work nodes.

[0187] Based on the implementation of steps S2056 and S2057, the bypass list corresponding to the robot task trajectory data unit can record the total number of time segments, the reference information of the immutable result objects corresponding to each time segment, and the time segment completion set, and can further execute steps S2058 to S20512.

[0188] The time-series segment completion set records time-series segments that have completed pre-computation, whose corresponding immutable result objects have been written, and whose results have passed validation. The time-series segment completion set can be represented as a time-series segment identifier set, a time-series segment sequence number set, or a bitmap. Reference information for the immutable result objects corresponding to each time-series segment can be recorded according to the mapping relationship between the time-series segment identifier or time-series segment sequence number and the corresponding immutable result object key.

[0189] Step S2058: After the corresponding pre-calculated subtask is completed at the working node, the bypass list corresponding to the robot task trajectory data unit is read, and the entity tag or object version identifier corresponding to the bypass list returned by the object storage is obtained.

[0190] In this embodiment, the working node that completes the pre-computation subtask reads the current bypass list according to the bypass list object key to obtain the currently recorded time series segment completion set, the reference information of the immutable result objects corresponding to each time series segment, and the completion status, and obtains the entity tag or object version identifier corresponding to the bypass list returned by the object storage.

[0191] If the bypass list does not yet exist, an initial bypass list can be created based on the condition that the bypass list does not exist. The initial bypass list can record the total number of time series segments, the set of empty completed time series segments, and the incomplete status. When multiple worker nodes concurrently create the initial bypass list, only one condition creation operation can succeed. Worker nodes whose condition creation fails will reread the already created bypass list.

[0192] Step S2059: Based on the time-series segments completed by the working nodes and their corresponding immutable result objects, merge and update the reference information of the immutable result objects corresponding to the time-series segments in the bypass list and the time-series segment completion set.

[0193] In this embodiment, according to the time sequence segment identifier, the object key and result verification information of the immutable result object corresponding to the time sequence segment completed by the current working node can be added to the bypass list, and the reference information of the immutable result objects corresponding to other time sequence segments already recorded in the bypass list can be retained. For the same time sequence segment already recorded in the bypass list, the reference information that has passed the verification can be retained to avoid repeatedly writing the same time sequence segment record.

[0194] Let the set of completed time-series segments be read as follows: The set of time series segments that have been newly completed and have passed result verification at the current working node is: The merged time series segmentation complete set It can be determined according to formula (4): (4) in, This represents a set union operation. When using a bitmap to represent the completion state of timing segments, an equivalent bitwise OR operation can be performed. Set union or bitwise OR operations are commutative, associative, and idempotent. Therefore, repeatedly merging the same timing segment will not delete other timing segments already recorded in the bypass list, nor will repeatedly performing the merge operation change the final completed set of timing segments.

[0195] Step S20510: When the merged and updated time series segment completion set indicates that all time series segments have been completed, set the completion status of the bypass list to complete.

[0196] In this embodiment, the complete set of time series segments can be determined based on the total number of time series segments recorded in the bypass list and the identifier or sequence number of each time series segment. When the merged and updated complete set of time series segments contains every time series segment in the complete set of time series segments, and the immutable result objects corresponding to each time series segment have been written and passed the result verification, the completion status of the bypass list is set to complete; otherwise, the bypass list remains incomplete.

[0197] The "complete" status here indicates that the bypass list has collected all candidate pre-computation results corresponding to the time series segments, but does not mean that the bypass list has become a formally released bypass list. This bypass list can still be published as a bypass list to be submitted, according to the following process of claiming scheduling generation verification and conditional submission.

[0198] Step S20511: Perform a conditional update on the merged and updated bypass list, based on the condition that the current entity label or object version identifier in the bypass list is consistent with the entity label or object version identifier obtained during this read.

[0199] The conditional update in step S20511 is used to prevent multiple worker nodes from overwriting each other when updating based on the same old version of the bypass list. Object storage is written to the merged and updated bypass list only if the current entity label or object version identifier in the bypass list matches the entity label or object version identifier read by the worker node in step S2058.

[0200] When the conditional update succeeds, the reference information, time-series segment completion set, and completion status of the immutable result object obtained from this merge are written to the bypass list. The object storage can generate new entity tags or object version identifiers for the updated bypass list. When the conditional update fails, it indicates that the bypass list has been updated by other worker nodes after this read, and the current worker node will not use content generated based on the old version to overwrite the current bypass list.

[0201] In step S20512, if the conditional update fails, the bypass list is reread, and based on the reread bypass list and the re-obtained entity tag or object version identifier, the merge update and conditional update are performed again.

[0202] In this embodiment, after a worker node rereads the bypass list, it merges the completed timing segments and the reference information of the corresponding immutable result objects with the information already recorded in the latest bypass list, and then performs a condition update again based on the latest entity tag or object version identifier. Since the timing segment completion set is merged using a set union operation or an equivalent bitwise OR operation, re-performing the merge will not delete the timing segment completion records already written by other worker nodes.

[0203] Work nodes can repeatedly perform reread, merge update, and condition update until the condition update is successful, it is confirmed that the current time sequence segment has been recorded in the bypass list, or the preset retry condition is met. When the preset retry condition is met, the work node can wait for a preset backoff time and then try again to reduce conflicts caused by multiple work nodes continuously updating the same bypass list concurrently.

[0204] For example, the bypass list records a total of 16 time series segments. The currently read completed time series segment set is {2}, the newly completed time series segment set of the current worker node is {5}, and the merged completed time series segment set is {2,5}. Since this completed set does not yet include all 16 time series segments, the bypass list remains in an incomplete state. When the subsequently merged completed time series segment set includes all 16 time series segments, and the immutable result objects corresponding to each time series segment pass the result validation, the completion status of the bypass list is set to complete.

[0205] Step S207: Perform a consistency check between the claimed scheduling generation and the current scheduling generation recorded in the task claiming object record.

[0206] In this embodiment, consistency verification is used to determine whether the worker node that generated the bypass list to be submitted is still in the current valid task scheduling round. By verifying the claimed scheduling generation before submitting the bypass list to be submitted, it is possible to prevent older worker nodes whose leases have expired or whose processing rights have been regained by other worker nodes from publishing pre-computation results.

[0207] In some implementations, if the claimed scheduling generation is consistent with the current scheduling generation, the consistency check can be confirmed as passed.

[0208] In some implementations, the task assignment object also records the work node identifier and lease expiration time. Step S206 can also be implemented in the following ways.

[0209] First, before submitting the list of bypasses to be submitted, the task assignment object is read to obtain the work node identifier, current scheduling generation, and lease expiration time recorded in the task assignment object. Then, if the assigned scheduling generation is consistent with the current scheduling generation, or if the work node identifier recorded in the task assignment object is consistent with the work node identifier of the work node that has obtained processing eligibility, the assigned scheduling generation is consistent with the current scheduling generation, and the current time is earlier than the lease expiration time, the consistency check is considered successful.

[0210] In this embodiment, the worker node sends a read request to the object storage based on the task claim object key to obtain the information currently recorded in the task claim object, instead of using the old object content cached when the claim was successful. If reading the task claim object fails, the worker node does not submit the pending submission bypass list and can re-initiate the read request or stop the current submission process.

[0211] Specifically, consistent work node identifiers indicate that the current task claiming object still records the corresponding work node as a work node eligible for processing; consistent claiming scheduling generation and current scheduling generation indicate that the processing eligibility of the corresponding work node belongs to the currently valid task scheduling round; and current time being earlier than lease expiration time indicates that the corresponding processing eligibility is still within the lease's valid timeframe. If all three conditions are met, the consistency check passes; if any condition is not met, the consistency check fails.

[0212] Let the claim scheduling generation obtained by the current working node upon successful claim be . The current scheduling generation of the task claiming object is... The generational consistency condition can be expressed as formula (5): (5) when Less than When the current worker node holds a scheduling generation lower than the current scheduling generation recorded in the object storage, it means that the current worker node belongs to an old scheduling round that has expired.

[0213] when Greater than This indicates that the local scheduling information stored by the current worker node is inconsistent with the current claim status in object storage. In both of these cases, the current worker node will not submit the list of bypasses to be submitted.

[0214] when equal However, if the work node identifier recorded by the task assignment object is inconsistent with the work node identifier of the current work node, or if the current time is at or after the lease expiration time, the consistency check will also fail, and the current work node will not submit the pending bypass list.

[0215] For example, when worker node A acquires processing eligibility, its assigned scheduling generation is 5. After worker node A's lease expires, worker node B re-acquires the pre-computation task and updates the current scheduling generation recorded in the task assignment object to 6. When worker node A resumes operation and prepares to submit the pending bypass list, it rereads the task assignment object and obtains... For 5 The value is 6. Since the claimed scheduling generation is inconsistent with the current scheduling generation, the consistency check fails, and worker node A does not submit its generated list of bypasses to be submitted.

[0216] Step S208: When the consistency check passes, perform a conditional commit on the bypass list to be committed.

[0217] Conditional commit refers to writing the bypass list to be committed to the storage location corresponding to the key of the bypass list object, based on the condition that the bypass list object does not exist or that the storage version of the bypass list remains consistent. Conditional commit can include conditional write and conditional update. Upon successful conditional commit, the bypass list to be committed written to object storage becomes the officially released bypass list, and one or more immutable result objects referenced by it become the official pre-computation results of the corresponding pre-computation task. If the conditional commit fails, the bypass list to be committed generated by the current worker node does not become the officially released bypass list.

[0218] refer to Figure 6 , Figure 6 This is an exploded flowchart of an embodiment of step S208 of this application.

[0219] In some implementations, step S208 may include steps S2081 to S2085.

[0220] Step S2081: If no bypass list is found, perform condition writing on the bypass list to be submitted, assuming that there is no bypass list identified by the bypass list object key in the object storage.

[0221] In this embodiment, the worker node sends a conditional write request to the object storage, including the bypass list object key, the bypass list to be committed, and a condition that the object does not exist. If the object storage does not yet have a corresponding bypass list entry for the bypass list object key, it writes the bypass list to be committed and returns a write success response. If the bypass list object key already has a corresponding bypass list entry, the conditional write request is rejected.

[0222] When multiple worker nodes concurrently perform conditional writes on the same bypass list object key, only writes that satisfy the condition that the object does not exist will succeed. If other worker nodes have already created the bypass list, the object does not exist condition, the write operation will fail, and the already created bypass list will not be overwritten.

[0223] Step S2082: When the bypass list is found, obtain the entity label or object version identifier returned by the bypass list and perform condition update on the bypass list to be submitted, based on the condition that the current entity label or object version identifier of the bypass list is consistent with the entity label or object version identifier obtained during the query.

[0224] The conditional update in step S2082 is used to avoid overwriting the bypass list that has already been updated by other worker nodes based on old version information when the bypass list already exists. Object storage only writes the bypass list to be committed to the storage location corresponding to the bypass list object key if the current entity label or object version identifier of the bypass list matches the entity label or object version identifier obtained when the worker node is queried.

[0225] When the condition update succeeds, the bypass list to be committed becomes the current stored version of the bypass list. When the condition update fails, it indicates that the bypass list has been created or updated by other worker nodes after this query, and the current worker node will not use the bypass list to be committed generated based on the old version to overwrite the current bypass list.

[0226] Step S2083: If the condition writing or condition update fails, reread the task claim object and re-query the bypass list.

[0227] In this embodiment, rereading the task claim object is used to obtain one or more of the following: the latest worker node identifier, the current scheduling generation, and the lease expiration time. Re-querying the bypass list is used to determine whether other worker nodes have submitted officially usable pre-computation results, and to obtain the latest completion status of the bypass list, pre-computation result reference information, and entity tags or object version identifiers. Worker nodes do not retry directly based on the cached task claim object status or bypass list version before conditional submission failure.

[0228] Step S2084: When the completion status of the bypass list retrieved again is complete and the pre-calculation result referenced by the bypass list passes the verification, abandon the submission of the bypass list to be submitted generated by the current working node.

[0229] The validation of pre-computed results referenced by the bypass list can be performed using the B2 availability validation method. When the bypass list retrieved again references officially available pre-computed results, the current worker node stops performing conditional commits on that bypass list object key and does not replace the officially released bypass list with the bypass list to be committed generated by the current worker node.

[0230] Immutable result objects that have been written to object storage but not referenced in the officially released bypass list for the current working node will not be considered as the officially released results of the corresponding pre-computation task. These immutable result objects can be retained for fault analysis or cleaned up when a preset retention period or garbage collection conditions are met.

[0231] Step S2085: When the claimed scheduling generation is consistent with the current scheduling generation of the reread task claiming object record, re-execute the conditional submission based on the re-query result of the bypass list.

[0232] In this embodiment, before resubmitting the conditions, it is also possible to verify whether the work node identifier recorded in the task claiming object is consistent with the work node identifier of the current work node, and whether the current time is earlier than the lease expiration time. When the claimed scheduling generation is consistent with the current scheduling generation, and all other verification conditions are met, the current work node resubmits the conditions based on the latest bypass list query results.

[0233] Specifically, if the bypass list is not found again, the conditional write operation is re-executed based on the absence of a bypass list identified by the bypass list object key in the object storage. If a bypass list with a completion status not indicated as "complete" is found again, the conditional update operation is re-executed based on the entity tag or object version identifier obtained during the bypass list re-query.

[0234] When the claimed scheduling generation is inconsistent with the current reread scheduling generation, the work node identifier recorded by the task claiming object is inconsistent with the work node identifier of the current work node, or the current time reaches or is later than the lease expiration time, the current work node will no longer retry submitting its generated pending submission bypass list.

[0235] Under high concurrency conditions, worker nodes can use random backoff or exponential backoff after a conditional commit failure. After the appropriate backoff time, they can reread the task claiming object, re-query the bypass list, and re-execute the conditional commit. Backoff only changes the timing of rereading and re-committing; it does not change the bypass list storage version or the claiming scheduling generation used for each conditional commit.

[0236] Step S209: When the conditions are successfully submitted, the successfully submitted bypass list is used as the official release bypass list of the pre-computation task, and the pre-computation results referenced in the official release bypass list are determined as the official release results of the pre-computation task.

[0237] The formally released bypass list is a list of bypasses that has been submitted according to the conditions in step S208 and is used to externally determine the formally released results of the corresponding pre-computation tasks. The formally released results are the pre-computation results stored in one or more immutable result objects referenced by the formally released bypass list and which have passed availability checks.

[0238] In this embodiment, when the object storage returns a successful condition write response or a successful condition update response, it can be determined that the condition has been successfully submitted. The successfully submitted bypass list is saved to the storage location corresponding to the bypass list object key, and can record one or more of the following: task key, configuration fingerprint, claimed scheduling generation, completion status, one or more result object keys, and corresponding result verification information. The object storage can also return entity tags or object version identifiers for officially released bypass lists to identify the current storage version corresponding to the officially released bypass list.

[0239] The pre-computation result reader retrieves the official release bypass list based on the bypass list object key corresponding to the pre-computation task. When the completion status of the retrieved official release bypass list is "complete," the pre-computation result reader accesses the corresponding immutable result object based on one or more result object keys recorded therein and performs availability checks based on the result verification information recorded in the bypass list. After one or more immutable result objects referenced by the official release bypass list have passed availability checks, the pre-computation result reader reads and uses the corresponding pre-computation result.

[0240] If the bypass list is not found, the completion status of the bypass list is not indicated as complete, or any necessary immutable result object referenced by the bypass list fails the availability check, the pre-computed result reader will not use the corresponding pre-computed result as the official release result, and may trigger the task recovery, re-claiming, or recalculation process.

[0241] Immutable result objects that have been written by worker nodes but not referenced in the officially published bypass list can exist in object storage. For example, immutable result objects generated by worker nodes whose conditional commits failed can still be retained in object storage. The pre-computation result reader does not determine whether the pre-computation task has been completed based on whether a certain immutable result object can be found through object list operations, whether a certain immutable result object can be read individually, or whether the corresponding object is recorded in a certain bucket index.

[0242] Therefore, the completion of the pre-computation task and the formal availability of the corresponding pre-computation results are determined by the availability verification of the formally released bypass list with a completion status of "complete," rather than by the existence of immutable result objects in object storage that are not effectively referenced by the formally released bypass list. The formally released bypass list is stored separately from the original data objects, so it is not necessary to modify or overwrite the original robot data when releasing the pre-computation results.

[0243] In some implementations, based on the bucket implementation, during task recovery, the bypass lists in multiple storage buckets can be scanned in parallel to obtain the pre-computation task set corresponding to the bypass list with a completion status of "complete". Based on the difference between the expected pre-computation task set and the pre-computation task set with a completion status of "complete", the incomplete pre-computation tasks can be determined.

[0244] Task recovery refers to re-determining the completion status of each pre-calculation task and identifying those that need to be continued or reprocessed when a worker node exits abnormally, the scheduling service restarts, the batch pre-calculation process is interrupted, or the completion status of batch tasks needs to be checked. The expected pre-calculation task set is a collection of all pre-calculation tasks expected to be completed, determined based on the robot dataset to be processed, robot task trajectory data units, temporal segmentation rules, and configuration information. Each pre-calculation task in the expected pre-calculation task set can be identified using a corresponding task key.

[0245] In this embodiment, multiple storage buckets can be assigned to multiple recovery worker nodes or multiple scanning threads, allowing each recovery worker node or scanning thread to scan one or more storage buckets. When scanning a storage bucket, the bypass list object keys in the corresponding storage bucket can be listed first, and then one or more of the following can be read: completion status, task key, configuration fingerprint, and pre-computed result reference information for each bypass list.

[0246] When the bypass list is in a "complete" state, the corresponding pre-computed task can be added to the pre-computed task set with a "complete" state based on the task key recorded in the bypass list. If multiple records with the same task key are found, the scan results can be deduplicated according to the task key. In some implementations, usability checks can be performed on immutable result objects referenced by bypass lists with a "complete" state, and only pre-computed tasks whose referenced results pass the usability check can be added to the pre-computed task set with a "complete" state.

[0247] After scanning multiple storage buckets, the task keys of each pre-computation task in the expected pre-computation task set can be compared with the task keys in the pre-computation task set with a completion status of "complete". Pre-computation tasks that exist in the expected pre-computation task set but not in the pre-computation task set with a completion status of "complete" are identified as incomplete pre-computation tasks.

[0248] Pre-computation tasks that do not have a corresponding bypass list, whose completion status is not indicated as "complete" in the found bypass list, or whose completion status is "complete" but whose referenced immutable result objects have failed availability checks, can all be identified as pre-computation tasks that require further inspection or reprocessing. Pre-computation tasks identified as incomplete can be added back to the pre-computation task list so that worker nodes can re-execute the query, claim, compute, and publish processes according to steps S203 to S209.

[0249] If a worker node is still submitting a bypass list during a task recovery scan, the corresponding bypass list can be queried again before adding the pre-computed task back to the pre-computed task list. This avoids incorrectly rescheduling pre-computed tasks that were completed during the scan as incomplete tasks. Even if the same pre-computed task is rescheduled, subsequent task claiming and conditional submission processes can still avoid unconditionally overwriting officially published pre-computed results based on the task claiming object key and the bypass list object key.

[0250] In some implementations, bucketed indexes or completion summaries can be used to reduce the number of bypass lists that need to be scanned. Bucketed indexes or completion summaries are used to improve the efficiency of task recovery scans, but are not the sole basis for determining whether a single pre-computation task is complete; whether a single pre-computation task is complete is still determined by the completion status of the corresponding officially released bypass list and the availability verification results of its referenced results.

[0251] For example, if the expected pre-computation task set includes 10,000 task keys, and after scanning multiple storage buckets in parallel, 9,800 task keys with a completion status of "complete" and whose reference results pass the availability check are obtained, then based on the difference between the two task sets, the pre-computation tasks corresponding to the remaining 200 task keys can be identified as incomplete pre-computation tasks.

[0252] In some implementations, immutable result objects that are not referenced by any officially released bypass list can also be periodically identified. For immutable result objects that have exceeded a preset retention time, whose corresponding task claim leases have expired, and for which there is no active release process, deferred reclamation can be performed. Deferred reclamation is used to release storage space and does not participate in the correctness judgment of officially released results.

[0253] The following section further explains the above method in conjunction with the complete processing procedure. Taking robot task trajectory data unit 1 in dataset F as an example, the task generation component generates the corresponding pre-calculated task. Here, robot task trajectory data unit 1 can refer to any robot task trajectory data unit in dataset F; the value 1 is only for illustrative purposes and does not constitute a limitation on the robot task trajectory data unit. Work node A sequentially executes steps S201 to S209: determining the task key and corresponding object key, querying the bypass list, obtaining processing eligibility and initial scheduling generation through conditional creation, performing pre-calculation and writing the immutable result object, generating the bypass list to be submitted, rereading the task claiming object for consistency verification, and performing conditional writing on the bypass list. After successful conditional writing, the bypass list becomes the officially released bypass list, and its referenced result becomes the officially released result of robot task trajectory data unit 1 under the corresponding configuration.

[0254] In the example of concurrent execution of different pre-computation tasks, worker node A processes robot task trajectory data unit 1, and worker node B processes robot task trajectory data unit 2. Robot task trajectory data unit 2 can refer to any robot task trajectory data unit in dataset F other than robot task trajectory data unit 1. The value 2 is for illustrative purposes only and does not constitute a limitation on the robot task trajectory data unit. The two pre-computation tasks have different task keys, task assignment object keys, result object keys, and bypass list object keys. The two worker nodes can perform pre-computation and conditional commit separately without needing to jointly read and modify a centralized task progress object.

[0255] In an example where the same precomputation task is repeatedly scheduled, worker node A acquires scheduling generation 5 and begins precomputation. After a network outage and lease expiration for worker node A, worker node B acquires scheduling generation 6 through a conditionally protected reclaim operation and successfully publishes the results. Upon recovery, worker node A cannot publish a bypass list because its claimed scheduling generation 5 is inconsistent with the current scheduling generation 6.

[0256] In the idempotent retry example, a network interruption occurs after a worker node completes the upload of an immutable result object. Upon recovery, the worker node first queries the bypass list again; if a complete bypass list already exists and its referenced result passes validation, the task is considered complete; if the bypass list has not yet been committed, but an immutable result object in the same claimed scheduling generation exists and passes validation, the result object is reused and consistency checks and conditional commits continue.

[0257] This application defines a task key, a task claiming object key, and a bypass list object key for a pre-computation task. When the bypass list is not found or its completion status is not indicated as "complete," a conditional operation is performed on the task claiming object key. This allows the worker node that successfully executes the conditional operation to acquire the processing qualification for the pre-computation task and obtain a claiming scheduling generation. This enables coordination of task processing qualifications among multiple worker nodes executing concurrently or sequentially taking over the same pre-computation task. The worker node that acquires processing qualification writes the pre-computation results to one or more immutable result objects. After the immutable result objects are written, a pending bypass list with a completion status of "complete" is generated. This aligns the writing status of the pre-computation results with the task completion status, reducing the overlap of pre-computation results and preventing the reader from accessing incomplete tasks based on their completion status. The process involves writing pre-computation results; before submitting the pending bypass list, a consistency check is performed between the claimed scheduling generation and the current scheduling generation recorded by the task claiming object. If they match, a conditional commit is performed on the pending bypass list, preventing work nodes whose claimed scheduling generations have expired from submitting their generated pending bypass lists and reducing the possibility of multiple work nodes concurrently committing, which could lead to the overwriting of task completion status or pre-computation result reference information. After successful conditional commit, the successfully submitted bypass list is used as the official release bypass list for the pre-computation task, and the pre-computation results referenced in the official release bypass list are determined as the official release results of the pre-computation task. This enables coordination of multiple work nodes' processing and result submission for the same pre-computation task, preventing expired results from being submitted or the same pre-computation task from being repeatedly released, and confirming the currently valid pre-computation results.

[0258] refer to Figure 7 , Figure 7 A schematic diagram of one embodiment of the robot dataset distributed pre-computation coordination device according to this application is shown.

[0259] like Figure 7 As shown, the robot dataset distributed pre-computation coordination device 700 may include a task acquisition module 701, an object key determination module 702, a bypass list query module 703, a task assignment module 704, a pre-computation execution module 705, a bypass list generation module 706, a consistency verification module 707, a conditional submission module 708, and a result publishing module 709.

[0260] The task acquisition module 701 is used to acquire pre-calculated tasks for robot task trajectory data units.

[0261] The object key determination module 702 is used to determine the task key, the task claiming object key, and the bypass list object key based on the pre-computed task. The task key is used to identify the pre-computed task, the task claiming object key is used to determine the task claiming object that records the current scheduling generation, and the bypass list object key is used to determine the bypass list that records the reference information and completion status of the pre-computed result.

[0262] The bypass list query module 703 is used to query the bypass list corresponding to the bypass list object key.

[0263] The task assignment module 704 is used to perform a conditional operation on the task assignment object key if no bypass list is found, or if a bypass list is found but the completion status of the bypass list is not indicated as complete. This operation enables the work node that successfully performs the conditional operation to obtain the eligibility to process the pre-computed task and to obtain the assignment scheduling generation.

[0264] The pre-computation execution module 705 is used by the qualified work nodes to execute pre-computation tasks and obtain the pre-computation results of the pre-computation tasks.

[0265] The module 706 for generating a list of bypasses to be committed is used to write pre-computed results into object storage to form one or more immutable result objects, and to generate a list of bypasses to be committed that references one or more immutable result objects and has a completion status of complete.

[0266] The consistency verification module 707 is used to perform consistency verification between the claimed scheduling generation and the current scheduling generation recorded by the task claiming object.

[0267] The conditional commit module 708 is used to perform a conditional commit on the list of bypasses to be committed when the consistency check passes.

[0268] The result publishing module 709 is used to, when the conditions are successfully submitted, use the successfully submitted bypass list as the official publishing bypass list of the pre-computation task, and determine the pre-computation results referenced in the official publishing bypass list as the official publishing results of the pre-computation task.

[0269] In some implementations, the object key determination module 702 is further configured to: Obtain the dataset identifier, robot task trajectory data unit identifier, optional time segment identifier or target time range, and configuration information corresponding to the pre-calculation task.

[0270] The task key is determined based on the dataset identifier, robot task trajectory data unit identifier, optional time sequence segment identifier or target time range, and configuration information.

[0271] The task claiming object key and the bypass list object key are determined based on the task key and the preset mapping relationship, respectively. The preset mapping relationship is used to characterize the deterministic correspondence between the task key of the same pre-computation task and the corresponding task claiming object key and bypass list object key.

[0272] In some implementations, the object key determination module 702 is further configured to: The configuration information is deterministically serialized to obtain a configuration byte sequence.

[0273] Perform a digest operation on the configuration byte sequence to obtain the configuration fingerprint.

[0274] The task key is determined based on the dataset identifier, robot task trajectory data unit identifier, optional time sequence segment identifier or target time range, and configured fingerprint.

[0275] In some implementations, when no task claim object corresponding to the task claim object key exists in the object storage, the task claim module 704 is further configured to: If there is no task claim object identified by the task claim object key in the object storage, a conditional creation operation is performed based on the task claim object key to create a task claim object. The created task claim object records the worker node identifier of the worker node that successfully created the task claim object, the initial scheduling generation, the claim time, and the lease expiration time. The claim scheduling generation is the same as the initial scheduling generation.

[0276] In some implementations, when a task claiming object corresponding to the task claiming object key already exists in the object storage, the task claiming module 704 is further configured to: Read the task claiming object and its object metadata, obtain the current scheduling generation and lease expiration time from the task claiming object, and obtain the entity tag or object version identifier returned by the object storage from the object metadata. The entity tag or object version identifier is used to identify the storage version of the task claiming object at the time of this read.

[0277] Compare the current time with the lease expiry time, and determine that the lease for the task assignment object has expired if the current time is at or after the lease expiry time.

[0278] Determine the target scheduling generation that is higher than the current scheduling generation based on the current scheduling generation.

[0279] If the current entity label or object version identifier of the task claiming object is consistent with the entity label or object version identifier obtained during this read, perform a condition update on the task claiming object. If the condition update is successful, update the current scheduling generation recorded in the task claiming object to the target scheduling generation, update the work node identifier recorded in the task claiming object to the work node identifier of the work node that reclaims the pre-computation task, and update the lease expiration time recorded in the task claiming object to the new lease expiration time. Here, the claimed scheduling generation is the target scheduling generation.

[0280] In some implementations, the bypass list query module 703 is further used for: When a bypass list with a completion status of "complete" is found, the corresponding one or more immutable result objects are accessed based on the object key of one or more immutable result objects recorded in the bypass list.

[0281] Availability verification is performed on one or more immutable result objects, wherein the availability verification includes determining that one or more immutable result objects exist, and determining that one or more of the following information of the one or more immutable result objects—actual byte count, content digest, output shape, data type, and original data corresponding range—are consistent with the corresponding information recorded in the bypass list.

[0282] When the availability check passes, the pre-computed result referenced by the bypass list is determined as the pre-computed result of the pre-computed task, and no further conditional operations are performed on the task claim object key.

[0283] In some implementations, the bypass list generation module 706 is further used for: Based on the task key and the claimed scheduling generation, determine the object key of one or more immutable result objects; Using the object key of one or more immutable result objects as storage identifiers, pre-computed results are written to object storage to form one or more immutable result objects. The immutable result objects are those that are not modified or overwritten in place after being written.

[0284] Verify one or more of the following for each of the one or more immutable result objects written: byte count, content digest, output shape, data type, and corresponding range of the original data.

[0285] In some implementations, the bypass list generation module 706 is further used for: When one or more immutable result objects pass the verification, write one or more of the following into the bypass list: the object key and claim scheduling generation of each immutable result object, as well as the number of bytes, content digest, output shape, data type and corresponding range of the original data. Set the completion status of the bypass list to complete, and generate a pending bypass list that references one or more immutable result objects and has a completion status of complete.

[0286] In some implementations, the condition submission module 708 is further configured to: If no bypass list is found, perform conditional write to the bypass list to be submitted, assuming that no bypass list identified by the bypass list object key exists in the object storage. Alternatively, When the bypass list is retrieved, the entity label or object version identifier returned by the bypass list is obtained as the object storage. The condition for updating the bypass list to be submitted is to ensure that the current entity label or object version identifier of the bypass list is consistent with the entity label or object version identifier obtained during the query.

[0287] If the condition write or condition update fails, reread the task claim object and re-query the bypass list; If the completed status of the bypass list retrieved again is "complete" and the pre-calculated results referenced by the bypass list pass the verification, then abandon the submission of the bypass list generated by the current working node.

[0288] When the claimed scheduling generation matches the current scheduling generation of the reread task claiming object record, the conditional submission is re-executed based on the results of the re-query of the bypass list. Specifically, if the bypass list is not found, the conditional write is re-executed based on the condition that there is no bypass list identified by the bypass list object key in the object storage. If a bypass list with a completion status not indicated as complete is found, the conditional update is re-executed based on the entity tag or object version identifier obtained when re-querying the bypass list.

[0289] In some implementations, when the robot task trajectory data unit includes multiple time-series segments, the pre-calculation execution module 705 is further used for: The work nodes that have obtained processing qualifications will determine each time segment as a pre-computation subtask of the pre-computation task, and will determine the time segment-level task claiming object key and the corresponding immutable result object object key for each pre-computation subtask.

[0290] Conditional operations are performed on the task claiming object keys for each time segment level, so that the work nodes that successfully execute the corresponding conditional operations are qualified to process the corresponding pre-computation subtasks, and the multiple work nodes that have obtained the processing qualifications for different pre-computation subtasks execute their respective pre-computation subtasks in parallel.

[0291] After a work node that has obtained the qualification to process the corresponding pre-computation subtask completes the corresponding pre-computation subtask, it writes the pre-computation results corresponding to each pre-computation subtask into an immutable result object identified by the object key of the corresponding immutable result object.

[0292] In some implementations, the bypass list corresponding to the robot task trajectory data unit records the total number of time segments, the reference information of the immutable result objects corresponding to each time segment, and the time segment completion set. The device 700 also includes a bypass list update module 710, which is used to read the bypass list corresponding to the robot task trajectory data unit after the corresponding pre-calculated subtask is completed at the work node, and obtain the entity tag or object version identifier corresponding to the bypass list returned by the object storage; according to the time segments completed by the work node and their corresponding immutable result objects, merge and update the reference information of the immutable result objects corresponding to the time segments and the time segment completion set in the bypass list; when the merged and updated time segment completion set indicates that all time segments have been completed, set the completion status of the bypass list to complete; perform a conditional update on the merged and updated bypass list based on the condition that the current entity tag or object version identifier of the bypass list is consistent with the entity tag or object version identifier obtained during this reading; if the conditional update fails, reread the bypass list, and perform the merge update and conditional update again based on the reread bypass list and the re-obtained entity tag or object version identifier.

[0293] In some implementations, the object key determination module 702 is further configured to: According to the preset bucketing rules, the target storage bucket is determined from multiple storage buckets based on the task key, and the bypass list object key is determined based on the target storage bucket and the task key.

[0294] During task recovery, the bypass lists in multiple storage buckets are scanned in parallel to obtain the pre-computation task set corresponding to the bypass list with a completion status of "complete". Based on the difference between the expected pre-computation task set and the pre-computation task set with a completion status of "complete", the incomplete pre-computation tasks are determined.

[0295] The specific processing of each module, the meaning of related terms, optional implementation methods, and their technical effects in the device embodiments can be found in the descriptions of the corresponding steps in the foregoing method embodiments, and will not be repeated here. The above modules can be implemented by software, hardware, firmware, or a combination thereof, and can also be combined or further split according to the deployment method of computing and storage resources.

[0296] The following is for reference. Figure 8 , Figure 8 A schematic diagram of an electronic device structure suitable for implementing embodiments of this application is shown.

[0297] Figure 8 The electronic device 800 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.

[0298] An electronic device comprising one or more processors.

[0299] Specifically, one or more processors can correspond to Figure 8 The processing device 801 shown may include a central processing unit, a graphics processing unit, an artificial intelligence accelerator, a tensor processor, or other general-purpose or special-purpose processors. Different processors may respectively perform at least one of the following processes: pre-computation task acquisition, object key determination, task assignment, pre-computation task execution, writing immutable result objects, result verification, bypass list submission, and task resumption.

[0300] Electronic devices also include storage devices on which one or more programs are stored.

[0301] Specifically, the storage device may include at least one of read-only memory (ROM) 802, random access memory (RAM) 803, and storage device 808. ROM 802 may store a boot program or a firmware program; RAM 803 may store programs and data used during the operation of the electronic device; storage device 808 may include a disk, solid-state drive, flash memory device, or other non-volatile storage device, and may store one or more programs for implementing a distributed pre-computation coordination method for robot datasets.

[0302] When one or more programs are executed by one or more processors, the one or more processors implement any of the foregoing method embodiments.

[0303] Specifically, one or more programs may include program instructions for acquiring a pre-computed task for a robot task trajectory data unit, program instructions for determining a task key, a task claiming object key, and a bypass list object key based on the pre-computed task, program instructions for querying the bypass list, program instructions for obtaining pre-computed task processing eligibility and claiming a scheduling generation through conditional operations, program instructions for executing the pre-computed task and writing one or more immutable result objects, program instructions for generating a bypass list to be submitted, program instructions for performing scheduling generation consistency checks, program instructions for performing conditional submission on the bypass list to be submitted, and program instructions for determining the official release result.

[0304] One or more programs may further include program instructions for deterministically serializing configuration information and generating configuration fingerprints, program instructions for updating task claiming objects based on entity tags or object version identifiers, program instructions for performing availability checks on immutable result objects, program instructions for executing pre-computation subtasks corresponding to multiple time-series segments in parallel, program instructions for merging and updating the time-series segment completion set, and program instructions for scanning multiple storage buckets in parallel and determining unfinished pre-computation tasks.

[0305] like Figure 8 As shown, the processing device 801, ROM 802, and RAM 803 can be connected to each other via bus 804. Input / output (I / O) interface 805 can also be connected to bus 804. When the electronic device 800 is running, the processing device 801 can execute the processing corresponding to the distributed pre-computation coordination method for the robot dataset according to the program stored in ROM 802 or according to the program loaded from storage device 808 into RAM 803.

[0306] I / O interface 805 can connect to input device 806, output device 807, storage device 808, and communication device 809. Input device 806 may include a keyboard, mouse, touch device, or other devices for inputting device operating parameters, pre-calculated configuration information, and task recovery parameters. Output device 807 may include a display or other devices for outputting pre-calculated task status, worker node status, bypass list status, and task recovery results.

[0307] The communication device 809 enables the electronic device 800 to communicate wired or wirelessly with scheduling services, worker nodes, object storage services, raw data storage nodes, or other electronic devices. The electronic device 800 can receive pre-calculated tasks through the communication device 809, read robot task trajectory data units, task assignment objects, and bypass lists, write immutable result objects to the object storage, and perform condition creation or condition update operations on the task assignment objects and bypass lists.

[0308] In some implementations, the processing device 801 may include multiple processors. One processor may perform pre-computation task acquisition, object key determination, and task scheduling processing; another processor may perform robot task trajectory data decoding, temporal sampling, feature extraction, or latent variable generation; and other processors may perform object storage access, result verification, and bypass list submission. Multiple processors may share RAM 803 or each have its own device memory, and may transmit pre-computation tasks, raw data, and pre-computation results via high-speed interconnect.

[0309] In some implementations, the electronic device 800 can be a single worker node, a single coordination server, or a single object storage access node, or it can be a distributed electronic device composed of multiple computing nodes. When the electronic device 800 is a distributed electronic device, the task acquisition module 701, object key determination module 702, bypass list query module 703, task assignment module 704, pre-computation execution module 705, bypass list generation module 706, consistency verification module 707, conditional submission module 708, result publishing module 709, and bypass list update module 710 can be deployed on the same computing node or on different computing nodes that can communicate with each other.

[0310] Not all components in electronic device 800 are required to conform to Figure 8 The setup is as shown. Depending on the specific implementation requirements, the electronic device 800 may omit some input or output devices, or it may add a graphics processor, artificial intelligence accelerator, direct memory access controller, high-speed network interface, object storage access accelerator, or other components suitable for performing robot data pre-computation and distributed coordination processing.

[0311] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the aforementioned method embodiments.

[0312] Specifically, a computer-readable storage medium can be a tangible medium capable of containing or storing computer programs. Computer-readable storage media can include portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, flash memory, solid-state drives, optical storage devices, magnetic storage devices, or any combination of the foregoing.

[0313] The computer-readable storage medium may be disposed within the aforementioned electronic device 800 or may be disposed independently of the aforementioned electronic device 800. The computer program stored in the computer-readable storage medium may include program instructions for causing the processor to perform pre-computation task acquisition, object key determination, bypass list lookup, task condition claiming, pre-computation result generation, immutable result object writing, scheduling generation consistency verification, bypass list condition commit, and formal result publication.

[0314] The computer program may also include program instructions for performing configuration fingerprint generation, expired lease reclaiming, immutable result object availability verification, parallel processing of time-series segmented pre-computation subtasks, time-series segmented completion set merging and updating, storage bucket determination, and incomplete task recovery scanning.

[0315] When a computer program is executed by one or more processors, the one or more processors may perform all or some of the steps in the foregoing method embodiments. When executed jointly by multiple processors, different processors may respectively perform different processes in task scheduling, conditional assignment, robot data pre-computation, object storage read / write, bypass list conditional submission, and task recovery.

[0316] A computer program product includes a computer program or computer instructions, which, when executed by a processor, implement any of the aforementioned method embodiments.

[0317] Specifically, the computer program product may include a computer program or computer instructions for implementing a distributed pre-computation coordination method for robot datasets. The computer program or computer instructions may be downloaded from a network and installed on the electronic device 800, or loaded onto the electronic device 800 from a computer-readable storage medium. When executed by the processor, the computer program or computer instructions cause the electronic device 800 to acquire the pre-computation task, determine the task key, the task claiming object key, and the bypass list object key, obtain processing eligibility through conditional operations, execute the pre-computation task and write it to an immutable result object, perform consistency verification on the claimed scheduling generation, publish the bypass list through conditional submission, and determine the final published result.

[0318] Computer programs or instructions can be written in one or more programming languages ​​and can be executed entirely on a single electronic device or separately on scheduling nodes, worker nodes, pre-computation nodes, object storage access nodes, and task recovery nodes. The program portions executed on different nodes can exchange pre-computation tasks, task keys, task claiming objects, immutable result objects, bypass lists, entity tags, object version identifiers, and task recovery information via wired or wireless networks.

[0319] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, electronic devices, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function.

[0320] It should be noted that in some alternative implementations, the functions marked in the boxes may be executed in a different order than that shown in the accompanying drawings. For example, two consecutively shown boxes may be executed substantially in parallel, or they may be executed in reverse order depending on the functions involved. Each box and combination of boxes in the block diagram or flowchart may be implemented by a dedicated hardware system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions.

[0321] The modules involved in the embodiments of this application can be implemented in software, in hardware, or in a combination of both. The names of the modules do not necessarily limit the modules themselves in certain circumstances. For example, the object key determination module 702 can also be described as a module for determining task keys, task claiming object keys, and bypass list object keys based on pre-computed tasks.

[0322] The above description is merely a partial embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of protection of this application is not limited to technical solutions constituted by specific combinations of the aforementioned technical features, but should also cover other technical solutions formed by combining the aforementioned technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the aforementioned features with technical features having similar functions disclosed in this application are also within the scope of this application.

Claims

1. A distributed pre-computation coordination method for robot datasets, characterized in that, The method includes: Acquire pre-calculated tasks for robot task trajectory data units; Based on the pre-computation task, a task key, a task claiming object key, and a bypass list object key are determined, wherein the task key is used to identify the pre-computation task, the task claiming object key is used to determine the task claiming object that records the current scheduling generation, and the bypass list object key is used to determine the bypass list that records the reference information and completion status of the pre-computation results. Query the bypass list corresponding to the bypass list object key; If the bypass list is not found, or if the bypass list is found but the completion status of the bypass list is not indicated as complete, a conditional operation is performed on the task claiming object key, so that the work node that successfully performs the conditional operation obtains the qualification to process the pre-computed task and obtains the claiming scheduling generation. The pre-computation task is executed by the work node that has obtained the processing qualification, and the pre-computation result of the pre-computation task is obtained; The pre-calculated results are written to object storage to form one or more immutable result objects, and a list of pending bypasses that reference the one or more immutable result objects and have a completion status of complete is generated. Perform a consistency check between the claimed scheduling generation and the current scheduling generation recorded by the task claiming object; When the consistency check passes, a conditional commit is performed on the list of bypasses to be submitted; When the conditions are successfully submitted, the successfully submitted bypass list is used as the official release bypass list of the pre-computation task, and the pre-computation results referenced by the official release bypass list are determined as the official release results of the pre-computation task.

2. The method according to claim 1, characterized in that, The step of determining the task key, task claiming object key, and bypass list object key based on the pre-calculated task includes: Obtain the dataset identifier, robot task trajectory data unit identifier, optional time segment identifier or target time range, and configuration information corresponding to the pre-calculation task; The task key is determined based on the dataset identifier, the robot task trajectory data unit identifier, the optional time segment identifier or the target time range, and the configuration information. The task claiming object key and the bypass list object key are determined according to the task key and the preset mapping relationship, respectively. The preset mapping relationship is used to characterize the deterministic correspondence between the task key of the same pre-computation task and the corresponding task claiming object key and bypass list object key.

3. The method according to claim 2, characterized in that, The step of determining the task key based on the dataset identifier, the robot task trajectory data unit identifier, the optional time segment identifier or the target time range, and the configuration information includes: The configuration information is deterministically serialized to obtain a configuration byte sequence; Perform a digest operation on the configuration byte sequence to obtain the configuration fingerprint; The task key is determined based on the dataset identifier, the robot task trajectory data unit identifier, the optional time segment identifier or the target time range, and the configuration fingerprint.

4. The method according to claim 1, characterized in that, When the task assignment object corresponding to the task assignment object key does not exist in the object storage, the conditional operation performed on the task assignment object key includes: If there is no task claim object identified by the task claim object key in the object storage, a conditional creation operation is performed based on the task claim object key to create the task claim object. The created task claim object records the worker node identifier, initial scheduling generation, claim time and lease expiration time of the worker node that successfully created the task claim object. The claim scheduling generation is the initial scheduling generation.

5. The method according to claim 1, characterized in that, When a task claiming object corresponding to the task claiming object key already exists in the object storage, the conditional operation performed on the task claiming object key includes: Read the task claiming object and its object metadata, obtain the current scheduling generation and lease expiration time from the task claiming object, and obtain the entity tag or object version identifier returned by the object storage from the object metadata, wherein the entity tag or the object version identifier is used to identify the storage version of the task claiming object at the time of this read; The current time is compared with the lease expiration time, and if the current time is equal to or later than the lease expiration time, it is determined that the lease corresponding to the task claiming object has expired; Determine a target scheduling generation higher than the current scheduling generation based on the current scheduling generation; If the current entity tag or object version identifier of the task claiming object is consistent with the entity tag or object version identifier obtained during this reading, a condition update is performed on the task claiming object. When the condition update is successful, the current scheduling generation recorded by the task claiming object is updated to the target scheduling generation, the work node identifier recorded by the task claiming object is updated to the work node identifier of the work node that reclaims the pre-computation task, and the lease expiration time recorded by the task claiming object is updated to the new lease expiration time, wherein the claimed scheduling generation is the target scheduling generation.

6. The method according to claim 1, characterized in that, After querying the bypass list corresponding to the bypass list object key, the method further includes: When a bypass list with a completion status of "complete" is found, the corresponding one or more immutable result objects are accessed based on the object key of one or more immutable result objects recorded in the bypass list. Availability verification is performed on the one or more immutable result objects, wherein the availability verification includes determining that all one or more immutable result objects exist, and determining that one or more of the actual byte count, content digest, output shape, data type, and original data corresponding range of each of the one or more immutable result objects are consistent with the corresponding information recorded in the bypass list; When the availability check passes, the pre-calculation result referenced by the bypass list is determined as the pre-calculation result of the pre-calculation task, and the conditional operation is no longer performed on the task claiming object key.

7. The method according to claim 1, characterized in that, The pre-calculated results are written to object storage to form one or more immutable result objects, including: Based on the task key and the claimed scheduling generation, determine the object key of each of the one or more immutable result objects; Using the object key of each of the one or more immutable result objects as a storage identifier, the pre-computed result is written into the object storage to form the one or more immutable result objects, wherein the immutable result objects are result objects that are not modified or overwritten in place after being written; The method further includes: validating one or more of the following for each of the one or more immutable result objects written: byte count, content digest, output shape, data type, and corresponding range of the original data.

8. The method according to claim 7, characterized in that, The generation of the list of pending bypasses that references the one or more immutable result objects and has a completion status of "complete" includes: When all one or more immutable result objects pass the verification, the object key and the claim scheduling generation of each of the one or more immutable result objects, as well as one or more of their byte count, content digest, output shape, data type, and original data corresponding range, are written into the bypass list, and the completion status of the bypass list is set to complete, generating a pending bypass list that references the one or more immutable result objects and has a completion status of complete.

9. The method according to claim 1, characterized in that, The conditional commit to the list of bypasses to be committed includes: If the bypass list is not found, conditional write is performed on the bypass list to be submitted, based on the condition that no bypass list identified by the bypass list object key exists in the object storage; or... When the bypass list is found, the object is stored as the entity tag or object version identifier returned by the bypass list, and the condition is updated on the bypass list to be submitted, based on the condition that the current entity tag or object version identifier of the bypass list is consistent with the entity tag or object version identifier obtained during the query. If the condition writing or condition update fails, the task claiming object is read again, and the bypass list is queried again. If the completion status of the bypass list retrieved again is complete and the pre-calculation result referenced by the bypass list passes the verification, the submission of the bypass list to be submitted generated by the current working node is abandoned. When the claimed scheduling generation matches the current scheduling generation of the reread task claiming object record, the conditional submission is re-executed based on the result of the re-query of the bypass list. Specifically, if the bypass list is not found, the conditional write is re-executed on the condition that there is no bypass list identified by the bypass list object key in the object storage. If a bypass list with a completion status not indicated as complete is found, the conditional update is re-executed based on the entity tag or object version identifier obtained when re-querying the bypass list.

10. The method according to claim 1, characterized in that, When the robot task trajectory data unit includes multiple time-series segments, the execution of the pre-calculation task by the worker node that has obtained the processing qualification includes: The working node that has obtained the processing qualification determines each of the time segments as a pre-computation subtask of the pre-computation task, and determines the time segment-level task claiming object key and the corresponding immutable result object object key for each of the pre-computation subtasks. Conditional operations are performed on each of the time-series segmented task claiming object keys, so that the work nodes that successfully execute the corresponding conditional operations acquire the processing qualification for the corresponding pre-computation sub-tasks, and the multiple work nodes that have acquired the processing qualification for different pre-computation sub-tasks execute their respective pre-computation sub-tasks in parallel. After a work node that has obtained the qualification to process the corresponding pre-computation subtask completes the corresponding pre-computation subtask, the pre-computation result corresponding to each pre-computation subtask is written into an immutable result object identified by the object key of the corresponding immutable result object.

11. The method according to claim 10, characterized in that, The bypass list corresponding to the robot task trajectory data unit records the total number of time-series segments, the reference information of the immutable result objects corresponding to each time-series segment, and the time-series segment completion set. The method further includes: After the corresponding pre-calculated subtask is completed at the work node, the bypass list corresponding to the robot task trajectory data unit is read, and the entity tag or object version identifier corresponding to the bypass list returned by the object storage is obtained. Based on the time segment completed by the work node and its corresponding immutable result object, merge and update the reference information of the immutable result object corresponding to the time segment in the bypass list and the time segment completion set. When the merged and updated time series segment completion set indicates that all time series segments have been completed, the completion status of the bypass list is set to complete; The conditional update is performed on the merged and updated bypass list, based on the condition that the current entity tag or object version identifier of the bypass list is consistent with the entity tag or object version identifier obtained during this reading. If the conditional update fails, the bypass list is reread, and based on the reread bypass list and the newly obtained entity tag or object version identifier, the merge update and the conditional update are executed again.

12. The method according to claim 1, characterized in that, Determining the bypass list object key based on the pre-computation task includes: According to the preset bucketing rules, the target storage bucket is determined from multiple storage buckets based on the task key, and the bypass list object key is determined based on the target storage bucket and the task key; The method further includes: During task recovery, the bypass lists in the multiple storage buckets are scanned in parallel to obtain the pre-computation task set corresponding to the bypass list with a completion status of "complete". Based on the difference between the expected pre-computation task set and the pre-computation task set with a completion status of "complete", the incomplete pre-computation tasks are determined.

13. A distributed pre-computation coordination device for robot datasets, characterized in that, The device includes: The task acquisition module is used to acquire pre-calculated tasks for robot task trajectory data units; The object key determination module is used to determine a task key, a task claiming object key, and a bypass list object key based on the pre-computation task. The task key is used to identify the pre-computation task, the task claiming object key is used to determine the task claiming object that records the current scheduling generation, and the bypass list object key is used to determine the bypass list that records the reference information and completion status of the pre-computation results. The bypass list query module is used to query the bypass list corresponding to the bypass list object key; The task assignment module is used to perform a conditional operation on the task assignment object key if the bypass list is not found, or if the bypass list is found but the completion status of the bypass list is not indicated as complete. This operation enables the work node that successfully executes the conditional operation to obtain the qualification to process the pre-computed task and to obtain the assignment scheduling generation. A pre-computation execution module is used to execute the pre-computation task by the work node that has obtained the processing qualification, and to obtain the pre-computation result of the pre-computation task; The module for generating a list of bypasses to be submitted is used to write the pre-calculated results into object storage to form one or more immutable result objects, and to generate a list of bypasses to be submitted that references the one or more immutable result objects and has a completion status of complete. The consistency verification module is used to perform consistency verification between the claimed scheduling generation and the current scheduling generation recorded by the task claiming object; The conditional commit module is used to perform a conditional commit on the list of bypasses to be committed when the consistency check passes. The result publishing module is used to, when the conditions are successfully submitted, use the successfully submitted bypass list as the official publishing bypass list of the pre-computation task, and determine the pre-computation results referenced by the official publishing bypass list as the official publishing results of the pre-computation task.

14. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-12.

16. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-12.

Images