Branch conflict detection method, apparatus, device and medium for feature data table

Abstract

This disclosure provides a method, apparatus, device, and medium for branch conflict detection of feature data tables. This disclosure relates to the field of computer technology, particularly to the fields of databases, data management, and artificial intelligence, and can be used in application scenarios such as generative search, intelligent document editing, intelligent assistants, virtual assistants, and intelligent e-commerce. Write operations on the feature data table generate corresponding snapshots, and in the case of adding or deleting columns, the metadata includes column change information. The method includes: determining the latest common snapshot of a first branch and a second branch of the feature data table, and determining a first change snapshot sequence for the first branch and a second change snapshot sequence for the second branch; based on the column change information included in the first and second change snapshot sequences, determining whether there are conflicting columns in the table structure, where a conflicting column is a column that exists in the table corresponding to the latest common snapshot, exists in one branch, and does not exist in the other branch.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of database, data management and artificial intelligence technology. It can be used in application scenarios such as generative search, intelligent document editing, intelligent assistant, virtual assistant, and intelligent e-commerce. Specifically, it relates to a branch conflict detection method for feature data tables, a branch conflict detection device for feature data tables, electronic devices, computer-readable storage media and computer program products. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include natural language processing, computer vision, speech recognition, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a branch conflict detection method for a feature data table, a branch conflict detection device for a feature data table, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a branch conflict detection method for a feature data table is provided. A snapshot is generated corresponding to a write operation on the feature data table, and when a column of the feature data table is added or deleted in the write operation, the metadata of the generated snapshot includes column change information indicating the column added or deleted in the write operation. The method includes: determining a latest common snapshot of a first branch and a second branch of the feature data table; determining snapshots in the first branch that follow the latest common snapshot as a first change snapshot sequence, and determining snapshots in the second branch that follow the latest common snapshot as a second change snapshot sequence; determining whether there is a table structure conflict column based on the column change information included in each of the first and second change snapshot sequences, wherein a table structure conflict column is a column that exists in the table corresponding to the latest common snapshot, exists in one branch, and does not exist in the other branch; and determining a conflict between the first branch and the second branch in response to determining the existence of a table structure conflict column.

[0006] According to another aspect of this disclosure, a branch conflict detection apparatus for a feature data table is provided. A corresponding snapshot is generated for a write operation on the feature data table, and when a column of the feature data table is added or deleted in the write operation, the metadata of the generated snapshot includes column change information indicating the column added or deleted in the write operation. The apparatus includes: a snapshot determination unit configured to determine the latest common snapshot of a first branch and a second branch of the feature data table; a sequence determination unit configured to determine snapshots in the first branch following the latest common snapshot as a first change snapshot sequence, and snapshots in the second branch following the latest common snapshot as a second change snapshot sequence; a conflict column determination unit configured to determine whether a table structure conflict column exists based on the column change information included in the first change snapshot sequence and the second change snapshot sequence, wherein a table structure conflict column is a column that exists in the table corresponding to the latest common snapshot, exists in one branch, and does not exist in the other branch; and a conflict determination unit configured to determine a conflict between the first branch and the second branch in response to determining the existence of a table structure conflict column.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described above.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described method.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0010] According to one or more embodiments of this disclosure, when a write operation is performed on a feature data table, a corresponding snapshot is generated, and column change information indicating the columns added or deleted by the write operation is recorded in the snapshot's metadata. During conflict detection, conflicts between branches are determined based on the column change information included in the change snapshot sequences of each branch. This eliminates the need to read the feature data table's data file, and the amount of data read is independent of the data file's size, thus shortening detection time and reducing resource consumption. Furthermore, by using the table corresponding to the latest common snapshot as a benchmark to determine conflicting columns in the table structure, conflict detection caused by changes in the table structure is achieved.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown; Figure 2 A flowchart of a branch conflict detection method for a feature data table according to an embodiment of the present disclosure is shown; Figure 3 A flowchart illustrating the process of determining whether a table structure conflict column exists according to an embodiment of the present disclosure is shown; Figure 4 A flowchart of a branch conflict detection method for a feature data table according to another embodiment of the present disclosure is shown; Figure 5 A flowchart illustrating the process for determining whether a column-level conflict column exists according to an embodiment of the present disclosure is shown; Figure 6 A flowchart of a branch conflict detection method for a feature data table according to yet another embodiment of the present disclosure is shown; Figure 7 A flowchart illustrating the process of determining an intersection size estimate according to an embodiment of the present disclosure is shown; Figure 8 A structural block diagram of a branch conflict detection apparatus for a feature data table according to an embodiment of the present disclosure is shown; Figure 9 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0016] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0017] In related technologies, existing conflict detection for feature data tables requires reading the entire data file, resulting in long processing times and high resource consumption. Furthermore, methods based on comparing data values ​​cannot detect conflicts caused by changes in table structure.

[0018] To address the aforementioned issues, this disclosure generates a corresponding snapshot when performing a write operation on the feature data table, and records column change information indicating the columns added or deleted during the write operation in the snapshot's metadata. During conflict detection, conflicts between branches are determined based on the column change information included in the change snapshot sequences of each branch. This eliminates the need to read the feature data table's data file, and the amount of data read is independent of the data file's size, thus shortening detection time and reducing resource consumption. Furthermore, by using the table corresponding to the latest common snapshot as a benchmark to determine conflicting columns in the table structure, conflict detection caused by changes in the table structure is achieved.

[0019] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0020] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0021] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of the methods of this disclosure.

[0022] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0023] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0024] Users can use client devices 101, 102, 103, 104, 105, and / or 106 for human-computer interaction. The client devices provide interfaces that enable users to interact with them. The client devices can also output information to the user through these interfaces. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0025] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0026] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0027] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0028] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0029] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0030] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0031] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0032] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0033] Figure 1The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0034] According to one aspect of this disclosure, a branch conflict detection method for a feature data table is provided. Figure 2 This is a flowchart illustrating a branch conflict detection method 200 for a feature data table according to an exemplary embodiment. Figure 2 As shown, method 200 may include steps S201 to S204.

[0035] A feature data table is a data table used to store feature data, where rows correspond to data entities and columns correspond to features. Feature data tables can be used for accessing features during the training or inference process of machine learning models. For example, a feature data table could be a wide table storing user features, where one column stores the user's click-through rate over a recent period, and another column stores the user's registration duration.

[0036] The feature data table can have multiple branches. Branches provide isolated write environments for the feature data table, and write operations on different branches do not affect each other. Multiple branches can include the main branch and other branches created based on the main branch. The first branch and the second branch can be any two branches that need to be conflict-checked. For example, the first branch is the development branch used by the algorithm team for experiments, and the second branch is the main branch. After the experiments on the development branch are completed, the data on the development branch needs to be merged into the main branch, and conflict detection is performed on the two branches before merging.

[0037] Write operations on the feature data table generate corresponding snapshots. A snapshot is a version record created after a write operation is completed. A snapshot includes metadata, which describes the write operation and the corresponding version information. A snapshot may also include a reference to the corresponding version's data file, which stores the data itself from the feature data table. Therefore, the snapshot's metadata is separate from the data file; reading the snapshot's metadata does not require reading the data file.

[0038] When a write operation adds or deletes columns in a feature data table, the metadata of the generated snapshot includes column change information, indicating the columns added or deleted in the write operation. This column change information can be recorded in the metadata by the write end performing the write operation when generating the snapshot, and is bound to the version corresponding to that write operation. For example, the column change information may include a mapping between the column identifier of the added or deleted column and the change type, where the change type includes addition and deletion.

[0039] For example, if a write operation deletes the column old_score_v1 and adds the column user_7d_ctr, the metadata of the snapshot generated by the write operation will include two entries for column change information. One entry records the column identifier old_score_v1 and the change type as deletion, and the other entry records the column identifier user_7d_ctr and the change type as addition.

[0040] In some embodiments, the feature data table can be implemented based on a columnar storage engine that supports a snapshot mechanism, such as using Apache Iceberg as the storage engine. In this case, column change information can be recorded in the key-value pair dictionary included in the snapshot metadata.

[0041] In step S201, the latest common snapshot of the first branch and the second branch of the feature data table is determined.

[0042] The two branches share a common history. Snapshots generated before the branches were created are shared by both branches, while snapshots generated after the branches developed independently belong to their respective branches. The latest shared snapshot is the most recently generated snapshot among the snapshots shared by the first and second branches; it corresponds to the last shared version before the two branches began their independent changes and can serve as a benchmark for subsequent comparisons.

[0043] In some embodiments, method 200 may be executed in response to a merge request for the first branch and the second branch, or in response to a comparison request for the two branches, or at a predetermined period.

[0044] For example, the set of snapshots contained in the first branch and the set of snapshots contained in the second branch can be obtained respectively, the snapshots common to the two sets can be determined, and the snapshot with the latest generation time among the common snapshots can be determined as the latest common snapshot. In other embodiments, the snapshot corresponding to the creation point of the branch can be recorded when the branch is created, and the latest common snapshot can be determined based on the recorded creation point.

[0045] In step S202, the snapshots in the first branch that are after the latest common snapshot are determined as the first change snapshot sequence, and the snapshots in the second branch that are after the latest common snapshot are determined as the second change snapshot sequence.

[0046] A snapshot following the latest common snapshot refers to a snapshot generated after the latest common snapshot and belonging to the corresponding branch. Since each snapshot corresponds to one write operation, the first change snapshot sequence reflects all write operations that have occurred in the first branch since the latest common snapshot, and the second change snapshot sequence reflects all write operations that have occurred in the second branch since the latest common snapshot. In other words, the two change snapshot sequences each carry change records for the two branches relative to the base version. In some embodiments, the snapshots in the change snapshot sequence can be arranged in the order they were generated.

[0047] In some embodiments, the change snapshot sequence can be empty. For example, if no write operations have occurred on a branch since the latest common snapshot, the change snapshot sequence for that branch is empty, indicating that the branch has not changed relative to the base version. An empty change snapshot sequence does not affect the execution of subsequent steps.

[0048] For ease of description, the first branch will be denoted as A, the second branch as B, the latest common snapshot as S0, the first change snapshot sequence as ΔA, and the second change snapshot sequence as ΔB.

[0049] In step S203, based on the column change information included in the first change snapshot sequence and the second change snapshot sequence, it is determined whether there are any conflicting columns in the table structure. Conflicting columns in the table structure are columns that exist in the table corresponding to the latest common snapshot, exist in one branch, and do not exist in another branch.

[0050] The table corresponding to the latest common snapshot refers to the table in the version corresponding to the latest common snapshot, where the columns that exist are those that were common to both branches before they diverged. If a column meets all three conditions mentioned above, it means that the column existed in the base version, was subsequently deleted by one branch, and was retained by the other branch. In the branch that retained the column, it may still be written to or used, while the branch that deleted the column intended to remove it. These two changes are incompatible at the table structure level, therefore the column is identified as a table structure conflict column.

[0051] Conversely, columns that do not meet the above conditions will not be identified as columns with table structure conflicts. For example, if a column does not exist in the table corresponding to the latest common snapshot and is added only in one branch, this column is a unilaterally added column, and there are no conflicting changes to this column in either branch. As another example, if a column is deleted in both branches, this column does not meet the condition of existing in only one branch, and the changes to this column are consistent in both branches.

[0052] It should be noted that if only the current table structures of the first and second branches are compared, the situation where a column exists on one side but not on the other cannot be further distinguished. This situation could arise from a branch deleting a column from the baseline version, or from a branch adding a column that was not present in the baseline version. These two situations correspond to two opposing conclusions: conflict and compatibility, respectively. Using the table corresponding to the latest common snapshot as the benchmark for judgment can distinguish between these two situations.

[0053] Regarding the determination of the existence of a column in a branch, in some embodiments, for each column in the table corresponding to the latest common snapshot, the column change information of each snapshot can be read sequentially along the first change snapshot sequence. Based on the added and deleted records related to that column, it can be determined whether the column exists in the first branch, and the same method can be used to determine whether the column exists in the second branch, thereby determining whether the column meets the conditions for a table structure conflict column. In other embodiments, the existence of a column in a branch can also be determined in conjunction with the current table structure information of the branch, which is also part of the metadata of the feature data table. In response to the existence of at least one column that meets the conditions, it is determined that a table structure conflict column exists.

[0054] In step S204, in response to determining that there is a conflict column in the table structure, a conflict is determined to exist between the first branch and the second branch.

[0055] The presence of a table structure conflict column indicates that the changes to the two branches are incompatible at the table structure level and cannot be directly merged, thus determining that a conflict exists between the first branch and the second branch. In some embodiments, a detection result indicating a conflict between the first branch and the second branch can be generated and provided to the party that initiated the conflict detection as a basis for deciding whether to perform subsequent processing such as branch merging. In some embodiments, in response to determining that no table structure conflict column exists, it can be determined that there is no conflict between the first branch and the second branch at the table structure level.

[0056] According to embodiments of this disclosure, a corresponding snapshot is generated when a write operation is performed on the feature data table, and column change information indicating the columns added or deleted during the write operation is recorded in the snapshot's metadata. During conflict detection, conflicts between branches are determined based on the column change information included in the change snapshot sequences of each branch. This eliminates the need to read the feature data table's data file, and the amount of data read is independent of the data file's size, thus shortening detection time and reducing resource consumption. Furthermore, by using the table corresponding to the latest common snapshot as a benchmark to determine conflicting columns in the table structure, conflict detection caused by changes in the table structure is achieved.

[0057] Specifically, the column change information read by conflict detection increases with the number of write operations and the number of columns changed, regardless of the number of rows in the feature data table. For example, in a scenario where the feature data table contains hundreds of millions of rows and hundreds of columns, and the data file reaches terabyte levels, the column change information included in two change snapshot sequences is typically only in the kilobyte range. The detection process does not read the data file, thus reducing detection time and computational resources required. Furthermore, by using the table corresponding to the latest common snapshot as a benchmark, cases where a column is added only in one branch will not be identified as a table structure conflict column, thereby reducing additional conflict handling for compatible changes; cases where a column is deleted in one branch while the other branch retains the column can be detected, preventing such table structure conflicts from entering the merged data.

[0058] According to some embodiments, the metadata of the generated snapshot may also include a parent snapshot identifier indicating the snapshot generated by the previous write operation. The snapshot generated by the previous write operation is the parent snapshot of that snapshot. Within the same branch, the parent snapshot of a snapshot is the snapshot generated by the previous write operation on that branch. For the first snapshot generated after the branch is created, its parent snapshot is the snapshot on which the branch was based. Thus, each snapshot is sequentially linked by its parent snapshot identifier, forming a parent-child relationship between snapshots. In this case, step S201, determining the latest common snapshot of the first branch and the second branch of the feature data table, may include the following sub-steps: determining the latest generated first snapshot in the first branch and the latest generated second snapshot in the second branch; and traversing upwards level by level based on the parent snapshot identifiers of the first snapshot and the second snapshot, and determining the first common ancestor snapshot obtained as the latest common snapshot.

[0059] Since the parent snapshot identifier is an attribute carried by the snapshot itself, traversal based on the parent snapshot identifier only involves the snapshot's metadata. Even if the first and second branches are created at different times, or if one of the branches has been updated synchronously with other branches, the common baseline of the two branches can be accurately located along the parent-child relationship of the snapshots, improving the accuracy of locating the latest common snapshot and the adaptability of the method to different branch states.

[0060] For example, one can trace back from the first and second snapshots along the parent snapshot identifier, recording the snapshots traversed on both branches. When a snapshot traced back on one branch has been traversed by the other branch, that snapshot is the first common ancestor snapshot. Since each snapshot has only one parent snapshot identifier, the parent-child relationships between snapshots form a tree structure, thus the first common ancestor snapshot exists and is unique.

[0061] Figure 3 A flowchart illustrating a process 300 for determining whether a table structure conflict column exists, according to an exemplary embodiment of the present disclosure, is shown. Process 300 can be used to implement step S203 in the method 200 described above. Figure 3 As shown, process 300 may include: step S301, determining a first set of deleted columns based on column change information included in the first change snapshot sequence, the first set of deleted columns including columns that exist in the table corresponding to the latest common snapshot, were deleted in the first branch after the latest common snapshot, and were not re-added; step S302, determining a second set of deleted columns based on column change information included in the second change snapshot sequence, the second set of deleted columns including columns that exist in the table corresponding to the latest common snapshot, were deleted in the second branch after the latest common snapshot, and were not re-added; step S303, determining a first intersection between the first set of deleted columns and the column set of the second branch; step S304, determining a second intersection between the second set of deleted columns and the column set of the first branch; step S305, determining the union of the first intersection and the second intersection to obtain a set of table structure conflict columns; and step S306, determining that there are table structure conflict columns in response to the table structure conflict column set being non-empty.

[0062] Therefore, by aggregating column change information along the change snapshot sequence, the set of deleted columns can be quickly obtained. This allows for the identification of columns actually deleted relative to the baseline version in the corresponding branch. Cases where columns are deleted and then added back in the sequence are not counted, thus achieving accurate determination of table structure conflicts. Furthermore, all of the above determinations are column-level set operations, and the processing cost is related to the number of columns, not the number of rows in the feature data table. Compared to comparing the entire dataset, this improves processing efficiency.

[0063] For ease of description, let's denote the first set of deleted columns as Δschema_A, the second set of deleted columns as Δschema_B, the column set of the first branch as schema_A, and the column set of the second branch as schema_B. The column set of a branch refers to the set of columns included in the current table structure of that branch; the table structure is also part of the metadata of the feature data table.

[0064] For example, column change information for each snapshot can be read sequentially along ΔA. For each changed column, the added and deleted records are applied in the order of the write operations to obtain the final change status of the column in the first branch. Columns whose final change status is deleted and which exist in the table corresponding to S0 are then included in Δschema_A. Δschema_B is obtained in the same way. The set of conflicting columns is denoted as C, then C = (Δschema_A ∩ schema_B) ∪ (Δschema_B ∩ schema_A), where ∩ represents the intersection operation and ∪ represents the union operation. A non-empty C indicates the existence of conflicting columns.

[0065] According to some embodiments, method 200 may further include the following steps (not shown in the figure): in response to determining that there are conflicting columns in the table structure, outputting a set of conflicting columns in the table structure and preventing the merging of the first branch and the second branch.

[0066] Therefore, the existence of conflicting columns in the table structure indicates that the table structures of the two branches are incompatible. In this case, preventing the merging can avoid the incompatible table structure changes from entering the merged feature data table. The output set of conflicting columns in the table structure indicates the specific columns where the conflict occurred, providing a basis for resolving table structure conflicts.

[0067] For example, blocking a merge may include rejecting or suspending merge requests for the first and second branches, and re-performing conflict detection after the table structure conflicts have been resolved. In some embodiments, the output may also include information about the changes to each conflicting column in the two branches, such as indicating in which branch the column was deleted or still exists.

[0068] According to some embodiments, when the write operation is to calculate at least one column in a feature data table by executing a feature operator, the metadata of the generated snapshot may also include an operator identifier and an operator version, wherein the operator identifier indicates the feature operator executed and the operator version indicates the version of the feature operator executed.

[0069] Feature operators are programs used to calculate the values ​​of columns in a feature data table. Feature operators are managed in a versioned manner, and the operator version is updated when the calculation logic of the feature operator changes. For example, a feature operator reads the user's behavior logs, calculates the user's click-through rate over the past seven days, and writes the calculation result into the corresponding column in the feature data table; when the statistical scope of the feature operator is adjusted from only counting clicks within the application to counting clicks from both within the application and on the web, the operator version is updated.

[0070] An operator identifier is used to uniquely identify a feature operator among multiple feature operators. For example, the operator identifier can be the resource name registered by the feature operator in the feature platform. For instance, if a write operation for a feature calculation is performed by a feature operator with the resource name user_7d_ctr_op and the operator version used is v2, then the metadata of the snapshot generated by this write operation records the operator identifier user_7d_ctr_op and the operator version v2. In some embodiments, the operator identifier and operator version can be stored as two separate entries in the metadata, or they can be concatenated into a single entry.

[0071] The columns written by the feature operator can be determined based on the definition information of the feature operator, such as the output column information of the feature operator registered in the feature platform.

[0072] Figure 4 A flowchart of a branch conflict detection method 400 for a feature data table according to an exemplary embodiment of the present disclosure is shown. The operations and effects of steps S401 to S404 in method 400 can be referred to the description of steps S201 to S204 in method 200 above, and will not be repeated here. Figure 4 As shown, method 400 may further include: step S405, in response to determining that there are no conflicting columns in the table structure, determining whether there are conflicting columns at the column level based on the operator identifier and operator version included in the first change snapshot sequence and the second change snapshot sequence, wherein the conflicting columns at the column level are columns written in the first branch and the second branch with different operator versions by the feature operator indicated by the same operator identifier; and step S406, in response to determining that there are conflicting columns at the column level, determining that there is a conflict between the first branch and the second branch.

[0073] Therefore, it is possible to detect column-level conflicts caused by feature calculations without reading data files, expands the types of conflicts covered by conflict detection, and the detection cost is independent of the number of rows in the feature data table, saving computational costs.

[0074] The basis for determining column-level conflicts lies in the determinism of feature operator execution. The computational logic of feature operators of the same version is fixed; for the same input data, the output produced by the program execution will inevitably be the same. Therefore, the column values ​​calculated by the same version of feature operators for the same row are deterministic. When the computational logic of a feature operator changes, the operator version is updated accordingly. Based on this, if the operator version corresponding to the same operator identifier is consistent in two branches, the values ​​of the corresponding columns in both branches are generated by the same computational logic, the data semantics are consistent, and there is no conflict. If the operator versions are inconsistent, the values ​​of the corresponding columns are generated by different computational logics, the data semantics of the column branch between the two branches, constituting a column-level conflict. Therefore, column-level conflicts can be determined based on the comparison of operator versions, without needing to read and compare the column values ​​themselves.

[0075] In some embodiments, for each column in the feature data table, the operator version used when the column was last written by feature calculation in the first branch and the second branch can be determined, and the column can be determined as a column-level conflict column if the operator versions on both sides are inconsistent.

[0076] In some embodiments, in response to determining that a column-level conflict column exists, row-level conflict detection can also continue after the column-level conflict column is recorded, thereby obtaining multiple types of conflicts between two branches in a single detection.

[0077] Figure 5A flowchart of a process 500 for determining whether a column-level conflicting column exists, according to an exemplary embodiment of the present disclosure, is shown. Process 500 can be used to implement step S405 in the method 400 described above. Figure 5 As shown, process 500 may include: step S501, determining a first version mapping based on operator identifiers and operator versions in a first change snapshot sequence, wherein the first version mapping indicates the operator version corresponding to the last write of each operator identifier in the first change snapshot sequence; step S502, determining a second version mapping based on operator identifiers and operator versions in a second change snapshot sequence, wherein the second version mapping indicates the operator version corresponding to the last write of each operator identifier in the second change snapshot sequence; step S503, for an operator identifier that exists in both the first version mapping and the second version mapping, comparing the operator version corresponding to the operator identifier in the first version mapping with the operator version corresponding to the operator identifier in the second version mapping; and step S504, in response to the existence of one or more operator identifiers with inconsistent corresponding operator versions, determining the column written in the feature data table by the feature operator indicated by one or more operator identifiers as a column-level conflict column.

[0078] Therefore, feature computation provides full coverage of column writes, with the latest write of the same feature operator determining the final state of the corresponding column on that branch. Using the operator version corresponding to the last write for comparison ensures that the judgment result matches the actual final state of the column, improving the matching degree between column-level conflict determination and the actual branch state, while reducing computational load and resource consumption. Furthermore, the construction and comparison of version mappings only involve operator identifiers and operator versions in the metadata; the processing overhead is related to the number of feature operators, without requiring comparison of the entire feature data, thus improving processing efficiency.

[0079] For ease of description, let the first version mapping be denoted as V_A and the second version mapping as V_B. For example, we can sequentially read the operator identifiers and operator versions from each snapshot along ΔA. For each operator identifier, the operator version corresponding to its last occurrence in ΔA is used as the value of that operator identifier in V_A. V_B can be constructed in the same way. Both V_A and V_B can be mappings with operator identifiers as keys and operator versions as values. Let the set of operator identifiers with column-level conflicts be denoted as C_col. For an operator identifier r that exists in both V_A and V_B, let the operator version corresponding to r in V_A be v1 and the operator version corresponding to r in V_B be v2. If v1 and v2 are inconsistent, then r belongs to C_col, i.e., C_col = {r | (r,v1) ∈ V_A, (r, v2) ∈ V_B, v1 ≠ v2}. If C_col is not empty, it means that there are column-level conflict columns. Column-level conflict columns are the columns written in the feature data table by the feature operators indicated by the operator identifiers in C_col.

[0080] For operator identifiers that exist only in one of V_A and V_B, the feature calculations on their corresponding single side do not constitute a conflict. In some embodiments, the operator versions corresponding to operator identifiers with inconsistent operator versions in each of the two branches can also be recorded for reference when resolving column-level conflicts.

[0081] According to some embodiments, when a write operation adds a row to a feature data table, the metadata of the generated snapshot may also include a cardinality estimation summary, which is generated based on the primary key of the row added by the write operation.

[0082] Rows in the feature data table are identified by a primary key, which uniquely identifies a row. For example, using the user ID as the primary key, each row in the feature data table stores the characteristics of one user. Adding rows to the feature data table introduces rows that were not previously present; for example, adding data for a batch of newly registered users to the feature data table.

[0083] A cardinality estimate digest is a data structure generated from a set to estimate the cardinality of that set, where cardinality refers to the number of distinct elements in the set. Cardinality estimate digests support cardinality estimation and union operations: a cardinality estimate digest yields an estimate of the cardinality of the corresponding set; the union of two cardinality estimate digests corresponds to the union of the two sets.

[0084] In some embodiments, the cardinality estimation digest consists of a preset number of counting units, and its size is determined by the number of counting units, regardless of the number of elements in the represented set. In other embodiments, the cardinality estimation digest can be stored using sparse coding when the cardinality of the represented set is low, and its serialization volume does not exceed the upper limit of the size corresponding to dense representation. Thus, the size of the cardinality estimation digest has an upper limit independent of the cardinality of the set.

[0085] For example, the cardinality estimation digest can adopt a HyperLogLog structure, where the counting unit is called a register and serves as the storage unit in the digest data structure. When the number of counting units is 1024, the serialized cardinality estimation digest under dense representation is approximately 1 kilobyte. For example, when the writing end performs a write operation to add rows, it extracts the primary key of each row being written, hashes each primary key, updates the cardinality estimation digest with the hash result, and serializes the cardinality estimation digest before writing it to the metadata of the generated snapshot. The serialized cardinality estimation digest can be stored as an encoded string in the key-value pair dictionary of the metadata.

[0086] Figure 6A flowchart of a branch conflict detection method 600 for a feature data table according to an exemplary embodiment of the present disclosure is shown. The operations and effects of steps S601 to S606 in method 600 can be referred to the description of steps S401 to S406 in method 400 above, and will not be repeated here. Figure 6 As shown, method 600 may further include: step S607, in response to determining that there are no column-level conflicting columns, determining an intersection size estimate based on the cardinality estimation summaries included in the first change snapshot sequence and the second change snapshot sequence, the intersection size estimate being an estimate of the size of the intersection between the set of primary keys of the rows added to the first branch and the set of primary keys of the rows added to the second branch; and step S608, based on the intersection size estimate, determining whether there is a row-level conflict between the first branch and the second branch, a row-level conflict referring to the addition of rows with the same primary key in the first branch and the second branch.

[0087] When two branches add rows with the same primary key, the same row may contain different data in the two branches, constituting a row-level conflict. The size of the cardinality estimation digest is much smaller than the primary key set itself, and can be stored in the snapshot's metadata and saved with each version. In this way, the size of the intersection of the primary key sets on both sides can be estimated using the cardinality estimation digest, which consumes very little space and computational resources, thus initially determining whether a row-level conflict exists. This eliminates the need to read the full data of the added rows, saving computational overhead and further expanding the types of conflicts covered by conflict detection.

[0088] In some embodiments, if at least one of the first change snapshot sequence and the second change snapshot sequence does not include a cardinality estimate summary, it indicates that no rows have been added to the corresponding branch, and there is no intersection between the primary keys of the rows added to the two branches. In this case, it can be determined that there is no row-level conflict, and steps S607 and S608 do not need to be executed. In some embodiments, the intersection size estimate can be compared with a preset threshold, and the existence of a row-level conflict can be determined based on the comparison result. The preset threshold is, for example, zero.

[0089] For example, in the case of a branch introducing hundreds of millions of rows, storing the entire primary key would require hundreds of megabytes of space, while the cardinality estimate digest would only be about 1 kilobyte.

[0090] Figure 7 A flowchart is shown of a process 700 for determining an intersection size estimate according to an exemplary embodiment of the present disclosure. Process 700 can be used to implement step S607 in the method 600 described above. Figure 7As shown, process 700 may include: step S701, performing a union operation on each cardinality estimate summary included in the first change snapshot sequence to obtain a first summary summary; step S702, performing a union operation on each cardinality estimate summary included in the second change snapshot sequence to obtain a second summary summary; step S703, performing a union operation on the first summary summary and the second summary summary to obtain a merged summary; and step S704, subtracting the cardinality estimate of the merged summary from the sum of the cardinality estimate of the first summary summary and the cardinality estimate of the second summary summary to obtain an intersection size estimate.

[0091] Therefore, by first performing a union operation on the cardinality estimation summaries within the same branch, the primary keys corresponding to multiple write operations that add rows in a branch are aggregated, thereby improving processing efficiency. The overhead of the union operation and cardinality estimation depends only on the size of the cardinality estimation summary and is independent of the number of rows added. Therefore, the above method saves computational overhead compared to comparing the entire dataset.

[0092] The above calculation is based on the inclusion-exclusion principle: the cardinality of the intersection of two sets is equal to the sum of the cardinality of the two sets minus the cardinality of their union. For ease of description, let the first summary be denoted as HLL_A, the second summary as HLL_B, the operation of obtaining the cardinality estimate from the summaries as estimate, the union operation as union, and the intersection size estimate as n. Then, n = estimate(HLL_A) + estimate(HLL_B) The cardinality estimation error is related to the number of counting units. For example, when the number of counting units is m, the upper bound of the error is approximately 1.04 / √m, and when m is 1024, the error is approximately 3.3%. In some embodiments, if a change snapshot sequence includes only one cardinality estimation summary, the result of performing a union operation on it is the cardinality estimation summary itself.

[0093] According to some embodiments, step S608 may include the following sub-steps not shown in the figure: in response to the intersection size estimate being no greater than zero, determining that there is no row-level conflict, and performing a merge of the first branch and the second branch.

[0094] Therefore, when row-level conflicts can be ruled out based on metadata estimation, the merge can be performed directly without reading data files, reducing unnecessary data reading in the merge process, shortening the latency from conflict detection to completion of the merge, and reducing the consumption of computing resources.

[0095] If the intersection size estimate is not greater than zero, it means that the size of the intersection of the primary key sets on both sides does not exceed the error range of the cardinality estimate, and the probability of adding rows with the same primary key to the two branches is extremely low. Furthermore, by the time this sub-step is executed, table structure conflicts and column-level conflicts have been eliminated, so the merge can be performed directly. For example, performing the merge may include applying the changes corresponding to the first change snapshot sequence to the second branch to obtain the merged feature data table.

[0096] According to some embodiments, step S608 may also include the following sub-steps not shown in the figure: in response to the intersection size estimate being greater than zero, reading the rows added by the first branch and the second branch respectively, and determining whether there is a row-level conflict by comparing the primary keys of the added rows.

[0097] Therefore, row data is only read when there is an estimated intersection of primary keys, reducing the number of times data files are read and the amount of data. Furthermore, the final determination is based on the actual comparison of primary keys, and the deviation caused by cardinality estimation does not affect the final result, thus improving the accuracy of row-level conflict determination.

[0098] For example, the reading and comparison can be performed by a distributed computing engine, such as starting a Spark task to perform a join on the rows added to each of the two branches based on their primary keys. Rows with the same primary key are the rows that have row-level conflicts, thus obtaining the set of primary keys with row-level conflicts and the corresponding row numbers. In some embodiments, the columns with different values ​​in the two branches can also be determined for reference when resolving row-level conflicts.

[0099] In some embodiments, a conflict detection report can be output after the detection is completed. The conflict detection report may include at least one of the following: a set of conflicting columns in the table structure, operator identifiers for column-level conflicts and their corresponding operator versions in the two branches, a set of primary keys experiencing row-level conflicts, and items that can be automatically merged. Items that can be automatically merged indicate changes that do not constitute a conflict between the two branches and can be directly merged, such as cases where different columns are written to in the two branches, or cases where only one branch has writes. Thus, the conflict detection report indicates both the various types of conflicts that need to be resolved and changes that can be directly merged without processing, providing a complete basis for branch merging.

[0100] In some embodiments, write operations to the feature data table are categorized into three types based on operation type: table structure modification, feature calculation, and data import. These three types of write operations correspond to column change information, operator identifiers and versions, and cardinality estimation summaries in the metadata, respectively. The semantics of any write operation are recorded in the snapshot's metadata, enabling the detection of table structure conflicts, column-level conflicts, and row-level conflicts to be performed based on the metadata. Specifically, table structure modification refers to adding or deleting columns, feature calculation refers to calculating column values ​​by executing feature operators, and data import refers to adding rows. For example, in the scenario employing the above three-level detection, the metadata read for detection is in the kilobyte range, and the response time for a single conflict detection can be reduced from hours in the full scan mode to seconds.

[0101] In some embodiments, the detection of conflicts at all levels, such as table structure conflicts, column-level conflicts, and row-level conflicts, can be performed in their entirety, or only one level or any combination thereof. For example, column-level conflict detection can be performed alone, i.e., steps S401, S402, S405, and S406 in method 400 can be performed without steps S403 and S404. In this case, the execution of step S405 is not contingent on the absence of a column with a table structure conflict. Similarly, row-level conflict detection can be performed alone, i.e., steps S601, S602, S607, and S608 in method 600 can be performed. In this case, the execution of step S607 is not contingent on the absence of a column with a column-level conflict. Furthermore, column-level and row-level conflict detection can be performed without table structure conflict detection. Accordingly, in the above embodiments, the snapshot's metadata may include one or more of the following: column change information, operator identifier and operator version, and cardinality estimation summary. Each piece of information is recorded independently during the write operation. The embodiments described above in conjunction with step S201 for determining the latest common snapshot, the embodiments described in conjunction with process 500 for determining column-level conflict columns, the embodiments described in conjunction with process 700 for determining the intersection size estimate, and the sub-steps described for step S608 are also applicable to the embodiments described above for performing only partial detection.

[0102] According to another aspect of this disclosure, a branch conflict detection apparatus for a feature data table is provided. Write operations on the feature data table generate corresponding snapshots, and when a column of the feature data table is added or deleted in the write operation, the metadata of the generated snapshot includes column change information indicating the column added or deleted in the write operation. Figure 8As shown, the branch conflict detection device 800 for a feature data table includes: a snapshot determination unit 801, configured to determine the latest common snapshot of a first branch and a second branch of the feature data table; a sequence determination unit 802, configured to determine snapshots in the first branch that follow the latest common snapshot as a first change snapshot sequence, and snapshots in the second branch that follow the latest common snapshot as a second change snapshot sequence; a conflict column determination unit 803, configured to determine whether there is a table structure conflict column based on column change information included in the first change snapshot sequence and the second change snapshot sequence, wherein a table structure conflict column is a column that exists in the table corresponding to the latest common snapshot, exists in one branch, and does not exist in the other branch; and a conflict determination unit 804, configured to determine a conflict between the first branch and the second branch in response to determining that there is a table structure conflict column.

[0103] It is understood that the operation and effect of units 801 to 804 in the branch conflict detection device 800 for the feature data table can be referred to the above description of steps S201 to S204 in the branch conflict detection method 200 for the feature data table, and will not be repeated here.

[0104] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0105] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0106] refer to Figure 9 The present invention describes a structural block diagram of an electronic device 900 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0107] like Figure 9As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of the electronic device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0108] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, output unit 907, storage unit 908, and communication unit 909. Input unit 906 can be any type of device capable of inputting information to electronic device 900. Input unit 906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 907 can be any type of device capable of presenting information, and can include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 908 can include, but is not limited to, hard disk and optical disk. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0109] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods, processes, and / or processes described above. For example, in some embodiments, these methods, processes, and / or processes may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the methods, processes, and / or processes described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform these methods, processes, and / or processes by any other suitable means (e.g., by means of firmware).

[0110] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0114] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0115] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0116] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0117] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A branch conflict detection method for a feature data table, wherein, A snapshot is generated for each write operation to the feature data table, and in the event that a column of the feature data table is added or deleted during the write operation, the metadata of the generated snapshot includes column change information indicating the column added or deleted during the write operation. The method includes: Determine the latest common snapshot of the first and second branches of the feature data table; The snapshots in the first branch that follow the latest common snapshot are determined as the first change snapshot sequence, and the snapshots in the second branch that follow the latest common snapshot are determined as the second change snapshot sequence. Based on the column change information included in the first and second change snapshot sequences, it is determined whether there are any conflicting columns in the table structure. A conflicting column is a column that exists in the table corresponding to the latest common snapshot, exists in one branch, and does not exist in the other branch; and In response to the determination that there is a conflict column in the table structure, a conflict is determined to exist between the first branch and the second branch.

2. The method according to claim 1, wherein, The metadata of the generated snapshot also includes a parent snapshot identifier indicating the snapshot generated by the last write operation, wherein determining the latest common snapshot of the first and second branches of the feature data table includes: Determine the most recently generated first snapshot in the first branch and the most recently generated second snapshot in the second branch; and Based on the parent snapshot identifiers of the first snapshot and the second snapshot, the snapshot is traversed upwards level by level, and the first common ancestor snapshot obtained is determined as the latest common snapshot.

3. The method according to claim 1 or 2, wherein, The step of determining whether there are conflicting columns in the table structure based on the column change information included in the first change snapshot sequence and the second change snapshot sequence includes: A first set of deleted columns is determined based on the column change information included in the first change snapshot sequence. The first set of deleted columns includes columns that exist in the table corresponding to the latest common snapshot, were deleted in the first branch after the latest common snapshot, and were not re-added. A second set of deleted columns is determined based on the column change information included in the second change snapshot sequence. The second set of deleted columns includes columns that exist in the table corresponding to the latest common snapshot, were deleted in the second branch after the latest common snapshot, and were not added back. Determine the first intersection between the first set of deleted columns and the set of columns in the second branch; Determine the second intersection between the second set of deleted columns and the set of columns in the first branch; Determine the union of the first intersection and the second intersection to obtain the set of conflicting columns in the table structure; and In response to the fact that the set of conflicting columns in the table structure is not empty, it is determined that there are conflicting columns in the table structure.

4. The method according to claim 3, further comprising: In response to the determination that there are conflicting columns in the table structure, the set of conflicting columns in the table structure is output, and the merging of the first branch and the second branch is prevented.

5. The method according to claim 1 or 2, wherein, When the write operation is to calculate at least one column in the feature data table by executing a feature operator, the metadata of the generated snapshot further includes an operator identifier and an operator version, the operator identifier indicating the feature operator executed, and the operator version indicating the version of the feature operator executed. The method further includes: In response to determining that no table structure conflicting columns exist, based on the operator identifiers and operator versions included in the first and second change snapshot sequences, it is determined whether column-level conflicting columns exist. These column-level conflicting columns are those where a feature operator indicated by the same operator identifier is written with different operator versions in the first and second branches. In response to determining the existence of the column-level conflict column, a conflict is determined between the first branch and the second branch.

6. The method according to claim 5, wherein, In response to determining that no table structure conflicting columns exist, based on the operator identifiers and operator versions included in the first change snapshot sequence and the second change snapshot sequence, determining whether there are column-level conflicting columns includes: A first version mapping is determined based on the operator identifier and operator version in the first change snapshot sequence. The first version mapping indicates the operator version corresponding to the last write of each operator identifier in the first change snapshot sequence. A second version mapping is determined based on the operator identifier and operator version in the second change snapshot sequence. The second version mapping indicates the operator version corresponding to the last write of each operator identifier in the second change snapshot sequence. For an operator identifier that exists in both the first version mapping and the second version mapping, compare the operator version corresponding to that operator identifier in the first version mapping with the operator version corresponding to that operator identifier in the second version mapping; and In response to the existence of one or more operator identifiers with inconsistent corresponding operator versions, the column in the feature data table where the feature operator indicated by the one or more operator identifiers is written is determined as the column-level conflict column.

7. The method according to claim 5, wherein, When the write operation adds rows to the feature data table, the metadata of the generated snapshot also includes a cardinality estimation summary, which is generated based on the primary key of the row added by the write operation. The method further includes: In response to determining that no column-level conflicting columns exist, an intersection size estimate is determined based on the cardinality estimation summaries included in the first change snapshot sequence and the second change snapshot sequence, wherein the intersection size estimate is an estimate of the size of the intersection between the set of primary keys of the rows added in the first branch and the set of primary keys of the rows added in the second branch; and Based on the intersection size estimate, it is determined whether there is a row-level conflict between the first branch and the second branch, where the row-level conflict refers to the addition of rows with the same primary key in both the first branch and the second branch.

8. The method according to claim 7, wherein, Determining the intersection size estimate includes: Perform a union operation on each cardinality estimate summary included in the first change snapshot sequence to obtain a first summary summary; Perform a union operation on each cardinality estimate summary included in the second change snapshot sequence to obtain a second summary summary; Perform a union operation on the first summary digest and the second summary digest to obtain a merged digest; and The intersection size estimate is obtained by subtracting the cardinality estimate of the merged summary from the sum of the cardinality estimate of the first summary summary and the cardinality estimate of the second summary summary.

9. The method according to claim 7, wherein, Determining whether a row-level conflict exists between the first branch and the second branch based on the intersection size estimate includes: In response to the intersection size estimate being no greater than zero, it is determined that there is no row-level conflict, and the first branch and the second branch are merged.

10. The method according to claim 7, wherein, Determining whether a row-level conflict exists between the first branch and the second branch based on the intersection size estimate includes: In response to the intersection size estimate being greater than zero, the rows added by the first branch and the second branch are read, and the existence of the row-level conflict is determined by comparing the primary keys of the added rows.

11. A branch conflict detection device for a feature data table, wherein, A snapshot is generated for each write operation to the feature data table, and in the event that a column of the feature data table is added or deleted during the write operation, the metadata of the generated snapshot includes column change information indicating the column added or deleted during the write operation. The apparatus includes: The snapshot determination unit is configured to determine the latest common snapshot of the first branch and the second branch of the feature data table; The sequence determination unit is configured to determine the snapshots in the first branch that are located after the latest common snapshot as a first changed snapshot sequence, and to determine the snapshots in the second branch that are located after the latest common snapshot as a second changed snapshot sequence; The conflict column determination unit is configured to determine whether a table structure conflict column exists based on column change information included in the first change snapshot sequence and the second change snapshot sequence, wherein the table structure conflict column is a column that exists in the table corresponding to the latest common snapshot, exists in one branch, and does not exist in the other branch; and The conflict determination unit is configured to determine a conflict between the first branch and the second branch in response to determining that a conflict column in the table structure exists.

12. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 10.

Images