Pipeline optimization method, device, equipment and storage medium based on hybrid strategy

Abstract

The present application discloses a pipeline optimization method, device, equipment and storage medium based on a hybrid strategy, which belongs to the field of data pipeline screening, including: determining the target description feature of the target data pipeline according to the demand description text of the target data pipeline, and determining the pipeline description feature of each data pipeline according to the pipeline description text of each of the multiple data pipelines to be selected; determining the text comparison matrix of each data pipeline according to each pipeline description feature and the target description feature; wherein the text comparison matrix is ​​used to characterize the matching degree between the target description feature and each pipeline description feature; determining the demand evaluation value of each data pipeline according to multiple text comparison matrices, and determining the data pipeline corresponding to the maximum demand evaluation value as the target data pipeline. The problem of inaccurate data pipeline screening is solved when there are data pipelines with similar functions.

Description

Technical Field

[0001] The present application belongs to the field of data pipeline screening, and specifically relates to a pipeline optimization method, device, equipment and storage medium based on a hybrid strategy. Background Art

[0002] In modern society, data has become an important resource for business and social development. Advances in data analysis and processing technologies have enabled companies to extract valuable information from massive amounts of data to support decision-making and business optimization.

[0003] At present, there are some technologies that attempt to solve the problem of data pipeline optimization. For example, rule-based pipeline optimization technology screens pipelines through pre-set rules to ensure that the selected pipeline can meet the basic needs of users.

[0004] Although the existing rule-based and model-based pipeline optimization technologies have solved the problem of pipeline screening to a certain extent, the rule-based pipeline optimization method is too dependent on preset rules and lacks flexibility, making it difficult to adapt to dynamically changing user needs. In particular, when there are data pipelines with similar functions, data pipeline screening is inaccurate. Summary of the invention

[0005] The present application aims to provide a method for at least solving the problem of inaccurate data pipeline screening when there are data pipelines with similar functions.

[0006] In a first aspect, an embodiment of the present application discloses a pipeline optimization method based on a hybrid strategy, comprising:

[0007] Determine a target description feature of the target data pipeline according to the demand description text of the target data pipeline, and determine a pipeline description feature of each of the multiple data pipelines to be selected according to the pipeline description texts of each of the multiple data pipelines;

[0008] Determining a text comparison matrix for each of the data pipelines according to each of the pipeline description features and the target description features; the text comparison matrix is ​​used to characterize the degree of matching between the target description features and each of the pipeline description features;

[0009] The demand evaluation value of each data pipeline is determined according to the multiple text comparison matrices, and the data pipeline corresponding to the maximum demand evaluation value is determined as the target data pipeline.

[0010] In a second aspect, the embodiment of the present application further discloses a pipeline optimization device based on a hybrid strategy, comprising:

[0011] An input module, configured to determine a target description feature of the target data pipeline according to a demand description text for the target data pipeline, and determine a pipeline description feature of each of the plurality of data pipelines to be selected according to respective pipeline description texts of the data pipelines;

[0012] A preprocessing module, used to determine a text comparison matrix of each of the data pipelines according to each of the pipeline description features and the target description features; the text comparison matrix is ​​used to characterize the matching degree between the target description features and each of the pipeline description features;

[0013] A selection module is used to determine the demand evaluation value of each data pipeline according to the multiple text comparison matrices, and determine the data pipeline corresponding to the maximum demand evaluation value as the target data pipeline.

[0014] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the pipeline optimization method based on the hybrid strategy as described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the pipeline optimization method based on the hybrid strategy as described in the first aspect are implemented.

[0016] In summary, in the embodiment of the present application, by analyzing the demand description text of the target data pipeline and generating the target description feature, an accurate understanding of the target data pipeline demand is ensured, and the problem of inaccurate screening caused by the limitations of the preset rules is avoided. At the same time, the pipeline description texts of multiple data pipelines to be selected are analyzed to generate the pipeline description feature of each data pipeline, ensuring a comprehensive understanding of the characteristics of each data pipeline, so that the flexibility and accuracy of screening are enhanced by comprehensively analyzing the characteristics of each data pipeline; and then, a text comparison matrix is ​​generated according to each pipeline description feature and the target description feature, respectively, to characterize the degree of match between the target description feature and each pipeline description feature, so as to provide an objective evaluation standard by quantifying the degree of match between the target description feature and the pipeline description feature, and avoid the error of subjective judgment; finally, according to multiple text comparison matrices, the demand evaluation value of each data pipeline is determined, and the data pipeline corresponding to the largest demand evaluation value is determined as the target data pipeline, ensuring that the selected target data pipeline best meets the user needs, thereby solving the problem of inaccurate pipeline screening under similar user needs. Therefore, based on the method of the embodiment of the present application, the pipeline optimization method based on the hybrid strategy generates target description features and pipeline description features, and uses the text comparison matrix to perform matching evaluation, and finally determines the data pipeline that best meets the needs, providing higher flexibility and accuracy, and being able to adapt to dynamically changing user needs. It significantly improves the accuracy and efficiency of pipeline screening and solves the problem of inaccurate data pipeline screening when there are data pipelines with similar functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In the attached picture:

[0018] Figure 1 It is a flow chart of the steps of a pipeline optimization method based on a hybrid strategy provided in an embodiment of the present application;

[0019] Figure 2 is a flowchart of another pipeline optimization method based on a hybrid strategy provided in an embodiment of the present application;

[0020] Figure 3 It is a data flow diagram under an embodiment of the present application;

[0021] Figure 4 It is a pipeline optimization model involved in the embodiment of the present application;

[0022] Figure 5 It is a block diagram of a pipeline optimization device based on a hybrid strategy provided in an embodiment of the present application;

[0023] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application;

[0024] Figure 7 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0027] Figure 1 This embodiment provides a pipeline optimization method based on a hybrid strategy.

[0028] The method may include the following steps:

[0029] Step 101 : determining a target description feature of a target data pipeline according to a demand description text for the target data pipeline, and determining a pipeline description feature of each data pipeline according to respective pipeline description texts of a plurality of data pipelines to be selected.

[0030] In some embodiments of the present application, the target description feature of the target data pipeline is determined based on the demand description text of the target data pipeline, and the pipeline description feature of each data pipeline is determined based on the pipeline description text of each of the multiple data pipelines to be selected. The target description feature is a vector representation generated by processing the text of the user's requirements using natural language processing (NLP) technology, which is used to accurately express the user's requirements. Similarly, the pipeline description feature is a vector representation generated by NLP processing the description text of each data pipeline to be selected, which is used to accurately express the characteristics of each data pipeline. By generating these features, a basis can be provided for subsequent matching and screening.

[0031] For example, on a data analysis platform, a user wants to find a pipeline that can quickly process large data sets. The user enters the requirement description text as "a pipeline that can process 1TB of data within 30 minutes is needed". The system first performs NLP processing on the requirement description text to generate the target description feature. At the same time, the system obtains the description texts of multiple data pipelines to be selected from the pipeline text database, for example: the description text of pipeline A is "suitable for large data set processing, with an average processing time of 20 minutes"; the description text of pipeline B is "suitable for medium data set processing, with an average processing time of 35 minutes"; the description text of pipeline C is "suitable for small data set processing, with an average processing time of 15 minutes". The system performs NLP processing on these description texts to generate pipeline description features for each data pipeline. Through the generation of these features, data support can be provided for text comparison and matching in subsequent steps to ensure the accuracy and effectiveness of the screening process.

[0032] Step 102, determining a text comparison matrix for each data pipeline according to each pipeline description feature and the target description feature.

[0033] Among them, the text comparison matrix is ​​used to characterize the matching degree between the target description features and each pipeline description features.

[0034] In some embodiments of the present application, a text comparison matrix for each data pipeline is determined based on each pipeline description feature and the target description feature. The text comparison matrix is ​​used to characterize the degree of match between the target description feature and each pipeline description feature. Specifically, by calculating the similarity between the target description feature and each pipeline description feature, a matrix is ​​generated, and each element in the matrix represents the degree of match between the target description feature and the corresponding pipeline description feature in a certain dimension. The generation of the text comparison matrix is ​​the basis for demand assessment and pipeline screening in subsequent steps.

[0035] For example, on a certain data analysis platform, the user's requirement description text is "A pipeline that can process 1TB of data within 30 minutes is needed", and the system has generated the target description features of the requirement description text. At the same time, the system also generates pipeline description features for multiple data pipelines to be selected, such as the description features of pipelines A, B, and C. The system generates a corresponding text comparison matrix by calculating the similarity between the target description features and the description features of pipelines A, B, and C. For example, assuming that the target description feature is vt, the description feature of pipeline A is vA, the description feature of pipeline B is vB, and the description feature of pipeline C is vC, then the text comparison matrix can be expressed as: (vt, vA) T ,(vt,vB) T ,(vt,vC) T ; Among them, the symbol T Represents the transpose of a matrix.

[0036] Step 103: determine the demand evaluation value of each data pipeline according to the multiple text comparison matrices, and determine the data pipeline corresponding to the maximum demand evaluation value as the target data pipeline.

[0037] In some embodiments of the present application, the demand assessment value of each data pipeline is determined based on multiple text comparison matrices, and the data pipeline corresponding to the largest demand assessment value is determined as the target data pipeline. The demand assessment value is obtained by analyzing and calculating the text comparison matrix, and is used to quantify the degree of match between each data pipeline and the target demand. By comparing the demand assessment values ​​of each data pipeline, the target data pipeline that best meets the user's needs can be determined, thereby improving the accuracy and efficiency of pipeline screening.

[0038] For example, on a data analysis platform, the system has generated a text comparison matrix between the target description features and the description features of multiple data pipelines to be selected. Assume that there are three data pipelines A, B and C to be selected, and their text comparison matrices are: (vt, vA) T ,(vt,vB) T ,(vt,vC) T ; then the system can calculate the demand assessment value of each data pipeline based on these text comparison matrices. For example, the demand assessment value of pipeline A is 0.8, the demand assessment value of pipeline B is 0.6, and the demand assessment value of pipeline C is 0.9. By comparing these demand assessment values, the system determines that the demand assessment value of pipeline C is the largest, so pipeline C is determined as the target data pipeline. In this way, it can be ensured that the selected target data pipeline best meets user needs, thereby solving the problem of inaccurate pipeline screening under similar user needs.

[0039] In summary, in the embodiment of the present application, by analyzing the demand description text of the target data pipeline and generating the target description feature, an accurate understanding of the target data pipeline demand is ensured, and the problem of inaccurate screening caused by the limitations of the preset rules is avoided. At the same time, the pipeline description texts of multiple data pipelines to be selected are analyzed to generate the pipeline description feature of each data pipeline, ensuring a comprehensive understanding of the characteristics of each data pipeline, so that the flexibility and accuracy of screening are enhanced by comprehensively analyzing the characteristics of each data pipeline; and then, a text comparison matrix is ​​generated according to each pipeline description feature and the target description feature, respectively, to characterize the degree of match between the target description feature and each pipeline description feature, so as to provide an objective evaluation standard by quantifying the degree of match between the target description feature and the pipeline description feature, and avoid the error of subjective judgment; finally, according to multiple text comparison matrices, the demand evaluation value of each data pipeline is determined, and the data pipeline corresponding to the largest demand evaluation value is determined as the target data pipeline, ensuring that the selected target data pipeline best meets the user needs, thereby solving the problem of inaccurate pipeline screening under similar user needs. Therefore, based on the method of the embodiment of the present application, the pipeline optimization method based on the hybrid strategy generates target description features and pipeline description features, and uses the text comparison matrix to perform matching evaluation, and finally determines the data pipeline that best meets the needs, providing higher flexibility and accuracy, and being able to adapt to dynamically changing user needs. It significantly improves the accuracy and efficiency of pipeline screening and solves the problem of inaccurate data pipeline screening when there are data pipelines with similar functions.

[0040] Figure 2 This is another pipeline optimization method based on a hybrid strategy provided in an embodiment of the present application.

[0041] The method may include the following steps:

[0042] Step 201 : acquiring historical operation record data of a plurality of candidate data pipelines from a pipeline text database, and acquiring operation indicator data of the data pipeline to be selected.

[0043] In some embodiments of the present application, historical operation record data of multiple candidate data pipelines are obtained from the pipeline text database, and the operation indicator data of the data pipeline to be selected is obtained. The historical operation record data includes detailed information of each data pipeline in the past execution, such as execution time, success rate, etc. The operation indicator data is a key indicator for evaluating and screening data pipelines, such as average execution time, maximum execution time, etc. By obtaining these data, basic data support can be provided for subsequent pipeline optimization to ensure the accuracy and reliability of the screening process.

[0044] For example, on a data analysis platform, there is a pipeline text database that stores historical operation record data of multiple data pipelines. Assume that there are three candidate data pipelines A, B, and C. In step 201, the system extracts the historical operation record data of these three pipelines from the database: the historical operation record data of pipeline A includes: execution time (20 minutes, 25 minutes, 22 minutes), success rate (90%, 85%, 88%); the historical operation record data of pipeline B includes: execution time (30 minutes, 35 minutes, 32 minutes), success rate (95%, 92%, 94%); the historical operation record data of pipeline C includes: execution time (15 minutes, 18 minutes, 17 minutes), success rate (80%, 82%, 85%). At the same time, the system also extracts the operation indicator data of these pipelines, such as average execution time, maximum execution time, etc., to provide data support for subsequent pipeline optimization. By obtaining and analyzing these data, the foundation can be laid for screening the most suitable pipeline in the subsequent steps.

[0045] Step 202: determine the candidate data pipeline corresponding to the historical operation record data matching the operation indicator data as the data pipeline to be selected.

[0046] In some embodiments of the present application, the candidate data pipeline corresponding to the historical operation record data matching the operation index data is determined as the data pipeline to be selected. Specifically, by analyzing the historical operation record data of each candidate data pipeline, the data pipeline that meets the preset operation index is screened out. These operation indexes may include key performance indicators such as execution time and success rate. Through this screening, it can be ensured that the data pipeline to be selected performs well in historical operation and meets the standards of user needs.

[0047] For example, on a data analysis platform, there are three candidate data pipelines A, B, and C. The historical operation record data and operation indicator data of these pipelines have been obtained in the previous steps. Assume that the preset operation indicators are: average execution time is less than 30 minutes, and the success rate is greater than 85%. Then if the average execution time of pipeline A is 22 minutes and the success rate is 88%, it meets the preset operation indicators; the average execution time of pipeline B is 32 minutes, and the success rate is 94%, which does not meet the preset execution time indicators; the average execution time of pipeline C is 17 minutes, and the success rate is 82%, which also does not meet the preset success rate indicators. Based on these data, the system can determine pipeline A as the data pipeline to be selected because its historical operation record data meets the preset operation indicators. Through this screening, it can be ensured that the pipelines used in the subsequent steps perform well in historical operations and can meet user needs.

[0048] Optionally, the operation indicator data includes an execution threshold, and step 202 includes the following sub-steps:

[0049] Sub-step 2021, determining the historical record data value corresponding to the operation indicator data from the historical operation record data.

[0050] In some embodiments of the present application, historical record data values ​​corresponding to the operation index data are determined from the historical operation record data. The operation index data include key performance indicators such as execution thresholds. By analyzing the historical operation record data, specific values ​​corresponding to these operation index data can be extracted. These historical record data values ​​are used for screening and evaluation of candidate data pipelines in subsequent steps to ensure the accuracy and effectiveness of the screening process.

[0051] For example, on a data analysis platform, the system has obtained historical operation record data of multiple candidate data pipelines. Assume that the operation indicator data includes a threshold of 30 minutes for the execution time. The system needs to extract the historical record data value corresponding to the execution time from the historical operation record data. The historical operation record data of pipeline A includes: execution time (20 minutes, 25 minutes, 22 minutes); the historical operation record data of pipeline B includes: execution time (30 minutes, 35 minutes, 32 minutes); the historical operation record data of pipeline C includes: execution time (15 minutes, 18 minutes, 17 minutes); then the system will extract the historical record data value corresponding to the execution time from these historical operation record data, that is, the execution time of pipeline A is 20 minutes, 25 minutes and 22 minutes, the execution time of pipeline B is 30 minutes, 35 minutes and 32 minutes, and the execution time of pipeline C is 15 minutes, 18 minutes and 17 minutes. These data values ​​will be used for screening and evaluation of candidate data pipelines in subsequent steps to ensure the accuracy of the screening process.

[0052] Sub-step 2022: determining the candidate data pipelines whose historical record data values ​​corresponding to the operation indicator data values ​​are less than the execution threshold as the data pipelines to be selected.

[0053] In some embodiments of the present application, the candidate data pipelines whose historical record data values ​​corresponding to the operation indicator data values ​​are less than the execution threshold are determined as the data pipelines to be selected. Specifically, by comparing the historical record data values ​​of each candidate data pipeline with the preset execution threshold, those pipelines that have performed well in historical operations and whose execution time is less than the threshold are screened out. These screened pipelines will be used as the data pipelines to be selected for further evaluation and selection in subsequent steps.

[0054] For example, on a data analysis platform, the system has extracted the execution time data values ​​of multiple candidate data pipelines from the historical operation record data, and the preset execution time threshold is 30 minutes. The system needs to determine the candidate data pipelines with an execution time of less than 30 minutes as the data pipelines to be selected. Among them, the execution time of pipeline A is 20 minutes, 25 minutes and 22 minutes, all less than 30 minutes; the execution time of pipeline B is 30 minutes, 35 minutes and 32 minutes, all not less than 30 minutes; the execution time of pipeline C is 15 minutes, 18 minutes and 17 minutes, all less than 30 minutes, then the system determines pipeline A and pipeline C as the data pipelines to be selected, because their historical record data values ​​are all less than the execution time threshold of 30 minutes. Through this screening, it can be ensured that the data pipeline to be selected has excellent performance in historical operation and meets the user's demand standards.

[0055] Optionally, the execution threshold includes one or more of an execution count threshold, a minimum historical execution time threshold, an average historical execution time threshold, and a maximum historical execution time threshold.

[0056] In some specific embodiments of the present application, the execution threshold may include one or more of an execution count threshold, a minimum historical execution time threshold, an average historical execution time threshold, and a maximum historical execution time threshold. By comparing the historical record data value of each candidate data pipeline with these execution thresholds, pipelines that have performed well in historical operations are screened out. These screened pipelines will be used as data pipelines to be selected for further evaluation and selection in subsequent steps.

[0057] For example, the corresponding filter conditions can be set as:

[0058] If the total number of pipeline runs is >10 and the pipeline run success rate is <80%, then the pipeline does not meet the execution conditions; and / or if the total number of pipeline runs is >10 and the pipeline minimum historical run time is >30min, then the pipeline does not meet the execution conditions; and / or if the total number of pipeline runs is >10 and the pipeline average historical run time is >45min, then the pipeline does not meet the execution conditions; and / or if the total number of pipeline runs is >10 and the pipeline maximum historical run time is >60min, then the pipeline does not meet the execution conditions.

[0059] Step 203: determining a target description feature of the target data pipeline according to the requirement description text of the target data pipeline, and determining a pipeline description feature of each data pipeline according to the pipeline description texts of each of the multiple data pipelines to be selected.

[0060] The method shown in this step has been explained in step 101 and will not be repeated here.

[0061] Optionally, in order to determine the target description feature of the target data pipeline according to the requirement description text for the target data pipeline, step 203 includes the following sub-steps:

[0062] Sub-step 2031 , extracting multiple demand words according to the demand description text to obtain demand word features of the target data pipeline.

[0063] Among them, each component in the demand word-unit feature corresponds one-to-one to multiple demand words.

[0064] In some embodiments of the present application, multiple demand words are extracted from the demand description text to obtain the demand word features of the target data pipeline. Requirement words are keywords or phrases extracted from the demand description text that can accurately express the user's needs. By converting these demand words into vector representations, a basis can be provided for subsequent word embedding and feature matching. Each component in the demand word feature corresponds one-to-one to multiple demand words, ensuring the comprehensiveness and accuracy of the demand description.

[0065] For example, on a data analysis platform, the user inputs a requirement description text of "need a pipeline that can process 1TB of data within 30 minutes". The system first performs word segmentation and keyword extraction on the requirement description text to obtain multiple requirement tokens, such as "30 minutes", "processing", "1TB of data", etc. The system converts these requirement tokens into requirement token features. Assuming that the extracted requirement tokens are: requirement token 1: "30 minutes"; requirement token 2: "processing"; requirement token 3: "1TB of data", the generated requirement token features can be expressed as:

[0066] v 需求 =["30 minutes","processing","1TB data"],

[0067] Among them, each component in the demand word feature corresponds to the extracted demand word. In this way, data support can be provided for subsequent word embedding and feature matching to ensure accurate understanding and expression of user needs.

[0068] Sub-step 2032, inputting the required word-unit features into a preset word-unit embedding function to obtain the original target description features of the target data pipeline.

[0069] In some embodiments of the present application, the demand word features are input into a preset word embedding function to obtain the original target description features of the target data pipeline. The word embedding function is a natural language processing technology used to convert words into high-dimensional vector representations. These vectors can capture the semantic relationship and contextual information between words, thereby more accurately expressing the user's needs. By inputting the demand word features into the word embedding function, the original target description features of the target data pipeline can be generated, providing a basis for subsequent feature matching and screening.

[0070] For example, suppose the system has extracted multiple demand tokens from the user demand description text and generated demand token features, such as: 需求 =["30 minutes", "processing", "1TB data"], the system inputs the required word feature into the preset word embedding function to obtain the original target description feature of the target data pipeline. The original target description feature generated at this time may be expressed as: 目标 =[e 30分钟 ,e 处理 ,e 1TB数据 ], where e 30分钟 、e 处理 and e 1TB数据 These are the embedding vectors of the three demand terms, "30 minutes", "processing" and "1TB of data". These embedding vectors can be numerical values ​​or arrays to represent the corresponding demand terms. In this way, the original target description features of the target data pipeline can be generated to provide data support for subsequent feature matching and screening, ensuring accurate understanding and expression of user needs.

[0071] Sub-step 2033, when the length of the original target description feature is less than a preset embedding length threshold, supplementary data is added to the end of the original target description feature to update the original target description feature, and when the length of the original target description feature is equal to the embedding length threshold, the updated original target description feature is used as the target description feature.

[0072] In some embodiments of the present application, when the length of the original target description feature is less than a preset embedding length threshold, supplementary data is added to the end of the original target description feature to update the original target description feature, and when the length of the original target description feature is equal to the embedding length threshold, the updated original target description feature is used as the target description feature. The supplementary data is usually some padding vectors used to ensure that the lengths of all vectors are consistent for subsequent processing. In this way, it can be ensured that the length of the target description feature meets the preset standard, thereby improving the accuracy of feature matching and screening.

[0073] For example, on a data analysis platform, the system has generated the original target description feature of the user demand description text. Assume that the feature is: v 目标 =[e 30分钟 ,e 处理 ,e 1TB数据 ], assuming that the preset embedding length threshold is 5, and the length of the current original target description feature is 3. In order to meet the embedding length threshold requirement, the system needs to add supplementary data at the end of the original target description feature. The supplementary data can be a zero vector or other filling vector, for example, let v 填充 =[0,0], then the updated original target description feature is: v 目标 =[e 30分钟 ,e 处理 ,e 1TB数据 ,0,0], when the length of the original target description feature is equal to the embedding length threshold, the system will update the original target description feature as the target description feature. In this way, the length of the target description feature can be ensured to be consistent, which is convenient for subsequent feature matching and screening, and improves the accuracy and effectiveness of the screening process.

[0074] Optionally, in order to determine the pipeline description feature of each data pipeline according to the pipeline description texts of each of the multiple data pipelines to be selected, step 203 includes the following sub-steps:

[0075] Sub-step 2034, extracting multiple description words according to the pipeline description text to obtain description word features of the data pipeline.

[0076] Each component in the description word element feature corresponds one-to-one to multiple description word elements.

[0077] In some embodiments of the present application, multiple description words are extracted from the pipeline description text to obtain the description word element features of the data pipeline. The description word element is a keyword or phrase extracted from the pipeline description text, which can accurately express the characteristics of the data pipeline. By converting these description words into vector representations, a basis can be provided for subsequent word element embedding and feature matching. Each component in the description word element feature corresponds one-to-one to multiple description words, ensuring the comprehensiveness and accuracy of the pipeline description.

[0078] For example, on a certain data analysis platform, the system needs to process the description texts of multiple data pipelines to be selected. Suppose the description text of pipeline A is "suitable for large data set processing, with an average processing time of 20 minutes". The system first performs word segmentation and keyword extraction on the description text to obtain multiple description terms, such as "large data set processing", "average processing time", "20 minutes", etc. The system converts these description terms into description term features. Suppose the extracted description terms are: Description term 1: "large data set processing", Description term 2: "average processing time", Description term 3: "20 minutes", then the generated description term features can be expressed as:

[0079] v 描述 =["Large dataset processing","Average processing time","20 minutes"].

[0080] Each component in the description word feature corresponds to the extracted description word. In this way, data support can be provided for subsequent word embedding and feature matching, ensuring accurate understanding and expression of data pipeline features.

[0081] Sub-step 2035, inputting the description word-unit features into a preset word-unit embedding function to obtain the original pipeline description features of the data pipeline.

[0082] In some embodiments of the present application, the descriptive word features are input into a preset word embedding function to obtain the original pipeline description features of the data pipeline. The word embedding function is a natural language processing technology used to convert words into high-dimensional vector representations. These vectors can capture the semantic relationship and contextual information between words, thereby more accurately expressing the characteristics of the data pipeline. By inputting the descriptive word features into the word embedding function, the original pipeline description features of the data pipeline can be generated, providing a basis for subsequent feature matching and screening.

[0083] For example, on a data analysis platform, the system has extracted the descriptive word element of pipeline A and generated descriptive word element features, such as:

[0084] v 描述 =["Large data set processing", "Average processing time", "20 minutes"], then the system inputs the descriptive word feature into the preset word embedding function to obtain the original pipeline description feature of the data pipeline. Assume that the generated original pipeline description feature may be expressed as:

[0085] v 管道 =[e 大数据集处理 ,e 平均处理时间 ,e 20分钟 ], where e 大数据集处理 、e 平均处理时间 and e 20分钟These are the embedding vectors of the three descriptive terms "large data set processing", "average processing time" and "20 minutes". In this way, the original pipeline description features of the data pipeline can be generated to provide data support for subsequent feature matching and screening, ensuring accurate understanding and expression of data pipeline features.

[0086] Sub-step 2036, when the length of the original pipeline description feature is less than a preset embedding length threshold, add supplementary data to the end of the original pipeline description feature to update the original pipeline description feature, and when the length of the original pipeline description feature is equal to the embedding length threshold, use the updated original pipeline description feature as the pipeline description feature.

[0087] In some embodiments of the present application, when the length of the original pipeline description feature is less than a preset embedding length threshold, supplementary data is added to the end of the original pipeline description feature to update the original pipeline description feature, and when the length of the original pipeline description feature is equal to the embedding length threshold, the updated original pipeline description feature is used as the pipeline description feature. The supplementary data is usually some padding vectors used to ensure that the lengths of all features are consistent for subsequent processing. In this way, it can be ensured that the length of the pipeline description feature meets the preset standard, thereby improving the accuracy of feature matching and screening.

[0088] For example, on a data analysis platform, the system has generated the original pipeline description feature of pipeline A. Assume that the feature is: v 管道 =[e 大数据集处理 ,e 平均处理时间 ,e 20分钟 ], assuming that the preset embedding length threshold is 5, and the length of the current original pipeline description feature is 3. In order to meet the embedding length threshold requirement, the system needs to add supplementary data at the end of the original pipeline description feature. The supplementary data can be a zero vector or other filling vector. For example, let v 填充 =[0,0], then the updated original pipeline description feature is: 管道 =[e 大数据集处理 ,e 平均处理时间 ,e 20分钟 ,0,0], when the length of the original pipeline description feature is equal to the embedding length threshold, the system will update the original pipeline description feature as the pipeline description feature. In this way, the length of the pipeline description feature can be ensured to be consistent, which is convenient for subsequent feature matching and screening, and improves the accuracy and effectiveness of the screening process.

[0089] Step 204 , determining a text comparison matrix for each data pipeline according to each pipeline description feature and the target description feature.

[0090] Among them, the text comparison matrix is ​​used to characterize the matching degree between the target description features and each pipeline description features.

[0091] The method shown in this step has been explained in step 102 and will not be repeated here.

[0092] Optionally, the target description feature and the pipeline description feature have the same data length, and step 204 includes the following sub-steps:

[0093] Sub-step 2041, taking the pipeline description features of the data pipeline as the first matrix row of the matrix, and taking the target description features as the second matrix row of the matrix, to obtain the original embedding matrix corresponding to each data pipeline.

[0094] In some embodiments of the present application, the pipeline description features of the data pipeline are respectively used as the first matrix row of the matrix, and the target description features are used as the second matrix row of the matrix to obtain the original embedding matrix corresponding to each data pipeline. In this way, the target description features and the pipeline description features can be combined into one matrix to facilitate subsequent processing and analysis. Each row of the original embedding matrix represents the target description features and the pipeline description features, respectively, ensuring the consistency and comparability of the data, and providing a basis for subsequent multi-layer coding block processing.

[0095] For example, on a data analysis platform, the system has generated target description features and pipeline description features for multiple data pipelines. Assume that the target description feature is: 目标 =[e 30分钟 ,e 处理 ,e 1TB数据 ,0,0],

[0096] The descriptive characteristics of pipeline A are: 管道A =[e 大数据集处理 ,e 平均处理时间 ,e 20分钟 ,0,0], then the system takes the description features of pipeline A as the first matrix row of the matrix and the target description features as the second matrix row of the matrix to generate the original embedding matrix: M 嵌入A =( v 目标 , v 管道A ) T ,In this way, the system can generate the original embedding matrix corresponding to each ,data pipeline, providing data support for the subsequent multi-layer encoding block ,processing, ensuring the accuracy and effectiveness of feature matching and ,screening.

[0097] In sub-step 2042, the obtained multiple original embedding matrices are respectively input into the pre-trained multi-layer encoding block to obtain the text comparison matrix of each data pipeline.

[0098] In some embodiments of the present application, the obtained multiple original embedding matrices are respectively input into the pre-trained multi-layer encoding block to obtain the text contrast matrix of each data pipeline. The multi-layer encoding block is a deep learning model, which is usually composed of multiple levels of neural networks and is used to process and extract high-level features of the input data. By inputting the original embedding matrix into the pre-trained multi-layer encoding block, a text contrast matrix can be generated, which is used to characterize the degree of match between the target description features and the pipeline description features. The generation of the text contrast matrix is ​​the basis for demand assessment and pipeline screening in subsequent steps.

[0099] For example, on a data analysis platform, the system has generated original embedding matrices corresponding to multiple data pipelines. Assume that the original embedding matrix of pipeline A is: M 嵌入A =( v 目标 , v 管道A ) T , where v 目标 =[e 30分钟 ,e 处理 ,e 1TB数据 ,0,0],v 管道A =[e 大数据集处理 ,e 平均处理时间 ,e 20分钟 ,0,0], the system inputs the original embedding matrix into the pre-trained multi-layer encoding block. Through the processing of the multi-layer encoding block, the system can extract the high-level features of the input matrix and generate the text contrast matrix T of pipeline A 对比A In this way, the system can generate a text comparison matrix for each data pipeline, providing data support for subsequent demand assessment and pipeline screening, ensuring the accuracy and effectiveness of the screening process.

[0100] Step 205 , determining the demand evaluation value of each data pipeline according to the multiple text comparison matrices, and determining the data pipeline corresponding to the maximum demand evaluation value as the target data pipeline.

[0101] The method shown in this step has been explained in step 103 and will not be repeated here.

[0102] Optionally, the text comparison matrix includes a first matrix row, and the first matrix row of the text comparison matrix is ​​used to store the pipeline description feature of the data pipeline corresponding to the text comparison matrix; in order to determine the demand assessment value of each data pipeline according to the multiple text comparison matrices, step 205 includes the following sub-steps:

[0103] Sub-step 2051, averaging the first matrix rows of the plurality of text comparison matrices bit by bit to obtain a mean matrix row.

[0104] In some embodiments of the present application, the first matrix rows of multiple text contrast matrices are bitwise averaged to obtain a mean matrix row. The first matrix row of the text contrast matrix stores the values ​​corresponding to the pipeline description features of each data pipeline, and a mean matrix row can be generated by bitwise averaging these matrix rows. The mean matrix row represents the average features of all data pipeline description features, which is used for subsequent deviation calculation and demand assessment.

[0105] For example, on a data analysis platform, the system has generated text comparison matrices for multiple data pipelines. Assume there are three data pipelines A, B, and C, and the first matrix row r of their text comparison matrices is A 、r B and r C They are: A =[0.8,0.6,0.7,0.5,0.9], r B =[0.7,0.5,0.6,0.4,0.8],

[0106] r C =[0.9,0.7,0.8,0.6,1.0], then the system can average these first matrix rows bit by bit, and the mean matrix row is: r 均值 =(r A +r B +r C ) / 3, the calculation result is: r 均值= [0.8, 0.6, 0.7, 0.5, 0.9],In this way, the system can generate mean matrix rows to provide data support for subsequent deviation calculation and demand assessment, ensuring the accuracy and effectiveness of the screening process.

[0107] Sub-step 2052, using the difference between the first matrix row and the mean matrix row of each text comparison matrix to update the text comparison matrix corresponding to the first matrix row, so as to obtain the deviation matrix of each data pipeline.

[0108] Each data bit of the first matrix row of the deviation matrix is ​​used to represent the difference in the corresponding data bit between the data pipeline corresponding to the deviation matrix and the target data pipeline.

[0109] In some embodiments of the present application, the text comparison matrix corresponding to the first matrix row of each text comparison matrix is ​​updated with the difference between the first matrix row and the mean matrix row to obtain the deviation matrix of each data pipeline. Each data bit of the first matrix row of the deviation matrix is ​​used to characterize the difference between the data pipeline corresponding to the deviation matrix and the target data pipeline in the corresponding data bits. By calculating these differences, the degree of match between each data pipeline and the target data pipeline can be quantified, providing a basis for subsequent demand assessment.

[0110] For example, on a data analysis platform, the system has generated text contrast matrices and mean matrix rows for multiple data pipelines. Assume there are three data pipelines A, B, and C, and the first matrix row of their text contrast matrices is: A 、r B and r C , and the mean matrix behavior is r 均值 , then the system updates the text contrast matrix by the difference between the first matrix row of each text contrast matrix and the mean matrix row to obtain the deviation matrix. For example, the first matrix row of the deviation matrix of pipeline A is calculated as follows: A =r A −r 均值 Similarly, the first matrix behavior of the bias matrices for pipeline B and pipeline C is: B =r B −r 均值 and d C =r C −r 均值 ,In this way, the system can generate a deviation matrix for each data pipeline, ,providing data support for subsequent demand assessment and pipeline ,screening, ensuring the accuracy and effectiveness of the screening ,process.

[0111] Sub-step 2053, inputting the first matrix row of the deviation matrix into the pre-trained evaluation function to obtain the demand assessment value.

[0112] In some embodiments of the present application, the first matrix row of the deviation matrix is ​​input into a pre-trained evaluation function to obtain a demand evaluation value. The evaluation function is a pre-trained model that quantifies the data in the first matrix row of the deviation matrix to generate a demand evaluation value. The demand evaluation value represents the degree of match between each data pipeline and the target data pipeline. Through this quantification, the data pipeline that best meets the user's needs can be more accurately evaluated and selected.

[0113] For example, on a data analysis platform, the system has generated deviation matrices for multiple data pipelines. Assume that the first matrix row of the deviation matrices of pipelines A, B, and C is d A d B and d C , then the system inputs the first matrix row of these deviation matrices into the pre-trained evaluation function. For example, the evaluation function can be a simple weighted sum function or a complex neural network model. Assuming that a weighted sum function is used, the weight of the evaluation function is w=[1,1,1,1,1] T , then the demand assessment value is calculated as follows: At this time, assuming that d A =[0,0,0,0,0],d B =[−0.1,−0.1,−0.1,−0.1,−0.1], dC =[0.1,0.1,0.1,0.1,0.1], then the demand assessment value score of pipeline A is A =d A ⋅w=0, pipeline B’s demand assessment score B =d B ⋅w=−0.5, and the demand assessment score of pipeline C C =d C ⋅w=0.5. In this way, the system can generate a demand assessment value for each data pipeline, provide data support for subsequent pipeline screening, and ensure the accuracy and effectiveness of the screening process.

[0114] refer to Figure 3 It is a data flow diagram in a specific embodiment of the present application. In some specific embodiments of the present application:

[0115] Step S1: Obtain operation history records, description texts, requirement description texts, and operation indicator data to screen candidate data pipelines.

[0116] Step S2: Screen the target data pipeline from the candidate data pipelines obtained in step S1 according to the requirement description text.

[0117] As reference Figure 4 As shown, a pipeline optimization model involved in an embodiment of the present application is used, which takes the target description feature of the user demand and the pipeline description feature as the pipeline metadata as input, generates input features containing N+M words according to the number of pipelines, and performs the following process:

[0118] R1: The input features are processed by a multi-encoding module, which contains multiple encoding blocks. Each encoding block contains a multi-head attention module, a normalization module, and a forward feedback network module, and finally obtains multiple output features corresponding to each data pipeline;

[0119] R2: Input the obtained multiple output features into the list network for comprehensive evaluation, and finally select the corresponding target data pipeline in the data pipeline.

[0120] In summary, in the embodiment of the present application, by analyzing the demand description text of the target data pipeline and generating the target description feature, an accurate understanding of the target data pipeline demand is ensured, and the problem of inaccurate screening caused by the limitations of the preset rules is avoided. At the same time, the pipeline description texts of multiple data pipelines to be selected are analyzed to generate the pipeline description feature of each data pipeline, ensuring a comprehensive understanding of the characteristics of each data pipeline, so that the flexibility and accuracy of screening are enhanced by comprehensively analyzing the characteristics of each data pipeline; and then, a text comparison matrix is ​​generated according to each pipeline description feature and the target description feature, respectively, to characterize the degree of match between the target description feature and each pipeline description feature, so as to provide an objective evaluation standard by quantifying the degree of match between the target description feature and the pipeline description feature, and avoid the error of subjective judgment; finally, according to multiple text comparison matrices, the demand evaluation value of each data pipeline is determined, and the data pipeline corresponding to the largest demand evaluation value is determined as the target data pipeline, ensuring that the selected target data pipeline best meets the user needs, thereby solving the problem of inaccurate pipeline screening under similar user needs. Therefore, based on the method of the embodiment of the present application, the pipeline optimization method based on the hybrid strategy generates target description features and pipeline description features, and uses the text comparison matrix to perform matching evaluation, and finally determines the data pipeline that best meets the needs, providing higher flexibility and accuracy, and being able to adapt to dynamically changing user needs. It significantly improves the accuracy and efficiency of pipeline screening and solves the problem of inaccurate data pipeline screening when there are data pipelines with similar functions.

[0121] refer to Figure 5 , which shows a pipeline optimization device 30 based on a hybrid strategy provided by an embodiment of the present application, comprising:

[0122] An input module 301 is used to determine a target description feature of a target data pipeline according to a demand description text for the target data pipeline, and to determine a pipeline description feature of each data pipeline according to respective pipeline description texts of a plurality of data pipelines to be selected;

[0123] A preprocessing module 302 is used to determine a text comparison matrix for each data pipeline according to each pipeline description feature and the target description feature; the text comparison matrix is ​​used to characterize the matching degree between the target description feature and each pipeline description feature;

[0124] The selection module 303 is used to determine the demand evaluation value of each data pipeline according to the multiple text comparison matrices, and determine the data pipeline corresponding to the maximum demand evaluation value as the target data pipeline.

[0125] Optionally, the input module 301 includes:

[0126] The first word-element submodule is used to extract multiple demand words according to the demand description text to obtain demand word-element features of the target data pipeline; each component in the demand word-element features corresponds to the multiple demand words one by one;

[0127] A first embedding submodule, configured to input the required word-unit features into a preset word-unit embedding function to obtain original target description features of the target data pipeline;

[0128] The first padding submodule is used to add supplementary data at the end of the original target description feature to update the original target description feature when the length of the original target description feature is less than a preset embedding length threshold, and to use the updated original target description feature as the target description feature when the length of the original target description feature is equal to the embedding length threshold.

[0129] Optionally, the input module 301 includes:

[0130] The second word-element submodule is used to extract multiple description words according to the pipeline description text to obtain description word-element features of the data pipeline; each component in the description word-element features corresponds to the multiple description words one by one;

[0131] A second embedding submodule is used to input the description word unit feature into a preset word unit embedding function to obtain the original pipeline description feature of the data pipeline;

[0132] The second padding submodule is used to add supplementary data at the end of the original pipeline description feature to update the original pipeline description feature when the length of the original pipeline description feature is less than a preset embedding length threshold, and to use the updated original pipeline description feature as the pipeline description feature when the length of the original pipeline description feature is equal to the embedding length threshold.

[0133] Optionally, the target description feature and the pipeline description feature have the same data length, and the preprocessing module 302 includes:

[0134] A raw embedding submodule, for taking the pipeline description features of the data pipeline as the first matrix row of the matrix and taking the target description features as the second matrix row of the matrix, to obtain raw embedding matrices corresponding to each data pipeline respectively;

[0135] The encoding submodule is used to input the obtained multiple original embedding matrices into the pre-trained multi-layer encoding block respectively to obtain the text contrast matrix of each data pipeline.

[0136] Optionally, the text comparison matrix includes a first matrix row, and the first matrix row of the text comparison matrix is ​​used to store the pipeline description feature of the data pipeline corresponding to the text comparison matrix; the selection module 303 includes:

[0137] An averaging submodule, used for averaging the first matrix rows of the plurality of text comparison matrices bit by bit to obtain a mean matrix row;

[0138] The deviation submodule is used to update the text comparison matrix corresponding to the first matrix row of each text comparison matrix by the difference between the first matrix row and the mean matrix row, so as to obtain the deviation matrix of each data pipeline; each data bit of the first matrix row of the deviation matrix is ​​used to represent the difference between the data pipeline corresponding to the deviation matrix and the target data pipeline in the corresponding data bit;

[0139] The normalization submodule is used to input the first matrix row of the deviation matrix into the pre-trained evaluation function to obtain the demand evaluation value.

[0140] Optionally, the pipeline optimization device 30 based on the hybrid strategy further includes:

[0141] A historical data module is used to obtain historical operation record data of multiple candidate data pipelines from a pipeline text database, and obtain operation indicator data of the data pipeline to be selected;

[0142] The screening module is used to determine the candidate data pipelines corresponding to the historical operation record data matching the operation indicator data as the data pipeline to be selected.

[0143] Optionally, the operation indicator data includes an execution threshold, and the screening module includes:

[0144] A screening and obtaining submodule is used to determine the historical record data value corresponding to the operation index data from the historical operation record data;

[0145] The screening submodule is used to determine the candidate data pipelines whose historical record data values ​​corresponding to the operation indicator data values ​​are less than the execution threshold as the data pipelines to be selected.

[0146] In summary, in the embodiment of the present application, by analyzing the demand description text of the target data pipeline and generating the target description feature, an accurate understanding of the target data pipeline demand is ensured, and the problem of inaccurate screening caused by the limitations of the preset rules is avoided. At the same time, the pipeline description texts of multiple data pipelines to be selected are analyzed to generate the pipeline description feature of each data pipeline, ensuring a comprehensive understanding of the characteristics of each data pipeline, so that the flexibility and accuracy of screening are enhanced by comprehensively analyzing the characteristics of each data pipeline; and then, a text comparison matrix is ​​generated according to each pipeline description feature and the target description feature, respectively, to characterize the degree of match between the target description feature and each pipeline description feature, so as to provide an objective evaluation standard by quantifying the degree of match between the target description feature and the pipeline description feature, and avoid the error of subjective judgment; finally, according to multiple text comparison matrices, the demand evaluation value of each data pipeline is determined, and the data pipeline corresponding to the largest demand evaluation value is determined as the target data pipeline, ensuring that the selected target data pipeline best meets the user needs, thereby solving the problem of inaccurate pipeline screening under similar user needs. Therefore, based on the method of the embodiment of the present application, the pipeline optimization method based on the hybrid strategy generates target description features and pipeline description features, and uses the text comparison matrix to perform matching evaluation, and finally determines the data pipeline that best meets the needs, providing higher flexibility and accuracy, and being able to adapt to dynamically changing user needs. It significantly improves the accuracy and efficiency of pipeline screening and solves the problem of inaccurate data pipeline screening when there are data pipelines with similar functions.

[0147] Reference Figure 6 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0148] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0149] The memory 504 is used to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0150] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0151] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0152] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0153] The input / output I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0154] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0155] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0156] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of the present application.

[0157] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0158] Figure 7FIG. 6 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server.

[0159] Reference Figure 7 , the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632, for storing instructions that can be executed by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the method provided in the embodiment of the present application.

[0160] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0161] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0162] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A pipeline optimization method based on a hybrid strategy, characterized in that: include: Determine a target description feature of the target data pipeline according to the demand description text of the target data pipeline, and determine a pipeline description feature of each of the multiple data pipelines to be selected according to the pipeline description texts of each of the multiple data pipelines; Determining a text comparison matrix for each of the data pipelines according to each of the pipeline description features and the target description features; the text comparison matrix is ​​used to characterize the degree of matching between the target description features and each of the pipeline description features; Determine a demand evaluation value for each of the data pipelines according to the plurality of text comparison matrices, and determine the data pipeline corresponding to the largest demand evaluation value as the target data pipeline; The text contrast matrix comprises a first matrix row, and the first matrix row of the text contrast matrix is ​​used to store pipeline description features of the data pipeline corresponding to the text contrast matrix; Determining the demand assessment value of each data pipeline according to the plurality of text comparison matrices includes: averaging the first matrix rows of the plurality of text comparison matrices bit by bit to obtain a mean matrix row; The text comparison matrix corresponding to each of the first matrix rows of the text comparison matrix is ​​updated by using the difference between the first matrix row of each of the text comparison matrix and the mean matrix row, so as to obtain the deviation matrix of each of the data pipelines; each data bit of the first matrix row of the deviation matrix is ​​used to represent the difference between the data pipeline corresponding to the deviation matrix and the target data pipeline in the corresponding data bit; The first matrix row of the deviation matrix is ​​input into a pre-trained evaluation function to obtain the demand assessment value.

2. The pipeline optimization method based on hybrid strategy according to claim 1, characterized in that: The determining the target description feature of the target data pipeline according to the requirement description text of the target data pipeline includes: Extracting multiple demand words according to the demand description text to obtain demand word features of the target data pipeline; each component in the demand word feature corresponds to the multiple demand words one by one; Inputting the required word-unit feature into a preset word-unit embedding function to obtain the original target description feature of the target data pipeline; When the length of the original target description feature is less than a preset embedding length threshold, supplementary data is added to the end of the original target description feature to update the original target description feature, and when the length of the original target description feature is equal to the embedding length threshold, the updated original target description feature is used as the target description feature.

3. The pipeline optimization method based on hybrid strategy according to claim 1, characterized in that: The determining of the pipeline description feature of each of the data pipelines according to the pipeline description texts of the respective data pipelines to be selected includes: Extracting multiple description words according to the pipeline description text to obtain description word element features of the data pipeline; each component in the description word element features corresponds to the multiple description words one by one; Inputting the descriptive word feature into a preset word embedding function to obtain the original pipeline description feature of the data pipeline; When the length of the original pipeline description feature is less than a preset embedding length threshold, supplementary data is added to the end of the original pipeline description feature to update the original pipeline description feature, and when the length of the original pipeline description feature is equal to the embedding length threshold, the updated original pipeline description feature is used as the pipeline description feature.

4. The pipeline optimization method based on hybrid strategy according to claim 1, characterized in that: The target description feature and the pipeline description feature have the same data length, and determining the text comparison matrix of each data pipeline according to each pipeline description feature and the target description feature respectively includes: Taking the pipeline description features of the data pipelines as the first matrix row of a matrix, and taking the target description features as the second matrix row of the matrix, respectively, to obtain original embedding matrices corresponding to each of the data pipelines; The obtained multiple original embedding matrices are respectively input into the pre-trained multi-layer encoding block to obtain the text comparison matrix of each data pipeline.

5. The pipeline optimization method based on hybrid strategy according to claim 1, characterized in that: The method further comprises: Acquire historical operation record data of multiple candidate data pipelines from a pipeline text database, and acquire operation indicator data of the data pipeline to be selected; The candidate data pipeline corresponding to the historical operation record data matching the operation indicator data is determined as the data pipeline to be selected.

6. The pipeline optimization method based on hybrid strategy according to claim 5, characterized in that: The operation indicator data includes an execution threshold, and determining the candidate data pipeline corresponding to the historical operation record data matching the operation indicator data as the data pipeline to be selected includes: Determine the historical record data value corresponding to the operation index data from the historical operation record data; The candidate data pipeline whose historical record data value corresponding to the operation indicator data value is smaller than the execution threshold is determined as the data pipeline to be selected.

7. A pipeline optimization device based on a hybrid strategy, characterized in that: include: An input module, configured to determine a target description feature of a target data pipeline according to a demand description text for the target data pipeline, and determine a pipeline description feature of each of a plurality of data pipelines to be selected according to respective pipeline description texts of the data pipelines; A preprocessing module, used to determine a text comparison matrix of each of the data pipelines according to each of the pipeline description features and the target description features; the text comparison matrix is ​​used to characterize the matching degree between the target description features and each of the pipeline description features; A selection module, configured to determine a demand evaluation value of each of the data pipelines according to the plurality of text comparison matrices, and determine the data pipeline corresponding to the largest demand evaluation value as the target data pipeline; The text comparison matrix comprises a first matrix row, and the first matrix row of the text comparison matrix is ​​used to store pipeline description features of the data pipeline corresponding to the text comparison matrix; the selection module comprises: An averaging submodule, used for averaging the first matrix rows of the plurality of text comparison matrices bit by bit to obtain a mean matrix row; A deviation submodule is used to update the text comparison matrix corresponding to the first matrix row of each text comparison matrix by using the difference between the first matrix row and the mean matrix row, so as to obtain the deviation matrix of each data pipeline; each data bit of the first matrix row of the deviation matrix is ​​used to represent the difference between the data pipeline corresponding to the deviation matrix and the target data pipeline in the corresponding data bit; The normalization submodule is used to input the first matrix row of the deviation matrix into the pre-trained evaluation function to obtain the demand assessment value.

8. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the pipeline optimization method based on a hybrid strategy as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the pipeline optimization method based on a hybrid strategy as described in any one of claims 1 to 6.

Images