Similar model identification method and device, equipment, storage medium and product

Abstract

The invention discloses a similar model recognition method and device, equipment, a storage medium and a product, relates to the technical field of data development, and discloses a similar model recognition method which comprises the steps that an original service field of a new application model is acquired; on the basis of the data blood relationship between the newly-built application model and a pre-model of the newly-built application model, identifying a pre-model field on which the original business field depends, and obtaining a standard business field of the pre-model field from a total business field standard library; updating the original business field by adopting the standard business field to obtain a standard application model of the new application model; and according to the similarity between the standard application model and the pre-model, determining the similarity between the new application model and the pre-model, so that the problems of repeated development of similar models and increase of model development cost are solved, and the model development cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of data development technology, and in particular to a method, device, equipment, storage medium and product for identifying similar models. Background Art

[0002] At present, the big data platform undertakes the data collection, data analysis, data development, deployment, and scheduling tasks of each tenant, and supports the daily data processing and other tasks of the enterprise. As the enterprise develops, the data analysis needs of various departments are increasing, and the analysis dimension is gradually changing from single business to integrated business analysis. Each business department applies for tenants on the big data platform to carry out data development work.

[0003] In the related art, the data development workflow of the big data platform is as follows: the data developers of each tenant of the big data platform develop models on the big data platform and input relevant standardized model information. The big data developers of the tenants develop programs on the big data platform, solidify the data analysis code, and complete the configuration of program scheduling information. The tenant launches the data analysis program, the program is automatically scheduled, the analysis statement is submitted to the corresponding database for automatic operation, and the analysis results are fed back.

[0004] During data development, it is impossible to determine whether the developed model exists in the big data platform. As a result, there is a need to repeatedly develop models similar to existing models, which increases development costs. Summary of the invention

[0005] The main purpose of this application is to provide a method, device, equipment, storage medium and product for identifying similar models, aiming to solve the problem of repeated development of similar models and increased development costs.

[0006] To achieve the above purpose, the present application proposes a similarity model recognition method comprising:

[0007] Get the original business fields of the newly created application model;

[0008] Based on the data lineage relationship between the newly created application model and the pre-model of the newly created application model, identify the pre-model fields that the original business fields depend on, and obtain the standard business fields of the pre-model fields from the full business field standard library;

[0009] The original business fields are updated using the standard business fields to obtain the standard application model of the newly created application model;

[0010] According to the similarity between the standard application model and the previous model, the similarity between the newly created application model and the previous model is determined.

[0011] In one embodiment, based on the data lineage relationship between the newly created application model and the front model of the newly created application model, before the step of identifying the front model field on which the original business field depends, and obtaining the standard business field of the front model field from the full business field standard library, the step further includes:

[0012] Get all business fields of the big data platform;

[0013] Standardizing the business fields of each of the big data platforms to obtain standard business fields of each of the business fields;

[0014] A mapping relationship between the business field and the standard business field is constructed to form the full business field standard library.

[0015] In one embodiment, if the front-end model is an interface layer model, based on the data lineage relationship between the newly created application model and the front-end model of the newly created application model, the steps of identifying the front-end model field on which the original business field depends, and obtaining the standard business field of the front-end model field from the full business field standard library include:

[0016] Based on the data lineage relationship between the newly created application model and the interface layer model, identify the business fields of the interface layer model corresponding to the original business fields;

[0017] Obtain the standard business fields of the business fields of the interface layer model from the full business field standard library;

[0018] The steps of using the standard business fields to update the original business fields and obtain the standard application model of the newly created application model include:

[0019] According to the standard business fields of the business fields of the interface layer model, the original business fields are updated to obtain a standard application model.

[0020] In one embodiment, based on the data lineage relationship between the newly created application model and the front model of the newly created application model, before the step of identifying the front model field on which the original business field depends, and obtaining the standard business field of the front model field from the full business field standard library, the step further includes:

[0021] Configure the dependency relationship between the newly created application model and the pre-model used by it, wherein the pre-model includes an interface layer model and / or an application model, and the interface layer model and the application model are models obtained through historical development;

[0022] Based on the dependency relationship, a data lineage relationship is formed between the newly created application model and the predecessor model of the newly created application model.

[0023] In one embodiment, the step of determining the similarity between the new application model and the previous model according to the similarity between the standard application model and the previous model includes:

[0024] Converting a standard service field corresponding to the standard application model into a first feature vector, and converting a service description corresponding to the standard application model into a second feature vector;

[0025] Calculating the similarity between the first feature vector and the feature vector of the standard business field of each front-end model to obtain a first similarity result, and calculating the similarity between the second feature vector and the feature vector of the standard business field of each front-end model to obtain a second similarity result;

[0026] Determining the similarity between the standard application model and each pre-model according to the first similarity result and the second similarity result;

[0027] The similarity is determined as the similarity between the newly-created application model and the previous model.

[0028] In one embodiment, after the step of determining the similarity between the new application model and the previous model according to the similarity between the standard application model and the previous model, the method further includes:

[0029] If the similarity between the newly created application model and the previous model is greater than or equal to a preset value, a prompt message indicating that the newly created application model is similar to the previous model is output, and similar fields between the newly created application model and the previous model are displayed.

[0030] In addition, to achieve the above purpose, the present application also proposes a similarity model recognition device, comprising:

[0031] The acquisition module is used to obtain the original business fields of the newly created application model;

[0032] An identification module is used to identify the front-end model fields that the original business fields depend on based on the data lineage relationship between the newly created application model and the front-end model of the newly created application model, and obtain the standard business fields of the front-end model fields from the full business field standard library;

[0033] An update module, used to update the original business fields with the standard business fields to obtain a standard application model for a newly created application model;

[0034] The similarity comparison module is used to determine the similarity between the newly created application model and the previous model based on the similarity between the standard application model and the previous model.

[0035] In addition, to achieve the above objectives, the present application also proposes a similarity model recognition device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the similarity model recognition method as described above.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the similarity model recognition method as described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the similarity model recognition method as described above are implemented.

[0038] The present application identifies the fields of the preceding model that the newly created application model depends on through the blood relationship between the newly created application model and the preceding model of the newly created application model. Since the standard business fields of the preceding model fields are known and stored in the standard library of all business fields, the standard business fields corresponding to the preceding model fields can be directly obtained from the standard library of all business fields to update the original business fields of the newly created application model, thereby achieving standardization of the newly created application model. Finally, the similarity between the standard application model and the preceding model is compared to obtain the similarity between the newly created application model and the preceding model. Since the fields corresponding to each model are standardized and the business field caliber is unified, the similarity comparison can more accurately identify whether the current newly created application model is a duplicate model, thereby avoiding repeated development of similar models and reducing development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the method for identifying similar models of the present application;

[0042] Figure 2 A schematic diagram of a flow chart provided for a second embodiment of a method for identifying similar models of the present application;

[0043] Figure 3 This is the blood relationship diagram of the big data platform model for this application;

[0044] Figure 4 A schematic diagram of a flow chart provided for the third embodiment of the method for identifying similar models of the present application;

[0045] Figure 5A schematic diagram of the module structure of a similar model recognition device according to an embodiment of the present application;

[0046] Figure 6 This is a schematic diagram of the structure of a similar model recognition device according to an embodiment of the present application.

[0047] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0049] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0050] In the related art, the data development workflow of the big data platform is as follows: the data developers of each tenant of the big data platform develop models on the big data platform and input relevant standardized model information. The big data developers of the tenants develop programs on the big data platform, solidify the data analysis code, and complete the configuration of program scheduling information. The tenant launches the data analysis program, which is automatically scheduled, submits the analysis statement to the corresponding database for automatic operation, and feeds back the analysis results. Since it is impossible to determine whether the developed model has similar models on the big data platform during data development, there is a situation of repeated development of models similar to existing models, which increases development costs.

[0051] In response to the above problems, the present application proposes a method for identifying similar models, and the main technical solutions include: obtaining the original business fields of the newly created application model; based on the data lineage relationship between the newly created application model and the predecessor model of the newly created application model, identifying the predecessor model fields on which the original business fields depend, and obtaining the standard business fields of the predecessor model fields from the full business field standard library; using the standard business fields to update the original business fields to obtain the standard application model of the newly created application model; determining the similarity between the newly created application model and the predecessor model based on the similarity between the standard application model and the predecessor model.

[0052] The present application identifies the fields of the preceding model that the newly created application model depends on through the blood relationship between the newly created application model and the preceding model of the newly created application model. Since the standard business fields of the preceding model fields are known and stored in the standard library of all business fields, the standard business fields corresponding to the preceding model fields can be directly obtained from the standard library of all business fields to update the original business fields of the newly created application model, thereby achieving standardization of the newly created application model. Finally, the similarity between the standard application model and the preceding model is compared to obtain the similarity between the newly created application model and the preceding model. Since the fields corresponding to each model are standardized and the business field caliber is unified, the similarity comparison can more accurately identify whether the current newly created application model is a duplicate model, thereby avoiding repeated development of similar models and reducing development costs.

[0053] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a similar model recognition device that can realize the above functions, etc. The following takes the similar model recognition device as an example to illustrate this embodiment and the following embodiments.

[0054] It should be noted that the big data platform is not a simple information technology platform that integrates multiple functions such as data storage, data management, data computing, data analysis and data mining. It is more of an operating mechanism that emphasizes resource integration, centralized allocation, capability accumulation and step-by-step execution. It is a collection of a series of data components or modules that empowers the improvement of enterprise data governance efficiency, the upgrading of business processes and organizational structure, and the refinement of operations and decision-making.

[0055] Tenant: usually refers to the user or owner of a cluster, who may exist in the form of projects, different applications or organizations. A tenant is equivalent to a resource pool in the big data platform. The administrator will configure certain cluster resources for the tenant, and all users under the tenant share the resources in these resource pools. Multiple tenants share a big data platform.

[0056] Field Standard Library: refers to a set of standards for regulating, defining, and managing fields in a database, also known as columns. These standards cover key attributes such as field naming, caliber, type, length, precision, default value, whether to allow nulls, and sensitivity level.

[0057] Field standard mapping: refers to the process of associating and mapping fields in different data sources, different systems or different databases according to a set of predefined standards and specifications. This process aims to ensure that the data can be consistent, accurate and understandable when it flows between different environments or systems. Here, field standard mapping usually involves the English name and Chinese name of the field.

[0058] Based on this, the present application embodiment provides a method for identifying a similar model. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the similarity model recognition method of the present application.

[0059] In this embodiment, the similarity model identification method includes steps S10 to S40:

[0060] Step S10, obtaining the original business fields of the newly created application model;

[0061] It should be noted that the newly created application model is an application newly developed by the tenant on the big data platform.

[0062] It should be noted that the original business field may include the Chinese name of the business field and the English name of the business field. The original business field of the newly created application model is customized by the developer of the newly created application model. Due to different development habits of different developers, the same business field is represented differently in different application models. For example, there is a business field whose corresponding field name is "user id". Its English name in one model is "a", and its English name in another model is "e".

[0063] Step S20, based on the data lineage relationship between the newly created application model and the front model of the newly created application model, identify the front model field that the original business field depends on, and obtain the standard business field of the front model field from the full business field standard library;

[0064] It should be noted that the pre-model can be an application model or an interface layer model. The pre-model is a model obtained through historical development, which can be one or more.

[0065] For example, refer to Figure 3 When the newly created application model is the C1 model, its corresponding pre-model can be an application model such as the B1 model, or it can be an A1 model, an A2 model, or an A3 model. When the newly created application model is the B1 model, its corresponding pre-model can be an interface layer model such as the A1 model, an A2 model, or an A3 model.

[0066] It should be noted that there is a data lineage relationship between the new application model and the previous model. Data lineage refers to a complete record of the series of transformations and processing processes that the data has undergone from its original source to its final destination. In short, it describes the source and destination of the data and how the data is modified or integrated during the processing. Therefore, the business fields used in the new application model may come from the business fields of the previous model, that is, the new application model may have the same business fields as the business fields of the previous model.

[0067] In a feasible implementation manner, the data lineage relationship between the newly created application model and the previous model can be obtained through configuration.

[0068] In another feasible implementation, SQL statements may be used to parse and identify the pre-model used by the newly created application model and the business fields of the pre-model.

[0069] In another feasible implementation, since the front-end model is a model obtained from historical development, each business field of the front-end model can be standardized in advance to obtain standard business fields, which are stored in a full business field standard library, and then the standard business fields of the front-end model fields are obtained from the full business field standard library.

[0070] Step S30, using the standard business field to update the original business field to obtain a standard application model for the newly created application model;

[0071] After obtaining the standard business fields of the front-end model fields, the standard business fields of the front-end model fields are used to replace the corresponding original business fields of the newly created application model. Each original business field of the newly created application model can be replaced and updated in the above manner, thereby achieving standardized processing of the newly created application model and obtaining a standard application model of the newly created application model.

[0072] Step S40, determining the similarity between the newly created application model and the previous model according to the similarity between the standard application model and the previous model.

[0073] It should be noted that the similarity here can be cosine similarity, Euclidean distance, etc.

[0074] In a feasible implementation, after converting the newly created application model into a standard application model, the similarity between each standard business field in the standard application model and each standard business field in the previous model can be calculated to further determine the similarity between the newly created application model and the previous model, thereby determining whether the newly created application model is a duplicate and similarly developed model.

[0075] In another feasible implementation, since both the standard application model and the front-end model may have multiple fields, the number of similar fields between the standard application model and each front-end model can be determined, the number of similar fields can be counted, and then the front-end model with the largest number of similar fields can be determined as the similar model of the newly created application model. That is, the front-end model with the largest number of similar fields is the front-end model that is most similar to the newly created application model, thereby achieving accurate judgment on whether the newly created application model is a duplicate similar model.

[0076] In another feasible implementation, during the launch phase or development phase of a newly created application model, if it is detected that a similar previous model exists for the newly created application model, corresponding prompt information or development rectification suggestions are generated to avoid the launch of duplicate application models.

[0077] This embodiment provides a method for identifying similar models. Through the blood relationship between the newly created application model and the previous model of the newly created application model, the previous model field that the newly created application model depends on is identified. Since the standard business field of the previous model field is known and stored in the full business field standard library, the standard business field corresponding to the previous model field can be directly obtained from the full business field standard library to update the original business field of the newly created application model, so as to achieve standardization of the newly created application model. Finally, the similarity between the standard application model and the previous model is compared to obtain the similarity between the newly created application model and the previous model. Since the fields corresponding to each model are standardized and the business field caliber is unified, the similarity comparison can more accurately identify whether the current newly created application model is a duplicate model, avoid repeated development of similar models, and reduce development costs.

[0078] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2 , before step S20, it also includes steps S01 to S03:

[0079] Step S01, obtaining all business fields of the big data platform;

[0080] All business fields of the big data platform can be provided by developers or obtained through analysis based on existing application models.

[0081] Step S02, standardizing the business fields of each of the big data platforms to obtain standard business fields of each of the business fields;

[0082] All acquired business fields may be preprocessed, such as cleaning, merging, deduplication, etc., and synonym and / or fuzzy word analysis may be performed on the preprocessed business fields to determine the standards of each business field and obtain the standard business fields of each business field.

[0083] Step S03: construct a mapping relationship between the business field and the standard business field to form the full business field standard library.

[0084] Standardize the Chinese and English names of the business fields of the big data platform according to business requirements to form a standard library of all business fields.

[0085] It should be noted that the functions of the full business field standard library include: providing unified field standardization information for each tenant's data development. Providing a field naming dictionary to help users register word roots and affixes to achieve the standardization of field commands. Supporting the creation, modification, deletion, and process approval functions of field standards to ensure the compliance and correctness of field standards. Providing visualization functions for field standards and data dictionaries to facilitate user understanding and use. Outputting field standards for model field standardization work.

[0086] In this embodiment, a full business field standard library is constructed according to the mapping relationship between business fields and standard business fields. By formulating a full business field standard library, it is convenient to perform standard processing of business fields automatically, so that the subsequently developed business fields can be developed according to the standard.

[0087] Furthermore, if the front-end model is an interface layer model, step S20 includes:

[0088] Step S21, based on the data lineage relationship between the newly created application model and the interface layer model, identifying the business field of the interface layer model corresponding to the original business field;

[0089] It should be noted that the interface layer model is the big data platform tenant interface layer model, and the big data platform tenant interface layer model is the data source of all data of the big data platform.

[0090] The big data platform collects metadata information of each interface layer model, including the Chinese and English names of the interface layer model corresponding to the business, and the Chinese and English names of the business fields.

[0091] In a feasible implementation, before step S21, the interface metadata information can be combined with the field standard information of the full business field standard library to enter the standard business field behind the original business field, including the English name of the standard business field and the Chinese name of the standard business field, to form a standard field mapping and realize the standardization of the interface layer model of the big data platform.

[0092] Specifically, a standard business field matching the original business field of the interface layer model is searched from the full business field standard library, and the original business field of the interface layer model is updated using the standard business field to obtain the business field of the interface layer model. The original business field of the interface layer model includes the Chinese and English names of the interface layer model and the Chinese and English names of the original business field. Based on these data, the corresponding standard business field can be searched from the full business field standard library.

[0093] For example, the big data platform collects metadata information of each interface layer model, collects the English name of the model field corresponding to the business, and the Chinese and English name of the field, and completes the standardization of the business field of the interface layer model based on the combination of the full business field standard library and the big data platform model library function. The standard business fields of the output interface layer model include: English name of the interface layer model, Chinese name of the interface layer model, English name of the business field, Chinese name of the business field, English name of the standardized mapped business field, and Chinese name of the standardized mapped business field. Figure 3 For example, the A1 interface layer model completes the standardization of the four business fields a, b, c, and d of the A1 interface layer model, and records the standardized information of the A1 interface layer model in the big data platform model library.

[0094] It should be noted that the main functions of the above-mentioned big data platform model library include: providing a unified model development management tool for each tenant during data development. Recording the standardized information of the big data platform model, including the English name of the model, the Chinese name of the model, the English name of the field, the Chinese name of the field, the English name of the standardized mapping field, the Chinese name of the standardized mapping field, etc. The model here can be an interface layer model, an application model, or other models developed on the big data platform. Output the standardized information of the model field.

[0095] It should be noted that the data lineage relationship between the newly created application model and the interface layer model can be obtained through configuration. When each tenant develops data on the big data platform, it configures the dependency relationship between its newly created application model and the used pre-model on the big data platform, thereby forming the data lineage relationship between the newly created application model and the interface layer model, as well as the data lineage relationship between each newly created application model and other application models.

[0096] In a feasible implementation, a dependency relationship is configured between a newly created application model and a predecessor model used therein, wherein the predecessor model includes an interface layer model and / or an application model, and the interface layer model and the application model are models obtained through historical development; based on the dependency relationship, a data lineage relationship is formed between the newly created application model and the predecessor model of the newly created application model.

[0097] Reference Figure 3 According to the data lineage, we start to sort out and identify the interface layer model, find the subsequent B1 model of the A1 interface layer model, which can be regarded as a new application model, identify the predecessor model of the B1 model as the A1, A2, and A3 models, and identify the business fields of the current B1 model as the corresponding business fields in the A1, A2, and A3 models through SQL analysis. This forms the data lineage relationship between the business fields of the B1 model and the business fields of the A1, A2, and A3 models.

[0098] Step S22, obtaining standard business fields of the business fields of the interface layer model from the full business field standard library.

[0099] Identify which business fields in the B1 model are the corresponding business fields in the A1, A2, and A3 models. For example, if the business fields of the B1 model are A1.a, A2.aa, A3.g, and B1.j, further identify through SQL parsing that the A1.a field in the B1 model comes from the A1 model, the A2.aa field comes from the A2 model, and the A3.g field comes from the A3 model. Identify that the original business fields of the B1 model come from the business fields of the A1, A2, and A3 models. Then directly record the original business fields in the B1 model corresponding to the standard business fields corresponding to A1, A2, and A3, including the English name and Chinese name of the field after field standardization mapping, and mark the standard business field corresponding to the original business field of the B1 model. Because the A1, A2, and A3 models have completed field standardization, the mapping field information is output. The B1 model only needs to identify which corresponding business fields are derived from the A1, A2, and A3 models, and then obtain the field standardization information derived from the A1, A2, and A3 models and substitute it into the B1 model, thereby achieving the effect of obtaining the standard business fields of the business fields of the interface layer model.

[0100] Further, step S30 includes:

[0101] Step S31, updating the original business field according to the standard business field of the business field of the interface layer model to obtain a standard application model.

[0102] After obtaining the standard business fields of the business fields of the interface layer model, the corresponding original business fields of the newly created application model are updated using the standard business fields, thereby obtaining a standard application model.

[0103] For example, the standard business fields of the B1 model identified and processed by the field standardization device, including the standard business field English name and the standard business field Chinese name, are output to the big data platform model library, and the field mapping English name and field mapping Chinese name of the B1 model are added by the big data platform model library to finally form the standard application model of the B1 model. The table information of the B1 model includes the field English name, the field Chinese name, the standardized mapping field English name, and the standardized mapping field Chinese name, as shown in the following table:

[0104]

[0105]

[0106]

[0107]

[0108] In a feasible implementation, if there are original business fields in the newly created application model that do not belong to the interface layer model, the original business fields can be associated and identified with standard business fields in the full business field standard library, and the identification results can be manually confirmed by business personnel.

[0109] For example, refer to Figure 3 If the B1.j field of the B1 model is not a business field in the A1, A2, and A3 models, the Chinese name of the field in the B1 model is associated with the Chinese name of the field in the full business field standard library for identification, and the identification result is manually confirmed by the business personnel. If it matches, the standardized mapping information of the B1.j field is recorded. If it does not match, the business personnel will manually standardize the B1.j field and enter the standard business field of the B1.j field into the full business field standard library.

[0110] In this embodiment, based on the data lineage relationship between the newly created application model and the interface layer model, the business fields of the interface layer model corresponding to the original business fields are identified, and the standard business fields of the business fields of the interface layer model are obtained from the full business field standard library, thereby achieving the effect of standardizing the original business fields of the newly created application model.

[0111] Based on the above embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 4 , step S40 comprises:

[0112] Step S41, converting the standard service field corresponding to the standard application model into a first feature vector, and converting the service description corresponding to the standard application model into a second feature vector;

[0113] Step S42, calculating the similarity between the first feature vector and the feature vector of the standard business field of each front model to obtain a first similarity result, and calculating the similarity between the second feature vector and the feature vector of the standard business field of each front model to obtain a second similarity result;

[0114] Step S43, determining the similarity between the standard application model and each pre-model according to the first similarity result and the second similarity result;

[0115] Step S44: determine the similarity as the similarity between the newly created application model and the previous model.

[0116] The similarity calculation between the standard application model and each pre-model is completed through NLP technology. The corresponding bag-of-words model + TF-IDF corpus is constructed using the standard business fields corresponding to the standard application model and the standard business fields of the pre-model. The target data is input, brought into the bag-of-words model to calculate the cosine similarity, and the results other than the data itself are output. At the same time, the corpus is constructed using the business description, the target data is input, brought into the corpus to calculate the cosine similarity, and the similarity results other than the data itself are output. The two results are combined to represent the result, and the similarity between the standard application model and each pre-model is output.

[0117] It should be noted that NLP technology is used to evaluate the similarity between different models. NLP technology is good at processing and understanding text data, so it can be applied to analyze model descriptions, field names, etc. to evaluate the similarity between them.

[0118] The bag-of-words model is a simple text representation method that treats text as an unordered collection of words, ignoring the order and grammatical structure of words. The number of occurrences of each word in the text (or the number of occurrences after some weight adjustment) is used as the feature of the word in the text. TF-IDF is a commonly used text feature extraction method used to evaluate the importance of a word in a document. TF represents the frequency of occurrence of a word in a document, and IDF is used to reduce the importance of words that appear frequently in all documents.

[0119] In this step, we first build a bag-of-words model based on the model's field name and description information, and then use the TF-IDF method to adjust the weights of the words to form a TF-IDF corpus.

[0120] In a feasible implementation, the target data is input and brought into the bag-of-words model to calculate the cosine similarity. The target data here refers to the model whose similarity needs to be evaluated. The field names and descriptions of these models are input into the constructed bag-of-words model, and the feature vectors of each model are calculated using the TF-IDF weights. Use cosine similarity to calculate the similarity between these feature vectors. Cosine similarity is an indicator that measures the directional similarity of two feature vectors. The value range is [-1,1]. The larger the value, the more similar the two vectors are. After calculating the similarity of all models, the similarity of each model to itself is excluded, which is usually 1 or a maximum value close to 1, and only the similarity results with other models are output.

[0121] In addition to similarity evaluation based on field names and descriptions, you can also use the business descriptions of the models to build another corpus. This corpus may contain richer contextual information, which helps to more accurately evaluate the similarity between models. Similarity calculation is performed based on the corpus built based on the business description.

[0122] The similarity result calculated based on the field name and description is combined with the similarity result calculated based on the business description in some form, such as weighted average, maximum value, etc., to obtain the final similarity evaluation result.

[0123] Finally, the similarity results between the target data and all other models are output. These results may be presented in lists, matrices, or other forms for easy analysis and comparison.

[0124] In this embodiment, the similarity between different models is evaluated by combining NLP technology, bag-of-words model, TF-IDF method and cosine similarity calculation to assist users in identifying and analyzing similar models.

[0125] Based on the above embodiments of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. On this basis, after step S40, it also includes:

[0126] Step S110: If the similarity between the newly created application model and the previous model is greater than or equal to a preset value, a prompt message indicating that the newly created application model is similar to the previous model is output, and similar fields between the newly created application model and the previous model are displayed.

[0127] It should be noted that the preset value can be set or adjusted according to actual conditions.

[0128] During the model development stage and the model launch stage, data model similarity identification prompts are provided to display the similarity between the newly created application model and the previous model, as well as specific field comparison information, thereby reducing the duplication rate of model development.

[0129] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the identification method of similar models of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0130] This application also provides a similar model recognition device, please refer to Figure 5 , the similar model recognition device includes:

[0131] The acquisition module 10 is used to acquire the original business fields of the newly created application model;

[0132] An identification module 20 is used to identify the front-end model field that the original business field depends on based on the data relationship between the newly created application model and the front-end model of the newly created application model, and obtain the standard business field of the front-end model field from the full business field standard library;

[0133] An updating module 30, configured to update the original business field with the standard business field to obtain a standard application model of the newly created application model;

[0134] The similarity comparison module 40 is used to determine the similarity between the newly created application model and the previous model according to the similarity between the standard application model and the previous model.

[0135] Optionally, the similarity model recognition device is also used to obtain all business fields of the big data platform; standardize each business field of the big data platform respectively to obtain a standard business field of each business field; construct a mapping relationship between the business field and the standard business field to form a full business field standard library.

[0136] Optionally, the identification module 20 is also used to identify the business fields of the interface layer model corresponding to the original business fields based on the data lineage relationship between the newly created application model and the interface layer model; and obtain the standard business fields of the business fields of the interface layer model from the full business field standard library.

[0137] Optionally, the updating module 30 is further configured to update the original business field according to the standard business field of the business field of the interface layer model to obtain a standard application model.

[0138] Optionally, the similar model identification device is also used to configure the dependency relationship between the newly created application model and the predecessor model used by it, wherein the predecessor model includes an interface layer model and / or an application model, and the interface layer model and the application model are models obtained through historical development; based on the dependency relationship, a data lineage relationship is formed between the newly created application model and the predecessor model of the newly created application model.

[0139] Optionally, the similarity comparison module 40 is also used to convert the standard business field corresponding to the standard application model into a first feature vector, and to convert the business description corresponding to the standard application model into a second feature vector; calculate the similarity between the first feature vector and the feature vector of the standard business field of each front-end model to obtain a first similarity result, and calculate the similarity between the second feature vector and the feature vector of the standard business field of each front-end model to obtain a second similarity result; determine the similarity between the standard application model and each front-end model based on the first similarity result and the second similarity result; and determine the similarity as the similarity between the newly created application model and the front-end model.

[0140] Optionally, the similar model identification device is also used to output a prompt message that the new application model is similar to the previous model if the similarity between the new application model and the previous model is greater than or equal to a preset value, and display the similar fields between the new application model and the previous model.

[0141] The similar model recognition device provided by the present application adopts the similar model recognition method in the above-mentioned embodiment, which can solve the problem of repeated development of similar models and increased development costs. Compared with the prior art, the beneficial effects of the similar model recognition device provided by the present application are the same as the beneficial effects of the similar model recognition method provided by the above-mentioned embodiment, and other technical features in the similar model recognition device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0142] The present application provides a similarity model recognition device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the similarity model recognition method in the above-mentioned embodiment one.

[0143] Reference below Figure 6 , which shows a schematic diagram of the structure of a similar model recognition device suitable for implementing the embodiment of the present application. The similar model recognition device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The identification device of the similar model shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0144] like Figure 6As shown, the similar model recognition device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the similar model recognition device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the recognition device of similar models to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows the recognition device of similar models with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0145] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0146] The similar model recognition device provided by the present application adopts the similar model recognition method in the above embodiment, which can solve the problem of repeated development of similar models and increased development costs. Compared with the prior art, the beneficial effects of the similar model recognition device provided by the present application are the same as the beneficial effects of the similar model recognition method provided by the above embodiment, and the other technical features in the similar model recognition device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0147] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0148] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0149] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the similarity model recognition method in the above-mentioned embodiment.

[0150] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0151] The computer-readable storage medium may be included in the similar model recognition device; or may exist independently without being assembled into the similar model recognition device.

[0152] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the similar model recognition device, the similar model recognition device enables the following: to obtain the original business fields of the newly created application model; based on the data lineage relationship between the newly created application model and the previous model of the newly created application model, to identify the previous model fields on which the original business fields depend, and to obtain the standard business fields of the previous model fields from the full business field standard library; to update the original business fields with the standard business fields to obtain the standard application model of the newly created application model; and to determine the similarity between the newly created application model and the previous model based on the similarity between the standard application model and the previous model.

[0153] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0155] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0156] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned similar model recognition method, and can solve the problem of repeated development of similar models and increased development costs. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the similar model recognition method provided by the above-mentioned embodiment, and will not be repeated here.

[0157] The present application also provides a computer program product, including a computer program, which implements the steps of the similarity model recognition method as described above when the computer program is executed by a processor.

[0158] The computer program product provided by the present application can solve the technical problem of repeated development of similar models and increased development costs. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the similar model identification method provided by the above embodiment, which will not be repeated here.

[0159] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for identifying a similar model, characterized in that: The method comprises: Get the original business fields of the newly created application model; Based on the data lineage relationship between the newly created application model and the front-end model of the newly created application model, identifying the front-end model field on which the original business field depends, and acquiring the standard business field of the front-end model field from the full business field standard library; The standard business field is used to update the original business field to obtain a standard application model of the newly created application model; According to the similarity between the standard application model and the pre-model, the similarity between the newly created application model and the pre-model is determined.

2. The method according to claim 1, characterized in that Before the step of identifying the front model field on which the original business field depends based on the data lineage relationship between the newly created application model and the front model of the newly created application model, and acquiring the standard business field of the front model field from the full business field standard library, the step further includes: Get all business fields of the big data platform; Standardizing the business fields of each of the big data platforms to obtain standard business fields of each of the business fields; A mapping relationship between the business field and the standard business field is constructed to form the full business field standard library.

3. The method according to claim 2, characterized in that If the front-end model is an interface layer model, the step of identifying the front-end model field on which the original business field depends based on the data lineage relationship between the newly created application model and the front-end model of the newly created application model, and acquiring the standard business field of the front-end model field from the full business field standard library includes: Based on the data lineage relationship between the newly created application model and the interface layer model, identify the business fields of the interface layer model corresponding to the original business fields; Acquire the standard business fields of the business fields of the interface layer model from the full business field standard library; The step of using the standard business field to update the original business field to obtain the standard application model of the newly created application model includes: The original service field is updated according to the standard service field of the service field of the interface layer model to obtain the standard application model.

4. The method according to claim 1, characterized in that Before the step of identifying the front model field on which the original business field depends based on the data lineage relationship between the newly created application model and the front model of the newly created application model, and acquiring the standard business field of the front model field from the full business field standard library, the step further includes: Configuring a dependency relationship between the newly created application model and a pre-model used by the newly created application model, wherein the pre-model includes an interface layer model and / or an application model, and the interface layer model and the application model are models obtained through historical development; According to the dependency relationship, a data lineage relationship between the newly created application model and a pre-model of the newly created application model is formed.

5. The method according to any one of claims 1 to 4, characterized in that: The step of determining the similarity between the newly created application model and the pre-model according to the similarity between the standard application model and the pre-model comprises: Converting a standard service field corresponding to the standard application model into a first feature vector, and converting a service description corresponding to the standard application model into a second feature vector; Calculating the similarity between the first feature vector and the feature vector of the standard business field of each of the front models to obtain a first similarity result, and calculating the similarity between the second feature vector and the feature vector of the standard business field of each of the front models to obtain a second similarity result; Determining the similarity between the standard application model and each of the pre-models according to the first similarity result and the second similarity result; The similarity is determined as the similarity between the newly created application model and the previous model.

6. The method according to any one of claims 1 to 4, characterized in that: After the step of determining the similarity between the new application model and the pre-model according to the similarity between the standard application model and the pre-model, the method further includes: If the similarity between the newly created application model and the previous model is greater than or equal to a preset value, a prompt message indicating that the newly created application model is similar to the previous model is output, and similar fields between the newly created application model and the previous model are displayed.

7. A similarity model recognition device, characterized in that: The device comprises: The acquisition module is used to obtain the original business fields of the newly created application model; An identification module, configured to identify the front-end model field on which the original business field depends based on the data lineage relationship between the newly created application model and the front-end model of the newly created application model, and obtain the standard business field of the front-end model field from a full business field standard library; An updating module, configured to update the original business field with the standard business field to obtain a standard application model for the newly created application model; The similarity comparison module is used to determine the similarity between the newly created application model and the pre-model according to the similarity between the standard application model and the pre-model.

8. A similarity model recognition device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the similarity model recognition method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the similarity model recognition method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the similarity model identification method according to any one of claims 1 to 6 are implemented.

Images