Application interface openness maturity evaluation method and device and electronic equipment

Abstract

The invention discloses an application interface openness maturity evaluation method and device and electronic equipment. The method comprises the following steps: acquiring an interface document, work order data and log data of a to-be-evaluated product, as well as an evaluation specification and an interface document template; based on the evaluation specification and the interface document template, performing evaluation processing on the interface document, and obtaining a first opening capability evaluation result of each application interface in the to-be-evaluated product; based on the work order data, obtaining a second openness assessment result of each application interface in the to-be-assessed product; performing statistical analysis processing on the log data to obtain a third opening capability evaluation result of each application interface in the to-be-evaluated product; and fusing the first opening capability evaluation result, the second opening capability evaluation result and the third opening capability evaluation result to obtain an application interface opening capability maturity evaluation result of the to-be-evaluated product. According to the method, more assessment points can be covered, so that the comprehensiveness of capability maturity assessment is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computers, and particularly relates to an application interface opening capability maturity evaluation method, an application interface opening capability maturity evaluation device, an electronic device and a readable storage medium. BACKGROUND

[0002] Cloud is an important infrastructure in the construction of national strategy and is a guarantee for the stability of digital society operation. As an infrastructure, cloud has been sinking rapidly since its inception and needs to be quickly integrated into industry scenarios. The demand for cloud is more diverse in the industry. For example, industry customized solutions are constantly upgraded. With the continuous enrichment of cloud products and the continuous clouding of various industry scenarios, customers' demand for cloud changes from complete solutions to customized solutions, and the requirements for product capabilities, performance and opening capabilities are also more explicit. Mature and complete opening capabilities are one of the core competencies of future cloud vendors. Evaluating the opening capability maturity of cloud products is a necessary operation for cloud product opening capability. However, in the prior art, due to the constant changes in the client call environment, the evaluation of the opening capability of cloud products can only define general rules. Moreover, due to the combination of application interface documents, examples, ecological tools and communities in the opening capability of cloud capability, the objective rule evaluation method cannot cover all the evaluation points. That is, the existing cloud product opening capability evaluation method can only evaluate the baseline capability and cannot evaluate the advancement of the opening capability, and cannot evaluate all the opening capabilities.

[0003] Therefore, the application interface opening capability maturity evaluation method in the prior art still needs to be improved. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide an application interface opening capability maturity evaluation method which can comprehensively evaluate the application interface opening capability from multiple evaluation points.

[0005] In order to solve the above technical problems, the present application is implemented as follows: In a first aspect, the embodiments of the present application provide an application interface opening capability maturity evaluation method, which comprises: obtaining interface documents, work order data and log data of a product to be evaluated, and evaluation specifications and interface document templates; based on the evaluation specifications and the interface document templates, performing evaluation processing on the interface documents to obtain first opening capability evaluation results of each application interface in the product to be evaluated; based on the work order data, obtaining second opening capability evaluation results of each application interface in the product to be evaluated; performing statistical analysis processing on the log data to obtain third opening capability evaluation results of each application interface in the product to be evaluated; fuse the first open capability evaluation result, the second open capability evaluation result and the third open capability evaluation result to obtain an application interface open capability maturity evaluation result of the product to be evaluated.

[0006] In a second aspect, an embodiment of the present application provides an application interface open capability maturity evaluation device, and the device comprises: an input information acquisition module, configured to acquire interface documents, work order data and log data of a product to be evaluated, and evaluation specifications and interface document templates; a first evaluation module, configured to perform evaluation processing on the interface documents based on the evaluation specifications and the interface document templates, and acquire first open capability evaluation results of each application interface in the product to be evaluated; a second evaluation module, configured to acquire second open capability evaluation results of each application interface in the product to be evaluated based on the work order data; a third evaluation module, configured to perform statistical analysis processing on the log data, and acquire third open capability evaluation results of each application interface in the product to be evaluated; an evaluation result fusion module, configured to fuse the first open capability evaluation result, the second open capability evaluation result and the third open capability evaluation result to obtain an application interface open capability maturity evaluation result of the product to be evaluated.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and the program or instruction is executed by the processor to implement the steps of the method in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, and the readable storage medium stores a program or instruction, and the program or instruction is executed by a processor to implement the steps of the method in the first aspect.

[0009] The application interface open capability maturity evaluation method disclosed in the embodiments of the present application has the following advantages: By acquiring interface documents, work order data and log data of a product to be evaluated, and evaluation specifications and interface document templates, and performing evaluation processing on the interface documents based on the evaluation specifications and the interface document templates, first open capability evaluation results of each application interface in the product to be evaluated are acquired, which are used to characterize interface integrity of the product to be evaluated from the interface capability dimension; based on the work order data, second open capability evaluation results of each application interface in the product to be evaluated are acquired, which realize characterization of user satisfaction of the product to be evaluated based on community, ecological tool and other data; the log data are statistically analyzed and processed to acquire third open capability evaluation results of each application interface in the product to be evaluated, so as to evaluate execution performance of the product to be evaluated based on example call data; finally, the first open capability evaluation results, the second open capability evaluation results and the third open capability evaluation results are fused to obtain application interface open capability maturity evaluation results of the product to be evaluated, which comprehensively evaluate application interface open capability maturity of the product to be evaluated from multiple dimensions, can cover more examination points, and thus improve comprehensiveness of application interface open capability maturity evaluation of the product to be evaluated. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a step flow chart of an application interface open capability maturity evaluation method provided by the embodiment of the application; Figure 2 is an interface document evaluation principle schematic diagram in the application interface open capability maturity evaluation method provided by the embodiment of the application; Figure 3 is a model training method schematic diagram provided by the embodiment of the application; Figure 4 is a device structure schematic diagram of the application interface open capability maturity evaluation device provided by the embodiment of the application; Figure 5 a block diagram of an electronic device for performing the method according to the application is schematically shown; and Figure 6 a storage unit for holding or carrying program code for implementing the method according to the application is schematically shown. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the application.

[0012] The application provides a method for evaluating the opening capability maturity of an application interface based on machine learning. On the basis of a standard specification general scoring model, combined with work order data and calling data, the access, running and user experience of the application interface are comprehensively evaluated to obtain the capability maturity evaluation result of the application interface, so as to continuously improve product documents, content communities and services.

[0013] The application will be described in detail below with reference to the accompanying drawings and specific embodiments and application scenarios.

[0014] As shown in the figure, the method for evaluating the opening capability maturity of an application interface disclosed by the application comprises steps 102 to 110. Figure 1

[0015] Step 102: Obtain the interface document, work order data and log data of the product to be evaluated, and evaluation specifications and interface document templates.

[0016] The work order data and the log data are obtained after data cleaning. The data cleaning operation includes but is not limited to one or more of the following: deleting invalid data, data deduplication processing, etc. For the specific implementation of data cleaning of the work order data and the log data, please refer to the prior art, which will not be described here in the application.

[0017] The product to be evaluated includes but is not limited to a cloud product. The work order data includes but is not limited to customer service data associated with the product to be evaluated. The log data includes but is not limited to calling data of the product to be evaluated. The evaluation specifications include but are not limited to industry specifications, enterprise specifications, requirements that the product interface needs to meet, evaluation items that need to be performed on the interface document, evaluation methods, etc. that need to be followed during the development and / or release of the product to be evaluated. Each requirement or specification in the evaluation specification is represented by a to-be-evaluated document sub-item. The interface document template is a template that needs to be followed for the interface document developed and / or released by the product to be evaluated.

[0018] The specific implementation of obtaining the interface document, work order data and log data of the product to be evaluated, and the evaluation specifications and interface document templates in the embodiments of the application is not limited.

[0019] Next, the opening capability maturity of the product to be evaluated is evaluated from multiple dimensions based on the data obtained in this step.

[0020] Step 104: Based on the evaluation specifications and the interface document templates, the interface document is evaluated and processed to obtain the first opening capability evaluation result of each application interface in the product to be evaluated.

[0021] ​Optionally, based on the evaluation specification and the interface document template, the interface document is evaluated to obtain the first open capability evaluation result of each application interface in the product to be evaluated, including: based on the evaluation specification and the interface document template, the interface document is parsed and annotated to obtain an application interface document area in the interface document, which is annotated with information of a document sub-item to be evaluated, wherein the information of the document sub-item to be evaluated includes: a first document sub-item to be evaluated, and / or a second document sub-item to be evaluated and an automatic evaluation method corresponding to the second document sub-item to be evaluated; for the application interface document area annotated with the second document sub-item to be evaluated, the corresponding automatic evaluation method is used to verify the application interface document area respectively to obtain the evaluation result corresponding to each second document sub-item to be evaluated; for the application interface document area annotated with the first document sub-item to be evaluated, based on the interface document, a preset document association feature is extracted, and the preset document association feature is evaluated by a preset first neural network model corresponding to each first document sub-item to be evaluated respectively to obtain the evaluation result corresponding to the first document sub-item to be evaluated; and the evaluation result corresponding to the first document sub-item to be evaluated and the evaluation result corresponding to the second document sub-item to be evaluated are integrated to obtain the first open capability evaluation result of each application interface in the product to be evaluated.

[0022] In the implementation process of the present application, the evaluation results of a plurality of document sub-items to be evaluated are integrated to obtain the first open capability evaluation result of each application interface in the product to be evaluated. Among the plurality of document sub-items to be evaluated, part of the document sub-items to be evaluated can be directly evaluated by using the automatic evaluation method in the evaluation specification, so as to obtain the evaluation result of the document sub-item to be evaluated based on the evaluation specification, and another part of the document sub-items to be evaluated need to be evaluated by combining machine learning. Finally, the evaluation results of the document sub-items to be evaluated obtained by using different evaluation methods are integrated to obtain the first open capability evaluation result of each application interface in the product to be evaluated.

[0023] In the embodiment of the present application, before evaluating the document sub-item, it is first determined which application interface document area is included in the interface document, and then it is determined which specification or specifications need to be used for evaluation according to the evaluation specification for each application interface document area, that is, it is determined that the document sub-item to be evaluated needs to be executed on the application interface document area, and what method is used for evaluation. Among them, the evaluation method includes the automatic evaluation method specified by the evaluation specification for the document sub-item to be evaluated, or the machine learning method is used.

[0024] Optionally, the step of parsing and annotating the interface document based on the evaluation specifications and the interface document template to obtain an application interface document area in the interface document annotated with information of the document sub-items to be evaluated includes: parsing the interface document according to the interface document template to obtain evaluation tags for the application interface document area in the interface document; performing structured processing on the evaluation specifications to obtain the document sub-items to be evaluated covered by each specification and / or the automatic evaluation method adapted to the document sub-items to be evaluated; and annotating the application interface document area based on the matching results of the evaluation tags of the application interface document area and the specifications to obtain an application interface document area annotated with information of the document sub-items to be evaluated.

[0025] Optionally, based on the matching results of the evaluation tags of the application interface document area and the specification, the application interface document area is annotated to obtain an application interface document area annotated with information of document sub-items to be evaluated. This includes: matching the evaluation tags of the application interface document area with each specification in the evaluation specification to determine the specification used to evaluate the application interface document area; and annotating the application interface document area with information of document sub-items to be evaluated using the specification used to evaluate the application interface document area to obtain an application interface document area annotated with information of document sub-items to be evaluated.

[0026] The following is combined with Figure 2 This paper describes a specific implementation method for obtaining the first document sub-item to be evaluated, the second document sub-item to be evaluated, and the automatic evaluation method corresponding to each of the second document sub-items to be evaluated in the interface document.

[0027] First, the interface document is parsed according to the interface document template to obtain the application interface document area. This area may include information such as: functional interface introduction, preparation work, interface constraints, URI (Universal Resource Identifier), request parameters, output parameters, and error codes. Then, each obtained application interface document area is labeled with an evaluation tag. Figure 2 Taking the application interface document area labeling diagram shown as an example, the evaluation label for the application interface document area "Functional Interface Introduction" is API-INDUST, and the evaluation label for the application interface document area "Preparation Work" is API-READY.

[0028] On the other hand, the evaluation specifications are structured to obtain each specification in the evaluation specifications. Each specification includes, but is not limited to, one or more of the following: evaluation tags, covered document sub-items to be evaluated, and the automatic evaluation method used for the document sub-items to be evaluated.Figure 2 Taking the illustrated diagram of the application interface document area as an example, the evaluation label for the functional interface description in the evaluation specification is API-INDUST. The document sub-items to be evaluated covered by this specification are api-adjust-01, api-adjust-02, and api-adjust-03. For each document sub-item to be evaluated, the parsed structured data also includes the evaluation content and automatic evaluation method of each document sub-item to be evaluated.

[0029] In practice, the evaluation specifications can be segmented item by item, and the segmentation results can be denoised to obtain the segmentation results for each specification. Then, the segmentation results are formatted and converted to obtain the evaluation tags associated with each specification, the document sub-items to be evaluated covered by the specification, and the automatic evaluation method used for each document sub-item. The document sub-items to be evaluated covered by each specification and the automatic evaluation method used for each document sub-item are determined by the specification. The evaluation tags associated with the specification are associated with the evaluation tags in the application interface document area of ​​the interface document template. The automatic evaluation method includes, but is not limited to, information such as evaluation content and scoring methods. For example, the evaluation specification may include the document sub-item to be evaluated, "whether the HTTP response code conforms to the standard," and the automatic evaluation method for that document sub-item, such as "whether the definition of the return code conforms to the standard using Spectral verification."

[0030] Next, the evaluation tags of the application interface document area and the specification items are matched to determine the specification used to evaluate the application interface document area. For example, the evaluation tags of the application interface document area and the evaluation tags of the specification can be encoded first. Then, vector association analysis is performed on the encoding to determine the target specification followed by each application interface document area. Taking the application interface document area and specification mentioned above as an example, it can be determined that the document sub-items to be evaluated covered by the specification associated with the application interface document area "Functional Interface Introduction" include: api-adjust-001, api-adjust-002, and api-adjust-003. Then, according to the document sub-items to be evaluated covered by the target specification and the automatic evaluation method corresponding to the document sub-items to be evaluated, the corresponding application interface document areas are labeled.

[0031] In embodiments of this application, the document sub-item to be evaluated includes: a first document sub-item to be evaluated and a second document sub-item to be evaluated. The second document sub-item to be evaluated is the document sub-item to be evaluated that can output an evaluation score through the automatic evaluation method. For example, the evaluation content of a document sub-item to be evaluated may be: whether the application interface document area is empty. If it is empty, the evaluation result of the document sub-item to be evaluated is 0 points; otherwise, the evaluation result of the document sub-item to be evaluated is 1 point. In this case, the document sub-item to be evaluated can be the first document sub-item to be evaluated. As another example, the evaluation content of a document sub-item to be evaluated may be: whether the application interface document area contains special characters. If it does, the evaluation result of the document sub-item to be evaluated is 0 points; otherwise, the evaluation result of the document sub-item to be evaluated is 1 point. In this case, the document sub-item to be evaluated can be the second document sub-item to be evaluated. Furthermore, for a document sub-item to be evaluated, if the evaluation method is empty, or if the evaluation content is: the quality of the application interface document area, then the document sub-item to be evaluated can be the second document sub-item to be evaluated.

[0032] After annotating the document sub-items to be evaluated in the application interface document area, the next step is to select an appropriate evaluation method based on the annotation results and evaluate the application interface document area.

[0033] For example, for each application interface document region marked with the second document sub-item to be evaluated, the application interface document region is evaluated using an automatic evaluation method corresponding to each of the second document sub-items to be evaluated marked in the application interface document region, to obtain the evaluation results corresponding to each of the second document sub-items to be evaluated marked in the application interface document region. Specific implementation methods for evaluating the application interface document region using a specified automatic evaluation method are described in the prior art and will not be repeated in this embodiment.

[0034] For example, for each application interface document region labeled with the first document sub-item to be evaluated, firstly, preset document association features are extracted based on the interface document. Then, the preset document association features are evaluated by a preset first neural network model corresponding to each of the first document sub-items to be evaluated labeled in the application interface document region, so as to obtain the evaluation results corresponding to each of the first document sub-items to be evaluated labeled in the application interface document region.

[0035] In the embodiments of this application, for each document sub-item to be evaluated, an evaluation task can be created to obtain a task list including several evaluation tasks. Then, the evaluation tasks of the task list are executed using a distributed method to obtain the evaluation result of each application interface document area based on the document sub-item to be evaluated.

[0036] Finally, for each application interface document area, the evaluation results of all the second document sub-items to be evaluated marked in the application interface document area, as well as the evaluation results of all the first document sub-items to be evaluated marked in the application interface document area, can be combined to form the evaluation result of the application interface document area.

[0037] Furthermore, by combining the evaluation results of all application interface documentation areas, the first open capability evaluation result of the application interfaces in the product to be evaluated is obtained.

[0038] In the embodiments of this application, a first neural network model is pre-trained for each first document sub-item to be evaluated.

[0039] Optionally, the preset first neural network model is trained using the following method: extracting preset document association features from the interface documents of the evaluated products to construct sample data; constructing sample labels for the corresponding sample data based on the capability maturity assessment results of the interface documents; and training the preset first neural network model for evaluating the corresponding document sub-items to be evaluated based on the sample data and the sample labels. The sample labels include, but are not limited to, the evaluation scores of the corresponding document sub-items to be evaluated in the interface documents of the evaluated products.

[0040] The types of preset document association features are obtained by performing association analysis on various metadata dimensions in the interface document with preset open capability evaluation indicators (such as the success rate and time consumption of the product to be evaluated), and are metadata dimensions that may maximize the impact on the open capability evaluation indicators. The metadata dimensions to be analyzed are determined according to application requirements.

[0041] The specific implementation of training the preset first neural network model for evaluating the corresponding document sub-items based on the sample data and the sample labels is described in the prior art and will not be repeated in the embodiments of this application.

[0042] By evaluating interface documents based on multiple document sub-items to be evaluated, and using a combination of standardization and machine learning, we can not only assess the baseline functionality of the interface documents (e.g., by using the automatic evaluation method specified in the standard to evaluate according to the evaluation content specified in the standard), but also assess the advancement of the interface documents (e.g., by using machine learning technology to evaluate the document sub-items to be evaluated that characterize the advancement of the interface documents).

[0043] Step 106: Based on the work order data, obtain the second open capability evaluation results of each application interface in the product to be evaluated.

[0044] Work order data is often non-standardized, making accurate and comprehensive evaluation based on rules impossible. In this application's embodiments, machine learning is used to obtain the second open capability evaluation results of each application interface in the product to be evaluated based on the work order data.

[0045] Optionally, based on the work order data, obtaining the second open capability evaluation result of each application interface in the product to be evaluated includes: extracting a first preset work order data association feature and a second preset work order data association feature of each application interface based on the work order data; performing classification prediction processing on the first preset work order data association feature of each application interface using a preset classification prediction model to obtain the evaluation result of each application interface corresponding to each first work order sub-item to be evaluated; and / or, performing evaluation processing on the second preset work order data association feature of each application interface using a preset second neural network model corresponding to each second work order sub-item to be evaluated to obtain the evaluation result of each application interface corresponding to each second work order sub-item to be evaluated; and combining the evaluation results of each application interface corresponding to each first work order sub-item to be evaluated and / or corresponding to each second work order sub-item to be evaluated to obtain the second open capability evaluation result of each application interface in the product to be evaluated.

[0046] Optionally, the first preset work order data association features are extracted by using techniques such as TF-IDF to extract features from the work order data. The extracted first preset work order data association features include, but are not limited to, one or more of the following features: work order initiation time, application interface unified identifier, number of errors, number of communications, whether there are impatient words, handler, processing time, platform information of the application to be evaluated, whether optimization is needed, etc.

[0047] Optionally, the types of the second preset work order data association features are obtained by performing association analysis on each metadata dimension in the work order data with preset open capability evaluation indicators (such as the success rate and time consumption of the product to be evaluated), and are metadata dimensions that may maximize the influence on the open capability evaluation indicators. The metadata dimensions used for analysis are determined according to application requirements.

[0048] Optionally, the preset classification prediction model is trained using the following method: First preset work order data association features are extracted from the work order data of the evaluated products to construct sample data corresponding to each application interface; based on the evaluation results of each first work order sub-item to be evaluated in the capability maturity assessment results of each application interface in the work order data, sample labels corresponding to each first work order sub-item to be evaluated are constructed; based on the sample data and the corresponding sample labels of each first work order sub-item to be evaluated, training data corresponding to each first work order sub-item to be evaluated is constructed; based on the training data corresponding to each first work order sub-item to be evaluated, the preset classification prediction model corresponding to each first work order sub-item to be evaluated is trained. The capability maturity assessment results include the evaluation results of multiple first work order sub-items to be evaluated and the evaluation results of multiple second work order sub-items to be evaluated.

[0049] For example, work order data for each application interface of N evaluated products (N being a natural number, such as N equals 2 or 3) and the capability maturity assessment results for each application interface can be collected. Then, the work order data and capability maturity assessment results are cleaned, and based on the cleaned work order data and capability maturity assessment results for each application interface, a first preset work order data association feature and an assessment result corresponding to each first work order sub-item to be evaluated are extracted for that application interface. Then, for each application interface, using the first preset work order data association feature of the work order data as sample data, and using the assessment results corresponding to each first work order sub-item to be evaluated in the capability maturity assessment results of that application interface as sample labels, training data corresponding to each first work order sub-item to be evaluated is constructed. Then, based on the training data corresponding to each first work order sub-item to be evaluated, a preset classification prediction model corresponding to the first work order sub-item to be evaluated is trained. The preset classification prediction model can be a support vector machine classification model.

[0050] During the evaluation phase of the application interfaces of the product to be evaluated, work order data for a specified time period (e.g., the past six months) of the application interfaces can be collected. The collected work order data is then cleaned. Following this, the first preset work order data association features for each application interface of the product to be evaluated are extracted, referring to the method described above. Then, the first preset work order data association features are used as inputs to preset classification prediction models corresponding to each first work order sub-item to be evaluated. Each preset classification prediction model predicts the evaluation result corresponding to the first work order sub-item to be evaluated for each application interface. The first work order sub-item to be evaluated includes, but is not limited to, one or more of the following: number of customer communications, customer evaluation type, customer response timeliness, whether a meeting was organized, etc.; the evaluation result includes, but is not limited to, the quantitative score corresponding to the first work order sub-item to be evaluated.

[0051] The preset second neural network model is trained using the following method: extracting the second preset work order data association features of each application interface based on the work order data of the evaluated products, and constructing sample data corresponding to each application interface; constructing sample labels for each second work order sub-item to be evaluated based on the evaluation results of each application interface's capability maturity evaluation results in the work order data; constructing training samples for each second work order sub-item to be evaluated based on the sample data and the sample labels; and training the preset second neural network model for each second work order sub-item to be evaluated based on the training samples for each second work order sub-item to be evaluated.

[0052] For example, work order data for each application interface of N evaluated products, as well as the capability maturity assessment results for each application interface, can be collected. Then, the work order data and capability maturity assessment results are cleaned, and based on the cleaned work order data and capability maturity assessment results for each application interface, a second preset work order data association feature and an assessment result corresponding to each second work order sub-item to be evaluated are extracted for that application interface. Then, for each application interface, using the second preset work order data association feature of the work order data of that application interface as sample data, and using the assessment results corresponding to each second work order sub-item to be evaluated in the capability maturity assessment results of that application interface as sample labels, training data corresponding to each second work order sub-item to be evaluated is constructed. Then, based on the training data corresponding to each second work order sub-item to be evaluated, a preset second neural network model corresponding to the second work order sub-item to be evaluated is trained. For specific implementation methods of training the preset second neural network model based on specified training data, please refer to the prior art; these will not be repeated in the embodiments of this application.

[0053] During the evaluation phase of the application interface of the product to be evaluated, work order data for a specified time period (e.g., the past six months) of the application interface can be collected. The collected work order data is then cleaned. Following this, the second preset work order data association features for each application interface of the product to be evaluated are extracted, referring to the method described above. Then, the second preset work order data association features are used as inputs to the preset second neural network model corresponding to each second work order sub-item to be evaluated. The preset second neural network model is used to infer the evaluation results corresponding to the second work order sub-item to be evaluated for each application interface. The second work order sub-item to be evaluated includes, but is not limited to, one or more of the following: communication, etc.; the evaluation results include, but are not limited to, the quantitative score corresponding to the second work order sub-item to be evaluated.

[0054] During the evaluation phase of the application interface of the product to be evaluated, for each application interface, the evaluation results of all the first work order items to be evaluated corresponding to the application interface and the evaluation results of all the second work order items to be evaluated corresponding to the application interface can be combined to obtain the second open capability evaluation result of the application interface.

[0055] By using machine learning to evaluate multiple work order sub-items, it is possible to cover not only more evaluation items but also complex work order data indicators.

[0056] Step 108: Perform statistical analysis on the log data to obtain the third open capability evaluation results of each application interface in the product to be evaluated.

[0057] Optionally, statistical analysis and processing are performed on the log data to obtain the third open capability evaluation results of each application interface in the product to be evaluated, including: performing statistical analysis and processing on the log data to obtain the call success rate and / or running time of each application interface in the product to be evaluated; and obtaining the third open capability evaluation results of each application interface in the product to be evaluated based on the call success rate and / or the running time.

[0058] The log data refers to the valid call log data of the application interface obtained after data cleaning.

[0059] Optionally, information such as the call period and return result of each application interface can be extracted from log data. Based on the call period and return result of each application interface, the call success rate and runtime of each application interface can be calculated. Then, for each application interface, the third open capability evaluation result of the application interface can be calculated based on the call success rate and / or the runtime. For example, for each application interface, the evaluation score calculated based on the call success rate and the evaluation score calculated based on the runtime can be added together as the third open capability evaluation result of the application interface.

[0060] Step 110: Combine the first open capability assessment result, the second open capability assessment result, and the third open capability assessment result to obtain the application interface open capability maturity assessment result of the product to be evaluated.

[0061] After the aforementioned steps, the first open capability evaluation result, the second open capability evaluation result, and the third open capability evaluation result of each application interface in the product to be evaluated can be obtained. Next, for each application interface, the first open capability evaluation result, the second open capability evaluation result, and the third open capability evaluation result can be weighted and summed to obtain the open capability evaluation result of the application interface.

[0062] Finally, by combining the assessment results of the open capabilities of all application interfaces in the product under evaluation, the maturity assessment result of the open capabilities of the application interfaces of the product under evaluation is obtained. For example, the average value of the assessment results of the open capabilities of all application interfaces in the product under evaluation can be used as the maturity assessment result of the open capabilities of the application interfaces of the product under evaluation. Alternatively, the minimum value of the assessment results of the open capabilities of all application interfaces in the product under evaluation can be used as the maturity assessment result of the open capabilities of the application interfaces of the product under evaluation.

[0063] It should be noted that the execution order of steps 104, 106 and 108 above can be adjusted.

[0064] like Figure 3 As shown in the embodiments of this application, when training the first neural network model and the second neural network model, in addition to constructing training data by collecting interface documents and work order data of the evaluated models, SMOTE (an oversampling method) can also be used to balance positive and negative samples. Then, the neural network models are trained separately based on the training data. Specifically, the first neural network model can be trained based on models with different structures. Then, the performance indicators of the first neural network models trained with various structures are compared, and the first neural network model with the best performance indicator is selected for subsequent application interface openness assessment. Similarly, the second neural network model can be trained based on models with different structures. Then, the performance indicators of the second neural network models trained with various structures are compared, and the second neural network model with the best performance indicator is selected for subsequent application interface openness assessment.

[0065] For example, the sklearn.pipeline tool (a tool for combining multiple data preprocessing steps with model training into a single workflow) can be used to model a first or second neural network model using logistic regression, random forest, ANNs (Artificial Neural Networks), LightGBM (Light Gradient Boosting Machine), or XGBOOST (eXtreme Gradient Boosting). After training, the performance metrics (such as ROC and AUC) of the first or second neural network models trained with different modeling results are compared, and the neural network model with the best performance metrics is selected for subsequent evaluation of application interface openness capabilities.

[0066] After testing, it was found that the embodiments of this application preferably use ANNs to model the first neural network model and the second neural network model.

[0067] In summary, the application interface open capability maturity assessment method disclosed in this application obtains the interface documentation, work order data, and log data of the product to be assessed, as well as the assessment specifications and interface documentation template. Based on the assessment specifications and the interface documentation template, the method evaluates the interface documentation to obtain the first open capability assessment result of each application interface in the product to be assessed, which is used to characterize the interface completeness of the product to be assessed from the interface capability dimension. Based on the work order data, the method obtains the second open capability assessment result of each application interface in the product to be assessed, which realizes the characterization of user satisfaction of the product to be assessed based on data such as community and ecosystem tools. The method performs statistical analysis on the log data to obtain the third open capability assessment result of each application interface in the product to be assessed, thereby evaluating the execution performance of the product to be assessed based on example call data. Finally, the method integrates the first open capability assessment result, the second open capability assessment result, and the third open capability assessment result to obtain the application interface open capability maturity assessment result of the product to be assessed. This method comprehensively evaluates the application interface open capability maturity of the product to be assessed from multiple dimensions, which can cover more assessment points and thus improve the comprehensiveness of the application interface open capability maturity assessment.

[0068] Based on the above embodiments, this application also discloses an application interface open capability maturity assessment device.

[0069] The application interface open capability maturity assessment device in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not make specific limitations.

[0070] The application interface open capability maturity assessment device in this embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment does not specifically limit its use.

[0071] See Figure 4 The application interface open capability maturity assessment device disclosed in this application includes: The input information acquisition module 402 is used to acquire the interface documents, work order data and log data of the product to be evaluated, as well as the evaluation specifications and interface document templates. The first evaluation module 404 is used to evaluate the interface document based on the evaluation specifications and the interface document template, and obtain the first open capability evaluation results of each application interface in the product to be evaluated. The second evaluation module 406 is used to obtain the second open capability evaluation results of each application interface in the product to be evaluated based on the work order data. The third evaluation module 408 is used to perform statistical analysis and processing on the log data to obtain the evaluation results of the third open capabilities of each application interface in the product to be evaluated. The evaluation result fusion module 410 is used to fuse the first open capability evaluation result, the second open capability evaluation result, and the third open capability evaluation result to obtain the application interface open capability maturity evaluation result of the product to be evaluated.

[0072] Optionally, the first evaluation module 404 is further configured to: Based on the evaluation specifications and the interface document template, the interface document is parsed and annotated to obtain an application interface document area in the interface document annotated with information of document sub-items to be evaluated. The information of the document sub-items to be evaluated includes: a first document sub-item to be evaluated, and / or, a second document sub-item to be evaluated and the automatic evaluation method corresponding to the second document sub-item to be evaluated. For the application interface document area marked with the second document sub-item to be evaluated, the corresponding automatic evaluation method is used to verify the application interface document area to obtain the evaluation result corresponding to each second document sub-item to be evaluated. For the application interface document area marked with the first document sub-item to be evaluated, a preset document association feature is extracted based on the interface document, and the preset document association feature is evaluated by a preset first neural network model corresponding to each of the first document sub-items to be evaluated, so as to obtain the evaluation result corresponding to the first document sub-item to be evaluated. By combining the evaluation results corresponding to the first document sub-item to be evaluated and the evaluation results corresponding to the second document sub-item to be evaluated, the first open capability evaluation result of each application interface in the product to be evaluated is obtained.

[0073] Optionally, the step of parsing and annotating the interface document based on the evaluation specifications and the interface document template to obtain an application interface document area in the interface document annotated with information about the document sub-items to be evaluated includes: The interface document is parsed according to the interface document template to obtain the evaluation tags of the application interface document area in the interface document; The evaluation specifications are structured to obtain the document sub-items to be evaluated covered by each specification and / or the automatic evaluation method adapted to the document sub-items to be evaluated; Based on the matching results of the evaluation tags of the application interface document area and the specification, the application interface document area is annotated to obtain an application interface document area annotated with information of the document sub-items to be evaluated.

[0074] Optionally, the preset first neural network model is trained using the following method: Based on the interface documents of the evaluated products, pre-defined document association features are extracted to construct sample data; Based on the capability maturity assessment results of the aforementioned interface documentation, sample labels are constructed for the corresponding sample data. Based on the sample data and the sample labels, the preset first neural network model is trained to evaluate the corresponding document sub-items to be evaluated.

[0075] Optionally, the second evaluation module 406 is further configured to: Based on the work order data, extract the first preset work order data association features and the second preset work order data association features of each application interface. By using a preset classification prediction model to classify and predict the association features of the first preset work order data for each application interface, the evaluation results for each first work order sub-item to be evaluated corresponding to each application interface are obtained; and / or, By using a preset second neural network model corresponding to each second work order sub-item to be evaluated, the data association features of the second preset work order of each application interface are evaluated and processed to obtain the evaluation results of each second work order sub-item to be evaluated corresponding to each application interface. By combining the evaluation results of each application interface corresponding to each first work order sub-item to be evaluated and / or the evaluation results corresponding to each second work order sub-item to be evaluated, the second open capability evaluation result of each application interface in the product to be evaluated is obtained.

[0076] Optionally, the preset classification prediction model is trained using the following method: Based on the work order data of the evaluated products, extract the first preset work order data association features of each application interface, and construct the sample data corresponding to each application interface. Based on the evaluation results of each first work order sub-item to be evaluated in the capability maturity assessment results of each application interface in the work order data, construct corresponding sample data and sample labels for each first work order sub-item to be evaluated. Based on the sample data and the sample labels corresponding to each first work order sub-item to be evaluated, training data corresponding to each first work order sub-item to be evaluated is constructed respectively. Based on the training data corresponding to each of the first work order sub-items to be evaluated, the preset classification prediction model corresponding to the first work order sub-item to be evaluated is trained respectively.

[0077] Optionally, the third evaluation module 408 is further configured to: Perform statistical analysis on the log data to obtain the call success rate and / or running time of each application interface in the product to be evaluated; Based on the call success rate and / or the runtime, obtain the third open capability evaluation results for each application interface in the product to be evaluated. The application interface open capability maturity assessment device disclosed in this application embodiment is used to implement the application interface open capability maturity assessment method described in this application embodiment. The specific implementation methods of each module of the device will not be repeated here, but can be referred to the specific implementation methods of the corresponding steps in the method embodiment.

[0078] In summary, the application interface open capability maturity assessment device disclosed in this application obtains the interface documentation, work order data, and log data of the product to be assessed, as well as the assessment specifications and interface documentation template. Based on the assessment specifications and the interface documentation template, it performs assessment processing on the interface documentation to obtain the first open capability assessment result of each application interface in the product to be assessed, which is used to characterize the interface completeness of the product to be assessed from the interface capability dimension. Based on the work order data, it obtains the second open capability assessment result of each application interface in the product to be assessed, realizing the characterization of user satisfaction of the product to be assessed based on data such as community and ecosystem tools. It performs statistical analysis processing on the log data to obtain the third open capability assessment result of each application interface in the product to be assessed, thereby evaluating the execution performance of the product to be assessed based on example call data. Finally, it integrates the first open capability assessment result, the second open capability assessment result, and the third open capability assessment result to obtain the application interface open capability maturity assessment result of the product to be assessed. It comprehensively assesses the application interface open capability maturity of the product to be assessed from multiple dimensions, which can cover more assessment points and thus improve the comprehensiveness of the application interface open capability maturity assessment of the product to be assessed.

[0079] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus embodiments, since they are fundamentally similar to the method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0080] The above provides a detailed description of the application interface open capability maturity assessment method and apparatus provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method of this application and its core idea. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the idea of ​​this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0082] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the electronic device according to the embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0083] For example, Figure 5An electronic device is shown that can implement the methods according to this application. The electronic device may be a PC, mobile terminal, personal digital assistant, tablet computer, etc. The electronic device conventionally includes a processor 510 and a memory 520, and program code 530 stored in the memory 520 and executable on the processor 510, which, when executing the program code 530, implements the methods described in the above embodiments. The memory 520 may be a computer program product or a computer-readable medium. The memory 520 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. The memory 520 has a storage space 5201 for the program code 530 of a computer program for performing any of the method steps described above. For example, the storage space 5201 for the program code 530 may include various computer programs for implementing the various steps in the methods described above. The program code 530 is computer-readable code. These computer programs can be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The computer program includes computer-readable code that, when executed on an electronic device, causes the electronic device to perform the method according to the above embodiments.

[0084] This application also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the application interface open capability maturity assessment method as described in this application embodiment.

[0085] Such a computer program product can be a computer-readable storage medium, which can have the same characteristics as... Figure 5 The memory 520 in the illustrated electronic device is similarly arranged with storage segments, storage spaces, etc. Program code can be stored, for example, in a compressed form on the computer-readable storage medium. The computer-readable storage medium is typically as shown in the reference... Figure 6 The portable or fixed storage unit is described above. Typically, the storage unit includes computer-readable code 530', which is code read by a processor and, when executed by the processor, implements the various steps of the method described above.

[0086] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this application. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0087] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0088] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for assessing the maturity of application interface open capabilities, characterized in that, include: Obtain the interface documentation, work order data, and log data of the product to be evaluated, as well as the evaluation specifications and interface documentation templates; Based on the evaluation specifications and the interface document template, the interface document is evaluated to obtain the first open capability evaluation results of each application interface in the product to be evaluated. Based on the work order data, obtain the second open capability evaluation results of each application interface in the product to be evaluated; The log data is statistically analyzed and processed to obtain the third open capability evaluation results of each application interface in the product to be evaluated. By integrating the first open capability assessment result, the second open capability assessment result, and the third open capability assessment result, the application interface open capability maturity assessment result of the product to be evaluated is obtained.

2. The method according to claim 1, characterized in that, The evaluation process, based on the evaluation specifications and the interface document template, evaluates the interface documents to obtain the first open capability evaluation results of each application interface in the product to be evaluated, including: Based on the evaluation specifications and the interface document template, the interface document is parsed and annotated to obtain an application interface document area in the interface document annotated with information of document sub-items to be evaluated. The information of the document sub-items to be evaluated includes: a first document sub-item to be evaluated, and / or, a second document sub-item to be evaluated and the automatic evaluation method corresponding to the second document sub-item to be evaluated. For the application interface document area marked with the second document sub-item to be evaluated, the corresponding automatic evaluation method is used to verify the application interface document area to obtain the evaluation result corresponding to each second document sub-item to be evaluated. For the application interface document area marked with the first document sub-item to be evaluated, a preset document association feature is extracted based on the interface document, and the preset document association feature is evaluated by a preset first neural network model corresponding to each of the first document sub-items to be evaluated, so as to obtain the evaluation result corresponding to the first document sub-item to be evaluated. By combining the evaluation results corresponding to the first document sub-item to be evaluated and the evaluation results corresponding to the second document sub-item to be evaluated, the first open capability evaluation result of each application interface in the product to be evaluated is obtained.

3. The method according to claim 2, characterized in that, The process of parsing and annotating the interface document based on the evaluation specifications and the interface document template yields an application interface document area within the interface document that is annotated with information about the document sub-items to be evaluated, including: The interface document is parsed according to the interface document template to obtain the evaluation tags of the application interface document area in the interface document; The evaluation specifications are structured to obtain the document sub-items to be evaluated covered by each specification and / or the automatic evaluation method adapted to the document sub-items to be evaluated; Based on the matching results of the evaluation tags of the application interface document area and the specification, the application interface document area is annotated to obtain an application interface document area annotated with information of the document sub-items to be evaluated.

4. The method according to claim 2, characterized in that, The preset first neural network model is trained using the following method: Based on the interface documents of the evaluated products, pre-defined document association features are extracted to construct sample data; Based on the capability maturity assessment results of the aforementioned interface documentation, sample labels are constructed for the corresponding sample data. Based on the sample data and the sample labels, the preset first neural network model is trained to evaluate the corresponding document sub-items to be evaluated.

5. The method according to claim 1, characterized in that, The step of obtaining the second open capability evaluation results of each application interface in the product to be evaluated based on the work order data includes: Based on the work order data, extract the first preset work order data association features and the second preset work order data association features of each application interface. By using a preset classification prediction model to classify and predict the association features of the first preset work order data for each application interface, the evaluation results for each first work order sub-item to be evaluated corresponding to each application interface are obtained; and / or, By using a preset second neural network model corresponding to each second work order sub-item to be evaluated, the data association features of the second preset work order of each application interface are evaluated and processed to obtain the evaluation results of each second work order sub-item to be evaluated corresponding to each application interface. By combining the evaluation results of each application interface corresponding to each first work order sub-item to be evaluated and / or the evaluation results corresponding to each second work order sub-item to be evaluated, the second open capability evaluation result of each application interface in the product to be evaluated is obtained.

6. The method according to claim 5, characterized in that, The preset classification prediction model is trained using the following method: Based on the work order data of the evaluated products, extract the first preset work order data association features of each application interface, and construct the sample data corresponding to each application interface. Based on the evaluation results of each first work order sub-item to be evaluated in the capability maturity assessment results of each application interface in the work order data, construct corresponding sample data and sample labels for each first work order sub-item to be evaluated. Based on the sample data and the sample labels corresponding to each first work order sub-item to be evaluated, training data corresponding to each first work order sub-item to be evaluated is constructed respectively. Based on the training data corresponding to each of the first work order sub-items to be evaluated, the preset classification prediction model corresponding to the first work order sub-item to be evaluated is trained respectively.

7. The method according to claim 1, characterized in that, The statistical analysis and processing of the log data to obtain the third open capability evaluation results of each application interface in the product to be evaluated includes: Perform statistical analysis on the log data to obtain the call success rate and / or running time of each application interface in the product to be evaluated; Based on the call success rate and / or the running time, obtain the third open capability evaluation results of each application interface in the product to be evaluated.

8. An application interface open capability maturity assessment device, characterized in that, The device includes: The input information acquisition module is used to acquire the interface documents, work order data, and log data of the product to be evaluated, as well as the evaluation specifications and interface document templates. The first evaluation module is used to evaluate the interface document based on the evaluation specifications and the interface document template, and obtain the first open capability evaluation results of each application interface in the product to be evaluated. The second evaluation module is used to obtain the second open capability evaluation results of each application interface in the product to be evaluated based on the work order data. The third evaluation module is used to perform statistical analysis and processing on the log data to obtain the evaluation results of the third open capabilities of each application interface in the product to be evaluated. The evaluation result fusion module is used to fuse the first open capability evaluation result, the second open capability evaluation result, and the third open capability evaluation result to obtain the application interface open capability maturity evaluation result of the product to be evaluated.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method as described in claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in claims 1-7.

Images