Ai discovery

Abstract

This disclosure relates generally to detecting artificial intelligence (AI) implementation in a software application comprising one or more application packages (APs). One or more processors extract one or more AP strings from the software application, which each represent an AP; and create a prompt for a machine learning model, trained to generate output text, comprising the one or more AP strings, the prompt representing instructions to provide a classification and provide functionality information of each of the one or more APs, the classification being AI relevant or non-AI relevant and the functionality information describing a functionality of the respective AP. The one or more processors then evaluate the machine learning model on the prompt to generate output text corresponding to the classification and the functionality information of each of the one or more APs; and generate a report of the AI implementation based on the output text.

Description

[0104] The emphasis on proper structural prompt design is important when aiming for higher precision in AI determination. In particular, it can be difficult to understand how to effectively prompt some machine learning models as these models are often offered as “black boxes”. In machine learning, a black box refers to giving a prediction (such as output text) based on an input without explaining or showing how the machine learning model has reached its precision. This is common with publicly available machine learning models. Therefore, such effectively design is desirable as it ensures that the machine learning model has a clear understanding of the task at hand, thus optimizing its response.

[0105] However, even with an engineered prompt, it could still give out an inaccurate answer with respect to complicated task descriptions. In particular, it has been observed that extensive and intricate directive prompts often lead to inaccurate prediction results. This is likely because this would be an “epic prompt” which may overwhelm the machine learning model. Such an epic prompt also has other disadvantages such as being, hard to debug, and only sometimes provides the desired output.

[0106] To provide a more accurate answer to the sophisticated tasks, in the preferred embodiment, processor 102 creates 202 the prompt for the machine learning model by creating a first prompt and a second prompt. This is based on a “divide-and-conquer” approach, in which the problem is decomposed into several intermediate steps and the final answer will be given out once these intermediate steps are solved successfully. This may also be referred to as ‘chaining’.

[0107] Decomposing the problem into several intermediate steps provides much more accurate results as the machine learning model can focus on a specific, intermediate task, rather than focusing on the broad, overall task. This approach also resembles a structural prompt design strategy, drawing inspiration from the “Single Responsibility Principle” (SRP) of Object-Oriented Design. This approach ensures that each prompt is tailored to execute a singular and specific task, thereby enhancing the clarity and precision of the model’s output. As such, each step in the chain is a single responsibility, giving the chain of prompts a modular design.

[0108] In the preferred embodiment, the first prompt represents instructions to provide the functionality information of each of the one or more APs and the second prompt represents instructions to provide the classification of each of the one or more APs. As such, in the preferred embodiment, server 108 evaluates 203 the machine learning model on the prompt by evaluating the machine learning model on the first prompt to generate initial output text and evaluating the machine learning model on the second prompt to generate the output text, where the second prompt is based on the initial output text.

[0109] While in some embodiments, server 108 evaluates 203 the machine learning model on the prompt by evaluating the machine learning model on the second prompt then evaluate the first prompt based on the output of evaluating the second prompt, the result described herein show that there is an advantage to evaluating the first prompt then the second prompt, as the machine learning model (i.e., a LLM such as ChatGPT in the preferred embodiment) can sometimes misclassify some key information without the reasoning that is provided by the functionality information.

[0110] In more detail, to make ChatGPT reliably complete the task of AI candidate determination, the overall task is split into two steps: (1) AI capability analysis (i.e., Step 3 of Fig. 4) and candidate detection (i.e., Step 4 of Fig. 4). After collecting all deduplicated candidates from Step 2 of Fig. 4, ChatGPT is prompt to generate the analysis emphasizing the functionality. For example, ChatGPT will provide the analysis of ‘Google’s ML Kit API for object detection and tracking in images and videos to the package com.google.mlkit.vision.objects. In addition, there might be some obfuscated or customized candidates that ChatGPT has no knowledge of, but it still could provide meaningful insights. This is especially useful as many new AI relevant APs are developed and released.

[0111] For instance, the package nlp.WordPieceModelPB can be hard to determine with rulebased methods. With the help of ChatGPT, the human-like speculation of ‘A library for tokenizing text using the WordPiece algorithm, implemented in Protocol Buffers format’ can effectively describe the highly potential functionality. Based on the generated analysis, ChatGPT is further prompted to classify if the candidate uses AI techniques. As ChatGPT can understand its generation well, the steps of AI capability analysis and AI candidate detection are straightforward by adopting the prompts of Step 3 and Step 4 of Fig. 4. AI service summarisation

[0112] The last step is AI service summarization, where a brief report is generated to describe what AI services the software application uses via the identified AI candidates. As ChatGPT and most LLMs always have limited context length, it is challenging to put all the determined AI candidate names and their corresponding analysis into the same context window. To address this limitation, the API-level candidates are excluded, as their functionalities are likely to be implied by their packages. As such, in the preferred embodiment, processor 102 classifies each of the one or more APs as an application programming interface (API) or a non-API and creates a third prompt further represents instructions to exclude APs classified as an API from the report.

[0113] By concatenating all determined AI candidate analyses except the APIs, ChatGPT is prompted to generate a brief AI service report, which is shown in Step 6 of Fig. 4. As such, processor 102 creates the third prompt based on the output text, the third prompt representing instructions to generate the report; and generating a report comprises evaluating the machine learning model on the third prompt to generate the report. Experimental Configuration

[0114] The preferred embodiment was used to produce the results described in the following section. The experimental configuration used to produce the results described in the following section will now be described. To make ChatGPT fully understand Steps 3, 4 and 6 of Fig. 4, five input-output examples are provided to the LLM for the few-shot in-context learning. For the actual implementation of the disclosed method, the default 4k context windows are used. As ChatGPT has limits for requests per minute, it is time-consuming to process one app with one candidate per request. However, when inputting too many candidates in one request, ChatGPT can likely miss a few candidates in the outputs and cause misalignment.

[0115] To balance task accuracy and processing speed, a batch size of 3 candidates is used and instructed ChatGPT to output in a JSON format, which can be easily parsed. The parameters of the model are mainly kept at their default values, except for the temperature and TopP parameters, which were adjusted from the default value of 1 to 0.2 and 0.95 to reduce the impact of ChatGPT’s output randomness but maintain the creativity to generate candidate analysis and app summary. Evaluation Aims

[0116] The preferred embodiment was evaluated to address the following questions in light of the aims of this evaluation: 1. How reliable is the disclosed method in determining AI capability for existing AI apps ? To answer this question, the effectiveness of the disclosed method is examined to determine the AI capabilities of AI Android apps. Meanwhile, the possibility of making the disclosed method more efficient is explored and trade-offs between processing time and reliability are presented. By answering this question, the efficacy of the disclosed method can be provided. 2. How reliable is the disclosed method in determining AI capability for apps in the wild? To answer this question, the disclosed method is compared to a rule-based approach on Android apps based on their performance determining AI capability of in-wild Android apps. By answering this question, the strengths of the disclosed method in AI app detection are demonstrated. 3. How useful is the AI summary generated by the disclosed method? In addition to pinpointing AI candidates and further apps, this is taken one step further by summing up and predict the potential AI capabilities concerning the identified AI candidates in this question. By answering this question, a better understanding of what type of AI is used is developed, which provides a more meaningful and insightful summary report for future app developers and researchers. 4. What insights can be gained from the disclosed method analysis? Given the successfully identified apps, the AI tasks and domains among the AI candidates and the determined AI apps are summarised. By answering this question, a further understanding of the detailed AI function provisions in these AI apps can be provided. Dataset

[0117] The following described experiments were conducted on two separate datasets. The first dataset was collected from AndroZoo (https: / / androzoo.uni.lu / ) and contained 56,682 Android apps. The second dataset was curated by the inventors, and contained 15,517 Android apps, as the first dataset did not contain the newly released apps, especially the ones implemented on newly released LLMs, such as ChatGPT.

[0118] Dataset 1

[0119] The disclosed method was compared to a rule-based approach, which identifies AIdriven Android apps via keywords matching on the dissected and decompiled Android apps. The rule-based approach successfully identified 56,682 AI apps from a total of 7,259,232 apps on AndroZoo (the latest Android version is considered while multiple versions are stored). 56,682 Android apps was used as a baseline to cross-validate the disclosed method. Fig. 5 illustrates the distribution of collected apps over the years.

[0120] Dataset 2

[0121] For the second dataset, AndroZoo was again utilised, where the latest release for every Android app was extracted from its provided CSV file. With these latest application releases, the npm package google-play-scraper (https: / / www.npmjs.com / package / google-play-scraper) was used by feeding package names to it to determine if the app is still publicly available on Google Play store and, if so, to retrieve the app metadata, especially the app description and release date.

[0122] To determine if the released app is Al-related or not, the AI App Filter process (Step 0 of Fig. 4) was used. Using this process, script marries the precision of rule-based systems with the adaptability and depth of machine learning models. This dual-layered approach guarantees a curated list of apps that genuinely resonate with AI and deep learning principles and applications.

[0123] Following Step 0 (AI App Filter) of the preferred embodiment, shown in Fig. 4,, 15,517 (only the latest released version is considered) Android apps were successfully identified from a total of 23,321,823 apps on AndroZoo. Going one step further, the release date was also extracted from the metadata provided on the Google Play Store and Fig. 5 shows the distribution of the released Android apps over the last decade. It clearly shows the rise of the released apps is at the same pace as the prevailing AI technologies. It is also worth mentioning that the total number of apps in Fig. 5 is 15,055 due to the release date missing in the given metadata among 462 Android apps, which only accounts for 2.98% (462 / 15,517). Therefore, the rising trend of AI apps over the past ten years is still valid. How reliable is the disclosed method in determining AI capability for existing AI apps ?

[0124] To address this question, whether the disclosed method can effectively identify AI capabilities for pre-determined AI apps is examined. To do so, dataset 1 was used. However, as datasetl only determines AI apps but does not provide ground truths of AI candidates, a subset of 382 AI apps were randomly sampled and used to perform the human validation to assess the disclosed method.

[0125] However, it is time-consuming and impractical to check every app in dataset 1 manually. As such, the online sample Size Calculator (https: / / www.surveysystem.com / sscalc.htm) was used to compute the number of sample apps (with a confidence level of 95% and margin of error of 5%), which provided the 382 apps randomly selected from the total of 56,682.

[0126] Fig. 6 shows the quantity distribution of three representative candidate types, packages, APIs, and HTTPS requests. Fig. 6a shows the total number of packages, APIs, and HTTPS requests after Step 1 (Candidate Extraction) of the preferred embodiment. Fig. 6b shows the final determined Al-related packages, APIs, and HTTPS requests after Steps 3-4 (AI capability analysis and detection) of the preferred embodiment.

[0127] The median values for the total considered packages, APIs, and strings in Fig. 6a are 433.5, 296, and 53.5, respectively. It is noteworthy that the number of packages, APIs, and HTTPS requests are all proactively filtered when extracted from Android APK files via a selfcollected whitelist of commonly non-AI candidates. The whitelist of non-AI candidates comprises the official Java and Android kernel-related packages and API signatures.

[0128] The median values for the determined AI packages, APIs, and HTTPS requests are 9.5, 33, and 0, respectively. Compared to the considered candidates, much more packages are removed as they are not reasoned as Al-related candidates while higher number of APIs are kept as AI candidates, which would play an essential role in AI app determination. This can be explained by the fact that most AI APIs only belong to certain AI packages. In terms of HTTPS requests, they are merely detected by the disclosed method, as the introduction of HTTPS requests for OpenAI is comparatively late and they are not pervasive among the collected experimental apps.

[0129] To examine the reliability of the disclosed method in determining AI candidates on the sampled dataset, multiple authors were asked to independently label 49,371 candidates appearing in the sampled 382 AI apps. Similar to the disclosed method, candidates are classified into two groups: AI candidates and non-AI candidates. True Positive is defined as the candidates that are correctly identified as AI candidates, while True Negative refers to the candidates that are correctly detected as non-AI candidates. After the human validation, a high precision of 98.05% and recall of 93.31% is achieved, with Cohen’s unweighted kappa coefficient of 0.82. The results indicate that the disclosed method is highly accurate in AI candidate determination and external observers are very likely to agree on the outcomes of the disclosed method.

[0130] The reliability of the disclosed method in identifying AI apps was then further evaluated, further validate how reliable the disclosed method is in identifying AI apps. An AI app is defined as the one that uses AI candidates. Based on the candidate analysis via the disclosed method, 378 out of the total of 382 apps were identified as AI apps while the remaining 4 as non-AI apps. The reason 4 apps were identified as non-AI ones is that only some file names ending with the keyword .model could be identified, but any complementary AI evidence such as Al-related package names, APIs could not be detected. Without further complementary information, it is difficult to determine what type of AI capabilities and where these abilities are harnessed. Therefore, these apps are not categorised as AI ones.

[0131] During the evaluation of the disclosed method, it was found that the disclosed method is relatively slow as it needs to generate the analysis for both AI and non-AI candidates at first. While the design of Step 3 (AI Capability Analysis) and Step 4 (AI Candidate Detection) is motivated by a “divide and conquer” approach, it may not be optimised for the disclosed method. To efficiently determine the AI capability, one intuitive approach is to classify AI candidates and then provide analysis only for AI candidates. To evaluate the reliability of this approach, the same evaluation settings is followed on the sampled 382 apps. The results of this evaluation is presented in Table 1, which is compared them with the ones from the original implementation. Analysis / Detection Detection / Analysis Al Candidate Al Apps Time (sec) AI Candidate liiiii Time (sec) 2859 378 638.76 2192 371 226.94 Table 1: Comparison of the reliability of the disclosed method between the implementations of analysis-then-detection and detection-then-analysis. The time is the average running time for an Android APK in seconds.

[0132] It is first observed that the processing time per Android app is significantly decreased by 65% after adopting the detection-then-analysis implementation. While the new approach is still able to detect most AI apps (371 vs. 378), it is noted that the number of detected AI candidates is much less than the original ones. The observation indicates the trade-off between the reliability and efficiency of the disclosed method. Based on the overall results, it is suggested that detection-then-analysis can only be used when developers are interested in finding AI apps. If developers seek to understand all possible AI capabilities of an app, it is logical that the analysis-then-analysis method is more effective to capture AI candidates.

[0133] In summary, the efficacy of the disclosed method regarding the performance of determining AI capability for existing AI apps is demonstrated. Human validation suggests that the disclosed method can achieve high precision (98.05%) and recall (93.31%) in detecting AI candidates and AI apps. How reliable is the disclosed method in determining AI capability for apps in the wild?

[0134] Here, the performance of the disclosed method in AI app detection is investigated, as well as its advantages compared to the state-of-the-art rule-based approaches. The performance of a rule-based approach is compared to the disclosed method. To have a fair comparison, the preferred embodiment of the disclosed method is evaluated on dataset 2, which contains 15,517 Android apps released on the Google Play Store. However, it is time-consuming to experiment on all the collected Android apps. To select a representative subset, the focus is first put on Android apps published over the past five years from 2019 to 2023, which will result in a total of 12,740, and then randomly pick 30% apps for each year. In total, there are 3,819 (1980 * 30% + 2336 * 30% + 2326 * 30% + 3053 * 30% + 3044 * 30%) Android apps for the experiment.

[0135] Following the same setting as in the previous section, the precision and recall regarding the AI candidate identification are computed. The disclosed method shows the precision of 97.32% and recall of 91.01%. The inter-annotator reliability measured by Cohen’s unweighted kappa coefficient is 0.80, indicating the high agreement with outcomes of the disclosed method. Rule-based approach Disclosed method Al Apps Non-AI Apps AI Apps Non-AI Apps 603 3216 1366 2453 Table 2: Number of AI and Non-AI apps classified by the rule-based approach and the disclosed method.

[0136] Table 2 represents the detection results regarding the rule-based approach and the disclosed method. Among the total 3,819 experimental Android apps, 603 of them are classified by the rule-based approach as AI Apps, while 1,366 are recognized as AI apps by the disclosed method, which is more than two times greater than that of the rule-based approach (242% more to be exact).

[0137] Fig. 7 shows a Venn diagram of detected AI apps in the rule-based approach and the disclosed method. Fig. 7 aims to illustrates the relationships between the outcomes of the disclosed method compared to the outcomes of the rule-based approach Android apps. In particular, Fig. 7 clearly shows that 546 Android apps, accounting for 90.55% (546 / 603) and 39.97% (546 / 1366) in the rule-based approach and the disclosed method, respectively, are both identified as AI Apps. The nine Android apps classified as AI by the rule-based approach were manually inspected and it was found that two apps were recognized as AI due to only one special tflite file being detected, while the remaining seven apps (App with the sha256 value: EC45FCB6288EDE61077FF80885F1A8EA99600E26B37E7DCABA7069C0134A4289) were classified as AI because the keywords, such as ‘DecisionTreeClassifier’ and ‘KNeighborsClassifier’, were detected in some descriptions provided in several JSON files, which are not convincing evidence to label these apps as AI.

[0138] Regarding the remaining 820 Android apps only classified by the disclosed method as AI, 86 Android apps (with a confidence level of 95% and margin of error of 10%) were randomly selected and cross-validated by two inventors. Consensuses was reached if classification conflicts happen between the two inventors. In the process of cross-validation, it was found that some Android apps provide AI capabilities by utilizing public APIs offered by OpenAI. During the cross-validation, it was found that that some Android apps provide AI capabilities by utilizing OpenAI APIs, which are impossible for the rule-based method to detect due to the limited rules.

[0139] More specifically, some AI app developers invoked OpenAI APIs directly in Java code, while others implemented API invocations in the programming language C / C++ and compiled into a native dynamic library, in which the HTTPS request (https: / / api.openai.com / vl / chat / completions) is reflected in the extracted compiled Java class file or native library .so file. Customers subscribing to these apps will need to provide an OpenAI key to make full use of the functions given by these apps. In the end, it was confirmed that 83 of 86 AI apps classified by the disclosed method were AI Apps.

[0140] In summary, high precision (97.32%) and recall (91.01%), and the high inter-annotator reliability with regard to the AI candidate’s identification reconfirm the effectiveness of the disclosed method and demonstrate its ability to extend to apps in the wild. Furthermore, the experimental result of detecting more than twice as many AI apps as the rule-based approach instantiates the outperformance of the disclosed method against the rule-based approach. How useful is the AI summary generated by the disclosed method?

[0141] In terms of determining whether an Android app is Al-based or not, the disclosed method could effectively provide an accurate answer with a bunch of evidence. Here, the usefulness of the disclosed method in identifying detailed AI functionalities is evaluated provided by the determined Al-based apps.

[0142] To fulfil this purpose, six participants are first recruited. These six participants had expertise in Android app analysis with at least three years research experiences in Android. 20 Android apps are random selected and determined as AI apps in the previous section. For every selected app, an app description summary is generated via ChatGPT by feeding it reasoned AI candidates and provide the generated summary and the detailed AI candidates to participants. In the meanwhile, the app description is also retrieved from Google Play store and also offer it to the participants. More specifically, the participants are provided with two types of description summary regarding each Android app. One is generated by ChatGPT with the help of the identified AI candidates, and the other is the description extracted from Google Play store.

[0143] To assess the quality of two different app description summaries,, participants are requested to evaluate the accuracy (i.e., Do you think that the summary is accurate concerning the AI functions the app provides?), completeness (i.e., Do you think that the summary is complete with regard to the provided AI tasks?), consistency (i.e., Do you think that the summary is concrete enough concerning the AI functions?), and recommendations for app users (i.e., Would you like to recommend the summary to other app users?) and developers (i.e., Would you like to recommend the summary to other app developers to inspire new app development?) by answering 5 linear scale questions with the rating scale spanning from 1 to 10. Lower rated scores indicate worse quality.

[0144] In total, 40 app descriptions were obtained for the selected 20 Android apps. To relieve the heavy burden of evaluating 40 app description summaries for every participant, the six participants were split into two groups. Each group of participants evaluated 10 Android apps and every app is evaluated by at least three different participants. Summary Type Accuracy Completeness Consistency Recom. App users Recom. App devs. Avg. lliilll IlMIIII liiiii lliilll liiiii iiBi® Med. Illillll Med, AI Generated 8.67 + 0.14 9 8.63 + 0.13 9 8.55 + 0.13 9 7.5 + 0.31 8 8.57 + 0.15 9 Developer offered lllllll liiiiiii lllllll® III liiiii lliilll lliilll iiiiiii lliB® liiiii fill® iiiiiii liiiii Table 3: Average and median scale values voted for sampled Android apps by six participants. Recom. App users and Recom. App devs. corresponds to Recommendation for app users or app developers. Ave. and Med. correspond to the Average and Median value.

[0145] Table 3 shows the study results delineated by the average and median scale values. It is observed that the summary generated by the disclosed method is more than 2 greater than the value for the original description (i.e., developers’ provided description) on most evaluation aspects, while the median value is 1 point greater than the value for original app descriptions. Moreover, the standard error of the mean (SEM) of the summary generated by the disclosed method is approximately one-third of the SEM of the original app description, indicating that participants more consistently agree on the quality of the summary generated by the disclosed method.

[0146] Table 3 clearly shows that the summary generated by the disclosed method is conspicuously better than the description provided by their corresponding developers. In addition, it was found that the voted scores for both the summary generated by the disclosed method and the original app description are quite close on the metric of Recommendation for app users. Specifically, the median value is the same and the difference between the average scores is only 0.63, which reflects that different participants may have different preferences when recommending summaries to app users. In general, the description with higher accuracy, completeness, and consistency generated by the automated approach of the disclosed method is cherished by developers and researchers but it may not be valued by app users. However, some participants believe that app users would gain benefits from such descriptions as well.

[0147] In summary, the participant study with regard to the metrics of accuracy, completeness, consistency, recommendation for app users or app developers for both the AI (the disclosed method) generated summary and the app developers provided one shows that the summary generated by the disclosed method is much more preferred by the participants. What insights can be gained from the analysis of the disclosed method? Data Type Package & API Model HTTP Req. Others Total #. items 10,285 liillil 11.081 Table 4: Different types of identified AI candidates.

[0148] Here, the aim is to unveil the AI services underlying the identified AI apps. Specifically, the aim is to classify their specific domains and applications for AI candidates implemented in the identified AI apps in a previous section. Table 4 shows the number of the identified AI candidates in different self-explained categories. It clearly shows that the majority of the candidates (92.82%) are package names and APIs extracted from APK file.

[0149] The second biggest category is the model, which refers to the on-device AI model files, such as .tflite and .caffemodel. However, it is noted that AI models only play a small part among all AI candidates. Apart from package names, APIs, and AI models, it was found that some apps rely on the HTTPS requests (e.g., OpenAI APIs) to provide AI capabilities. For the remaining candidates which share few common patterns, these candidates are considered to be ‘Others’.

[0150] With these identified AI candidates, a classification scheme can be defined which covers diverse applications and domains. As there are a significant number of AI candidates, it is impractical to perform the classification manually. Therefore, ChatGPT was utilised to classify AI domains and applications of each candidate.

[0151] Fig. 8 shows the distributions of classified domains and applications of the determined AI candidates. The area of the sector in different colour represents its proportion. On the top level, the AI services are classified into six common AI processes, including Computer Vision (54.80%), Machine Learning and Data Analysis (26.85%), Natural Language Processing (NLP) (13.38%), Others (2.75%), Audio and Speech Processing (1.90%), and Augmented Reality (0.32%), which is listed in the order of its proportion among the total number of AI candidates. Some of the candidates are given the domain name Others as it is hard for ChatGPT to determine the concrete AI process, such as some imported high-level package names.

[0152] It clearly represents that the majority of the AI functions are computer vision oriented. To be more specific, the detailed tasks in the second level in the domain of computer vision are Object Detection (45.97%), Text Recognition (17.95%), and Face Recognition (16.77%), etc. The second largest AI domain is Machine Learning and Data Analysis, which is occupied mostly by the detailed Data Processing (86.69%), such as the APIs invoked to do some tensor operations and input and output data parsing. The last and the second to last AI domain summarized in the AI candidates are Augmented Reality and Audio and Speech Processing, respectively.

[0153] Based on the classified AI candidate information, the domains and tasks are further categorised on the app level. For each application, the occurrence of domains and tasks associated with AI candidates are tallied. The application is then categorized under the domain and task that demonstrates the highest frequency. Table 5 shows the number of apps in different domains. Domains Machine Learning and Data Analysis Computer Vision Natural Language Processing (NLP) Audio and Speech Processing Others Augmented Reality #. Apps 643 liogggggggggggg^ iiiiiii iliiM Table 5: Number of Android apps in different domains.

[0154] The majority of AI applications fall into the Data Analysis domain, whereas most AI candidates are classified under Computer Vision. This discrepancy is primarily due to the overlap of candidates implemented across various apps, with a significant concentration in the Computer Vision category. Furthermore, the number of apps in both the Computer Vision and Data Analysis domains is nearly equal, placing them as the top two categories in both AI candidate classification and app categorization. In essence, the majority of AI candidates predominantly utilize machine learning and data processing algorithms to offer vision related features, aiding in everyday tasks like face and handwritten character recognition. Conversely, virtual reality functions are minimally incorporated in these apps. This distribution likely stems from the longstanding research and user demand for machine learning and vision-related functions. In contrast, virtual reality functions may lack compelling application scenarios and are not currently considered essential by most users.

[0155] In summary, the predominate commonly used AI services among Android apps and their underlying AI candidates are computer vision (54.80%) and data processing (26.85%) related as a result of popular daily tasks to fulfil for users via all sorts of mobile apps while the augmented reality-oriented ones are least provided due to the compelling application scenarios lacking despite it is a novel and popular concept nowadays. Discussion Implications

[0156] Security mitigation for software artifacts

[0157] Security patches are constantly merged into vulnerable software artifacts, such as the Android Operating System and the app involved third-party libraries, to prevent severe software compromising problems (e.g., confidential information leak) from happening. The results described herein reveal that by asking LLM about the implementation basics, it could respond with accurate, at least to some extent, descriptions of whether they are Al-related or not. However, LLM can but is not limited to only responding to classification inquiries. With careful and heuristic prompt design, LLM can also be used to do code generation, bug fixes, etc. One possibility would be to use LLM as a neural knowledge to inquire if the API involves some existing vulnerabilities, for example, the vulnerability record in public Common Vulnerabilities and Exposures (CVE). Both researchers and developers could benefit from the vulnerability inquiries for LLM in that they could efficiently and effectively fix the potential vulnerabilities during their artifact’s development. Consequently, more reliable and secure artifacts could be achieved.

[0158] Software artifacts analysis for researchers

[0159] The results described herein validate the potential of LLM in code summarization. The prevalence of cross-platform frameworks requires that code summarization should be taken on multiple programming languages. However, it is not straightforward to merge the summarizations on different programming languages due to the interactions between them. Detailed analysis of the interactions between different languages and careful design of prompts are necessary to make full use of the powerful potential offered by LLMs.

[0160] Advanced AI identification approaches and questionnaire with app developers

[0161] In the process of AI app identification, manual analysis reveal that it is hard to accurately identify any AI candidates for some self-claimed AI Android apps. More specifically, it was found that some Android apps claim that they are implemented with advanced AI functionalities and tasks in their app descriptions provided on the Google Play Store. The mismatch between the application description and AI implementation can hence be addressed by the disclosed method. Conclusion

[0162] This disclosure provides an automated approach which leverages the advanced LLM to discovery AI candidates and pinpoint AI functionalities in Android apps. Experiments on sampled Android apps show that the disclosed method could have a better performance in AI app identification compared to rule-based approaches. The AI tasks and domains were then characterized among the collected AI candidates. The results described herein revealed that the most popular AI service is computer vision related while the virtual reality-based functions are actually least provided even though it is one of the most popular concepts gaining most attentions both from AI researchers and early adopters nowadays. A user study was also conducted with respect to the usefulness of the disclosed method and the study results confirmed that the app summary generated by the disclosed method is much preferred by participants with regard to the metrics of accuracy, completeness, consistency, and recommendation for app developers.

[0163] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

1. A computer-implemented method for detecting artificial intelligence (AI) implementation in a software application comprising one or more application packages (APs), the method comprising:extracting one or more application package (AP) strings from the software application, each of the one or more AP strings representing one of the one or more APs;creating a prompt for a machine learning model, trained to generate output text, comprising the one or more AP strings, the prompt representing instructions to provide a classification of each of the one or more APs and provide functionality information of each of the one or more APs, the classification being AI relevant or non-AI relevant and the functionality information describing a functionality of the respective AP;evaluating the machine learning model on the prompt to generate output text corresponding to the classification and the functionality information of each of the one or more APs; andgenerating a report of the AI implementation in the software application based on the output text.

2. The method of claim 1, wherein creating the prompt for the machine learning model comprises creating a first prompt and a second prompt, wherein,the first prompt represents instructions to provide the functionality information of each of the one or more APs; andthe second prompt represents instructions to provide the classification of each of the one or more APs.

3. The method of claim 2, wherein evaluating the machine learning model on the prompt comprises evaluating the machine learning model on the first prompt to generate initial output text and evaluating the machine learning model on the second prompt to generate the output text, wherein the second prompt is based on the initial output text.

4. The method of any one of the preceding claims, wherein the method further comprises analysing a description of the software application before extracting one or more AP strings from the software application to identify the one or more APs.

5. The method of claim 4, wherein analysing the description of the software application comprises performing a keyword search using terms relevant to AI.

6. The method of claim 4 or 5, wherein analysing a description of the software application comprises creating an initial prompt based on the description of the software application, the initial prompt representing instructions to provide a classification of the software application, the classification of the software application being likely to implement AI or unlikely to implement AI.

7. The method of any one of the preceding claims, wherein extracting the one or more AP strings from the software application comprises applying a static analysis method to the software application.

8. The method of any one of the preceding claims, whereinthe method further comprises creating a third prompt based on the output text, the third prompt representing instructions to generate the report; andgenerating the report comprises evaluating the machine learning model on the third prompt to generate the report.

9. The method of claim 8, wherein the method further comprises classifying each of the one or more APs as an application programming interface (API) or a non-API and creating the third prompt further represents instructions to exclude APs classified as an API from the report.

10. The method of claim 8 or 9, wherein the first, second or third prompt comprises learning examples indicative of a desired output from evaluating the machine learning model on the first, second or third prompt, respectively.

11. The method of claim 10, wherein the learning examples are few-shot learning examples.

12. The method of any one of the preceding claims, wherein the method further comprisesstoring each AP string of APs classified as AI relevant and respective functionality information on an AI relevant database and storing each AP string of APs classified as non-AI relevant and respective functionality information on a non-AI relevant database.

13. The method of claim 12, wherein the method further comprises, after extracting one or more AP strings from the software application, accessing the AI relevant database and the nonAI relevant database to determine whether any one of the one or more AP strings are stored on either database.

14. The method of claim 13, wherein the method further comprises, upon determining that any one of the one or more AP strings are stored on either database, retrieving the functionality information from the respective database and excluding the respective AP from the prompt, wherein the report comprises the functionality information retrieved from the respective database.

15. The method of any one of the preceding claims, wherein the one or more APs are one or more of:an API;an asset file; and a library.

16. The method any one of the preceding claims, wherein the method further comprises evaluating the machine learning model on a context prompt to provide context to the machine learning model to provide a desired output.

17. The method of any one of the preceding claims, wherein the method further comprises, upon classifying one of the one of more APs as AI relevant, generating and transmitting an alert to indicate detection of AI implementation in the software application to a user.

18. The method of any one of the preceding claims, wherein the software application is an Android application.

19. Software that, when executed by a computer, causes the computer to perform the method of any one of the preceding claims.

20. A system for detecting artificial intelligence (AI) implementation in a software application comprising one or more application packages (APs), the system comprising: one or more processors configured to:extract one or more application package (AP) strings from the software application, each of the one or more AP strings representing one of the one or more APs;create a prompt for a machine learning model, trained to generate output text, comprising the one or more AP strings, the prompt representing instructions to provide a classification of each of the one or more APs and provide functionality information of each of the one or more APs, the classification being AI relevant or non-AI relevant and the functionality information describing a functionality of the respective AP;evaluate the machine learning model on the prompt to generate output text corresponding to the classification and the functionality information of each of the one or more APs; andgenerate a report of the AI implementation in the software application based on the output text.

Images