Android smart watch prototype application automatic generation method based on program synthesis

Abstract

The application discloses a kind of based on program synthesis's android smart watch prototype application automatic generation method, comprising the following steps: step one, artificial record page set is formed;Step two, automatically synthesize multiple component selection sub-set, and select an optimal component selection sub;Step three, form component set, and establish the inter-component position constraint suitable for watch application for the component in component set;Step four, solve the inter-component position constraint established for the page, form component position set, generate smart watch page;Step five, generate smart watch prototype application in combination with event forwarding component.The present application makes up the lack of existing smart watch application synthesis technology;Generate a large number of smart watch prototype applications, thereby greatly expanding and enriching application store;For developer or end user, when in urgent need, according to smart phone application to generate its smart watch prototype, thereby greatly helping developer to relieve development pressure, facilitate end user use.

Description

Technical Field

[0001] This invention relates to a method for automatically generating Android smartwatch prototype applications based on program synthesis. It is a technical means to automatically generate smartwatch prototype applications for an Android smartphone application using program synthesis with minimal manual annotation. It belongs to the fields of program synthesis and Android application technology. Background Technology

[0002] Since the Apple Watch was first released in 2015, smartwatches have become a rapidly growing market. It is predicted that smartwatch shipments will exceed 140 million units in 2021, and WearOS (an Android variant designed for smartwatches) will see its market share increase by 72.8% in 2022. The lightweight design of smartwatches greatly simplifies simple daily tasks for end-users, such as receiving notifications, making calls, and quickly adding reminders.

[0003] However, a subtle gap exists between smartphone developers and smartwatch apps, causing smartwatch apps to lag far behind smartphone apps in terms of potential. Specifically, due to limited screen space, performance, and battery life, the design language of smartwatch apps differs significantly from that of smartphone apps. Porting an app to a smartwatch platform requires substantial engineering effort, including redesigning the user experience and porting the app interface to the watch. This has resulted in only 4,000 smartwatch apps currently available on the Google Play Store, compared to approximately 2.8 million smartphone apps (as of May 2021). For most apps, smartwatch users cannot fully enjoy the convenience that smartwatches offer.

[0004] Currently, there is no relevant background technology that can automatically generate prototype applications for Android smartphones on smartwatches. Therefore, the key to the automatic generation method for Android smartwatch prototype applications disclosed in this invention is to make up for the lack of existing background technology, enrich the existing smartwatch application store, and facilitate the use of developers and end users. Summary of the Invention

[0005] In view of the lack of existing background technology and the current state of Android smartwatch app stores, this invention provides a method for automatically generating Android smartwatch prototype applications based on program synthesis.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for automatically generating Android smartwatch prototype applications based on program synthesis is presented. This method requires only a few component annotations to automatically synthesize the smartwatch application, eliminating the need for other manual labor such as design and development. It includes five steps:

[0008] Step 1: Developers or end users manually annotate a small number of components that need to be displayed on the Android smartwatch on the Android smartphone application, forming a set of manually annotated components, and manually record the page where each annotated component is located, forming a set of manually recorded pages;

[0009] Step 2: For each page in the set of manually recorded pages and the set of manually annotated components corresponding to that page, automatically synthesize multiple component selection subsets, and automatically select the optimal component selector from these subsets;

[0010] Step 3: For each page in the manually recorded page set, use the optimal component selector synthesized for that page to select other components that need to be displayed on the watch from the unlabeled components to form a component set, and establish inter-component positional constraints for the components in the component set that are suitable for the watch application.

[0011] Step 4: For each page in the manually recorded page set, solve for the positional constraints between components established for that page to form a component position set. Use the component position set to rearrange the components in the component set to generate the smartwatch page.

[0012] Step 5: All generated smartwatch pages form a smartwatch page collection. All pages in the smartwatch page collection are automatically aggregated and combined with the event forwarding component to generate a smartwatch prototype application.

[0013] Furthermore, in step one, the component is the basic unit that constitutes the application, used to display application information to the user, and the user can also operate it; in step one, the component is a collection of a series of "attribute-value" tuples, and these attributes include component ID, component position, child components, etc.

[0014] Furthermore, in step one, the page is the basic unit by which the application displays a set of relevant information to the user. It is composed of components based on the kinship relationship between the components, i.e., the parent-child relationship, and is arranged in a tree structure.

[0015] Furthermore, in step two, the component selector can select all components whose attributes satisfy a specified attribute constraint relationship from a specified page; the components annotated by the developer or end user must be on a page-by-page basis, that is, the developer or end user must record the page where the component is located when annotating, and can find all the attributes of the corresponding component on the page based on the manual annotation.

[0016] Furthermore, in step three, the positional constraint between the two components is a binary relationship established between the two components based on their positional attributes in the mobile application.

[0017] Furthermore, in step four, the rearranged components must satisfy the positional constraints between components and be displayed appropriately and correctly on the smartwatch screen.

[0018] Furthermore, in step five, the generated smartwatch prototype application is used to demonstrate the core functions of the original smartwatch application, and the core business of the original smartwatch application is completed through user operation.

[0019] Furthermore, the component selector is a set of special paths connected by the "∨" symbol. Each special path takes the form " / l0 / l1 / ... / l K / … / l N , is a chain of attribute constraint relationships connected by the " / " symbol, where:

[0020] 1)l K (0≤K≤N) represents the attribute constraint relationship, which is used to select the K-level component based on the attribute constraint;

[0021] 2) Attribute constraint relationships are in the form of "attribute operator value", which means that if the "attribute-value" of a K-level component satisfies the relationship, the component will be selected. For example, "id=main" means that the component attribute ID is "main". If the component ID of a K-level component is "main", the constraint relationship is satisfied and the component will be selected.

[0022] 3) The attribute constraint relationship can be "*", which means always true, that is, any K-level component will be selected unconditionally;

[0023] 4) Attribute constraint relationships can be “**”, indicating selection across multiple levels, and can be at level zero;

[0024] 5) Special paths cannot be terminated by "**", i.e., l N It cannot be “**”.

[0025] Furthermore, when performing component selection and composition, the manual recording page is designated as U, and the manually labeled components in U are designated as v1, v2, ..., v n ∈U, which is divided into the following sub-steps:

[0026] 1) For each component v i (i = 1, 2, ..., n) Enumerate all possible choices v i The component selectors form a component selector set X. i Assume component v i If it belongs to a K-level component, the enumeration process will select a child component from the component. K"Start, along the v in the U page tree" i The path to the root component is continuously used, utilizing each component (all v) along that path. i The parent component's component properties are composed hierarchically, and the property constraints are... J (0≤J≤K), until synthesis of l0, during which time all selectable v are recorded. i Component selection child to X i middle;

[0027] 2) Merge all X i (i = 1, 2, ..., n) is X U Thus X U Any selector in the list can simultaneously select v1, v2, ..., v n Considering that component selectors of the form " / ** / *" will unconditionally select all components in U, this sub-step needs to delete component selectors of the form " / ** / *" during merging. This is done greedily, and the optimal merge is selected from all possible merges after deleting such component selectors.

[0028] 3) From X U Select the optimal component selector x U ∈X U And return, the optimal chooser needs to appropriately generalize the selection ability, not only from U to choose v1, v2, ..., v n ∈U, you can also select other unlabeled components that need to be displayed on the smartwatch.

[0029] Furthermore, in step five, the event forwarding component is used to forward user events occurring on each smartwatch application page to the mobile application, execute the user events, and return the execution results to the smartwatch user for rendering.

[0030] The beneficial effects of this invention are as follows: Considering that there is currently no relevant background technology that can automatically generate prototype applications for Android smartphones on smartwatches, the automatic generation method for Android smartwatch prototype applications provided by this invention fills the gap in existing background technologies. The smartwatch prototype applications generated using this invention can be correctly displayed on the smartwatch and can be operated by users, completing the core functionalities of the original smartphone application. This invention can achieve fully automatic generation of Android smartwatch prototype applications with minimal manual annotation costs, and the generated prototype applications can complete the core functions of the original smartphone application. It can automatically analyze manually annotated components, synthesize the component selectors required for each smartwatch page, and then use them to select other components that still need to be displayed on the watch from unannotated components. It can automatically establish and solve the positional constraints between components, ensuring that the rearranged components can be correctly displayed on the smartwatch. This invention generates a large number of smartwatch prototype applications for smartwatch app stores based on existing mobile applications, thereby greatly expanding and enriching the app store; it can also automatically generate smartwatch prototypes for developers or end users based on smartphone applications when urgently needed, thereby greatly helping developers alleviate development pressure and facilitating end users' use. Attached Figure Description

[0031] Figure 1 This describes the process of steps one, two, three, and four of the present invention;

[0032] Figure 2 This is the process for step five of the present invention. Detailed Implementation

[0033] The invention will now be described in further detail with reference to the accompanying drawings.

[0034] This invention discloses an automatic generation method for Android smartwatch prototype applications based on program synthesis, whose components, pages, and component selectors are as follows:

[0035] Component: In this invention, a component is defined as a set of attribute-value pairs, including component ID, component position, child components, etc.

[0036] Page: This invention defines an Android application page as a rooted tree, where each node is a component and edges represent parent-child relationships. In the page, branch nodes are typically container components that hold other components as child components, while leaf nodes are used to display information and handle user events.

[0037] Components are hierarchically classified according to their position on the page, with the root component being a level 0 component and the direct child components of a level K component (K≥0) being level K+1 components.

[0038] Component selectors: The component selectors defined in this invention are a set of special paths connected by the "∨" symbol. Each special path takes the form " / l0 / l1 / ... / l K / … / l N , is a chain of attribute constraint relationships connected by the " / " symbol, where:

[0039] 1)l K (0≤K≤N) represents the attribute constraint relationship, which is used to select the K-level component based on the attribute constraint;

[0040] 2) Attribute constraint relationships are generally in the form of "attribute operator value", which means that if the "attribute-value" of a K-level component satisfies the relationship, the component will be selected. For example, "id=main" means that the component attribute ID is "main". If the component ID of a K-level component is "main", the constraint relationship is satisfied and the component will be selected.

[0041] 3) The attribute constraint relationship can be "*", which means always true, that is, any K-level component will be selected unconditionally;

[0042] 4) Attribute constraint relationships can be “**”, indicating selection across multiple levels (can be zero levels);

[0043] 5) Special paths cannot be terminated by "**", i.e., l N It cannot be “**”.

[0044] For example, component selector

[0045] / * / (id=main)∨ / ** / (id=digit_2)

[0046] This selects all first-level components with component ID "main" and (any level) components with component ID "digit_2" within the specified page.

[0047] This invention mainly comprises five steps:

[0048] Step 1: Developers or end users manually annotate a small number of components that need to be displayed on the Android smartwatch on the Android smartphone application, forming a set of manually annotated components, and manually record the page where each annotated component is located, forming a set of manually recorded pages;

[0049] Step 2: For each page in the set of manually recorded pages and the set of manually annotated components corresponding to that page, automatically synthesize multiple component selection subsets, and automatically select the optimal component selector from these subsets;

[0050] Step 3: For each page in the manually recorded page set, use the optimal component selector synthesized for that page to select other components that need to be displayed on the watch from the unlabeled components to form a component set, and establish inter-component positional constraints for the components in the component set that are suitable for the watch application.

[0051] Step 4: For each page in the manually recorded page set, solve for the positional constraints between components established for that page to form a component position set. Use the component position set to rearrange the components in the component set to generate the smartwatch page.

[0052] Step 5: All generated smartwatch pages form a smartwatch page collection. All pages in the smartwatch page collection are automatically aggregated and combined with the event forwarding component to generate a smartwatch prototype application.

[0053] In step one, components are the basic units that make up an application. They are used to display application information to the user, and the user can also operate them. For example, in a calculator application, the user can input formulas into the calculator by operating components such as "numbers" and "operators." The calculator calculates the formula and displays the result to the user through the "calculation result" component.

[0054] In step one, the components must meet the definition of components outlined above in this invention. In this step, developers or end-users need to carefully read the instruction manual for the smartphone application, thoroughly use and experience the application, and deeply understand the basic services and operational logic it provides. Accordingly, developers or end-users need to manually label a small number (usually 1-2 per page) of components that need to be displayed in the smartwatch application for pages related to the core functions of the smartphone application. Developers or end-users only need to record the unique attributes of these components so that each component can be uniquely identified.

[0055] In step one, the page must meet the definition of a page as described above in this invention. In the aforementioned steps, the components annotated by the developer or end-user must be on a page-by-page basis; that is, the developer or end-user must record the page containing the component while annotating. This is to enable this invention to find all attributes of the corresponding component on the page based on manual annotation. The page must be recorded in JSON, XML, or other commonly used, custom, and parsable formats.

[0056] In step two, the component selector must satisfy the definition of a component selector as described above in this invention. A component selector can select all components from a specified page whose attributes satisfy a given attribute constraint. For example, a component selector can select all components with ID "digit_4" on a specified page based on the attribute constraint "id = digit_4". In this step, this invention will independently analyze the components annotated on each page and use programmatic synthesis technology to synthesize a component selector that can select all annotated components on the analyzed page. Considering that there may be multiple such component selectors, this invention will select the optimal component selector. Specifically, assuming the manually recorded page is U, and the manually annotated components in U are v1, v2, ..., v n ∈U, which is divided into the following sub-steps:

[0057] 1) This invention will provide each component v i (i = 1, 2, ..., n) Enumerate all possible choices v i The component selectors form a component selector set X. i Assume component v i If it belongs to a K-level component, the enumeration process will select a child component from the component. K "Start, along the v in the U page tree" i The path to the root component is continuously used, utilizing each component (all v) along that path. i The parent component's component properties are composed hierarchically, and the property constraints are... J (0≤J≤K), until synthesis of l0. During this process, record all possible choices v. i Component selection child to X i middle.

[0058] 2) Merge all X i (i = 1, 2, ..., n) is X U Thus X U Any selector in the list can simultaneously select v1, v2, ..., v n Considering that component selectors of the form " / ** / *" will unconditionally select all components in U, this sub-step needs to remove component selectors of the form " / ** / *" during the merge and select the optimal merge from all possible merges.

[0059] 3) From X U Select the optimal component selector x U ∈X U And return. The optimal chooser needs to appropriately generalize the selection ability, not only from U, but also to select v1, v2, ..., v. n ∈U, you can also select other unlabeled components that need to be displayed on the smartwatch.

[0060] In step three, the positional constraint between the two components is a binary relationship established based on their positional attributes within the mobile application. For example, component 1 must be placed above component 2. This step will apply its optimal component selection sub-x to each page U. U This yields the set V of all components in U that need to be displayed in the smartwatch application. U For set V U Any two components v in i and v j This step generates v based on the component position attributes of both. i and v j The positional constraints between components. Each positional constraint between components consists of two parts: horizontal constraints and vertical constraints, which represent v respectively. i and v j In terms of horizontal and vertical positional relationships. For example, v i Both need to be in v j Above (horizontal positional relationship), it also needs to be in v j To the right (vertical positional relationship).

[0061] In step four, the rearranged components must satisfy both the positional constraints between components and be appropriately and correctly displayed on the smartwatch screen. To ensure proper display of components on the Android smartwatch, this invention utilizes a grid system to divide each watch page into an M×N grid, where M represents the maximum number of components that each row of a watch page can hold, and N represents the maximum number of components that each column of a watch page can hold. When solving for the set of positional constraints between components in each page U, to ensure proper display of components on the Android smartwatch, horizontal and vertical constraints based on M and N must be added to each component simultaneously to prevent component overflow. Finally, based on the solved component positions, this invention generates the corresponding smartwatch page for each recorded mobile phone page.

[0062] In step five, the present invention will aggregate all smartwatch pages to generate a smartwatch prototype application.

[0063] Figure 1 The diagram shows the workflow of steps one through four. The main function is to analyze manually annotated components and synthesize smartwatch pages to prepare for prototype application generation. The inputs and outputs of these four steps are shown in Table 1. The input is a set of manually annotated components and the corresponding manually recorded pages; the output is the smartwatch pages. The manually annotated components and pages must be core functional pages and components of the smartphone application. Therefore, developers or end-users need to have a deep understanding of the basic and core functions of the smartphone application. Component annotations must ensure unique (unambiguous) querying within the page, and the page must record information in JSON, XML, or other commonly used, custom, and parsable formats. Due to the complexity of the four steps, the specific implementation is described in three parts.

[0064] Table 1 shows the inputs and outputs for steps one through four.

[0065]

[0066] The component selector is used to select components from a specified page based on constraints. In this sub-step, the component selector synthesis component analyzes and extracts labeled components and other unlabeled components from the manually recorded page using component annotation information, and populates their attribute information. Combining the attribute information, the component selector synthesis component enumerates all component selectors that can select each component. Assuming the candidate component belongs to level K, this enumeration process will begin with the component selector " / l". K "Starting, along the path from the candidate component to the root component in the page tree, continuously synthesize attribute constraints by using the component properties of each component (which is the parent component of the candidate component) along the path level by level." J (0≤J≤K), until a synthesis of l0 is achieved. During this process, the component selectors of all selectable candidate components are recorded.

[0067] This section will generate a set of candidate component selectors for each component. For each component, since there are a large number of candidate component selectors, in order to reasonably reduce the number and improve performance, some selection can be made during the enumeration process: for example, only some component properties that are meaningful to component selection (such as component ID) are considered, while component properties that are meaningless to component selection (such as component position) are ignored.

[0068] Considering that component selectors in the form of " / ** / *" will unconditionally select all components on the manually recorded page, the initial merging of all attribute constraints will be performed hierarchically during the selection of the optimal component, and special paths equivalent to " / ** / *" after merging will be removed. This initial merging process is greedy, meaning that each merge will select the optimal merge from all possible merges. Given that some attribute constraints cannot be merged, this initial merging will generate multiple candidate sets.

[0069] Next, the component selectors in multiple candidate sets are further merged using the Cartesian product and the "∨" symbol to produce a single set. Each component selector in this set can select every component in the manually labeled component set at the same time, and is not equivalent to " / ** / *".

[0070] Finally, the merits of each component selector in the set are evaluated based on the complexity of each component selector and the number of available components, and the optimal component selector is selected from them.

[0071] The optimal component selector is used to select all components that need to be displayed on the smartwatch page on the manual recording page, forming a component set.

[0072] Based on the component position attributes of each component in the component set, establish positional constraints between any two components in the component set. At the same time, based on the size M and N of the smartwatch grid system, solve the positional constraints between components to obtain the position of each component displayed on the smartwatch page, forming a component position set.

[0073] Based on the component location set and the component set, the components are rearranged on the smartwatch screen to form the final smartwatch page.

[0074] Figure 2 The diagram shows step five, whose main function is to generate a smartwatch prototype application based on the synthesized smartwatch pages. Table 2 shows the input and output of step five. The input is the set of smartwatch pages synthesized in the previous steps and the event forwarding component; the output is the smartwatch prototype application.

[0075] The event forwarding component forwards user events occurring on each smartwatch application page to the mobile application, executes the user events, and returns the execution results to the smartwatch user for rendering. This component can be implemented based on existing mature technologies, such as WebRTC, and is not within the scope defined by the appended claims.

[0076] The smartwatch page collection and event forwarding component are automatically aggregated into a smartwatch prototype application: the smartwatch prototype application uses the pages in the smartwatch page collection as the user interface and the event forwarding component as the event handling callback method.

[0077] Table 2 Input and Output of Step 5

[0078]

[0079] It should be noted that the terms such as "upper", "lower", "left", "right", "front", and "back" used in the invention are only for clarity of description and are not intended to limit the scope of the invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the invention.

[0080] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for automatically generating Android smartwatch prototype applications based on program synthesis, characterized in that, It includes five steps: Step 1: Developers or end users manually annotate a small number of components that need to be displayed on the Android smartwatch on the Android smartphone application, forming a set of manually annotated components, and manually record the page where each annotated component is located, forming a set of manually recorded pages; Step 2: For each page in the manually recorded page set and the corresponding set of manually annotated components, automatically synthesize multiple component selection subsets, and automatically select the optimal component selector from these subsets; the component selector is a set of special paths connected by the "∨" symbol, each special path being in the form " / l0 / l1 / ... / l K / … / l N , is a chain of attribute constraint relationships connected by the " / " symbol, where: 1) l K 0 ≤ K ≤ N represents an attribute constraint relationship, which is used to select the K-level component based on the attribute constraint; 2) The attribute constraint relationship is in the form of "attribute operator value", which means that if the "attribute-value" of a K-level component satisfies the relationship, the component will be selected. When "id = main" means that the component attribute ID is "main", if the component ID of a K-level component is "main", the constraint relationship is satisfied and the component will be selected. 3) The attribute constraint relationship is " "" indicates tautology, meaning that any K-level component will be selected unconditionally. 4) The attribute constraint relationship is " ", indicates a selection spanning multiple levels, including level zero; 5) Special paths cannot use " "The end, i.e., l" N It cannot be " ”; When performing component selection and composition, the manual record page is designated as U, and the manually labeled components in U are designated as v1, v2, ..., v n ∈U, which is divided into the following sub-steps: 1) For each component v i Let i = 1, 2, ..., n, and enumerate all choices v. i The component selectors form a component selector set X. i Assuming component v i If it belongs to a K-level component, the enumeration process will select a child component from the component. K "Start, along the v in the U page tree" i The path to the root component continuously synthesizes attribute constraints by using the component properties of each component along that path. J , 0 ≤ J ≤ K, until synthesis l0, during which all choices v are recorded. i Component selection child to X i middle; 2) Merge all X i Let i = 1, 2, ..., n be X U Thus X U Any selector in the list simultaneously selects v1, v2, ..., v n Considering " / / The "" component selector will unconditionally select all components in U; ​​this sub-step is deleted during the merge. / The component selector is greedily selected from all possible merges; 3) From X U Select the optimal component x U ∈X U And return, the optimal chooser not only selects v1, v2, ..., v from U. n ∈U, and then select other unlabeled components that need to be displayed on the smartwatch; Step 3: For each page in the manually recorded page set, use the optimal component selector synthesized for that page to select other components that need to be displayed on the watch from the unlabeled components to form a component set, and establish inter-component positional constraints for the components in the component set that are suitable for the watch application. Step 4: For each page in the manually recorded page set, solve for the positional constraints between components established for that page to form a component position set. Use the component position set to rearrange the components in the component set to generate the smartwatch page. Step 5: All generated smartwatch pages form a smartwatch page collection. All pages in the smartwatch page collection are automatically aggregated and combined with the event forwarding component to generate a smartwatch prototype application.

2. The method for automatically generating Android smartwatch prototype applications based on program synthesis according to claim 1, characterized in that, In step one, a component is the basic unit that constitutes an application and is used to display application information to the user, who can also operate it. A component is a collection of "attribute-value" pairs, and these attributes include component ID, component position, and child components.

3. The method for automatically generating Android smartwatch prototype applications based on program synthesis according to claim 1, characterized in that, In step one, a page is the basic unit by which an application displays a set of relevant information to a user. It is composed of components based on the relationships between them and is arranged in a tree structure.

4. The method for automatically generating Android smartwatch prototype applications based on program synthesis according to claim 1, characterized in that, In step two, the component selector selects all components whose attributes satisfy a specified constraint relationship from a specified page. The components annotated by the developer or end user must be on a page-by-page basis. That is, when annotating, the developer or end user must record the page where the component is located and find all the attributes of the corresponding component on the page based on the manual annotation.

5. The method for automatically generating Android smartwatch prototype applications based on program synthesis according to claim 1, characterized in that, In step three, the positional constraint between the two components is a binary relationship established between the two components based on their positional attributes in the mobile application.

6. The method for automatically generating Android smartwatch prototype applications based on program synthesis according to claim 1, characterized in that, In step four, the rearranged components must satisfy the positional constraints between components and be correctly displayed on the smartwatch screen.

7. The method for automatically generating Android smartwatch prototype applications based on program synthesis according to claim 1, characterized in that, In step five, the generated smartwatch prototype application is used to demonstrate the core functions of the original smartwatch application and to complete the core business of the original smartwatch application through user operation.

8. The method for automatically generating Android smartwatch prototype applications based on program synthesis according to claim 1, characterized in that, In step five, the event forwarding component is used to forward user events occurring on each smartwatch application page to the mobile application, execute the user events, and return the execution results to the smartwatch user for rendering.

Images