A method, system, medium, and apparatus for interactive interface generation

Abstract

The application provides a technical solution for interactive interface generation, which comprises the following steps: collecting multi-modal data related to a context, processing the multi-modal data by modal to obtain single-modal feature data vectors, splicing the single-modal feature data vectors into a first feature data vector, processing the first feature data vector to obtain a second feature data vector, determining a current state and a context label based on the single-modal feature data vectors; determining a plurality of second feature data vectors based on a preset time window, and combining the second feature data vectors into a current vector sequence, inputting the current vector sequence into an intent prediction model to obtain a first atomized operation confidence score vector corresponding to a preset atomized operation library, determining K components with the highest confidence score scores in the first atomized operation confidence score vector, and obtaining a candidate atomized operation set; determining display attributes of a current UI element based on the candidate atomized operation set, the current state and the current context label, and performing rendering processing based on the display attributes of the current UI element and the display attributes of a previous UI element to generate an interactive interface. The efficiency and adaptability of the dynamically generated interactive interface and the user experience can be improved.

Description

Technical Field

[0001] This application relates to the field of smart device technology, and more particularly to a technology for generating interactive interfaces. Background Technology

[0002] With the rapid development of mobile internet, the number of applications installed on Android-based smart devices (usually through distribution or installation of APK files) has increased dramatically. To easily and quickly find and use these applications on their desktop or interface, users typically need to manually organize them, such as creating folders and sorting application icons. This leads to the following pain points: 1. Heavy management burden Users need to spend a lot of time and effort to organize and maintain a clear desktop layout. As the number of applications increases, the desktop will inevitably become cluttered, and users will actually take on the role of "desktop administrator".

[0003] 2. Low efficiency of use Even if the desktop is organized, completing a task corresponding to a specific scenario (such as playing music on the way to work) usually requires multiple steps: unlocking the screen, finding and opening the relevant application, and clicking several times within the application.

[0004] Therefore, how to enable smart devices to adaptively generate interactive interfaces to facilitate user operation and improve operational efficiency is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In order to at least partially solve the above-mentioned technical problems, the purpose of this application is to provide a method, system, medium and device for generating interactive interfaces.

[0006] According to one aspect of this application, a method for generating an interactive interface is provided, applied to a smart device, wherein the method includes: Real-time acquisition of context-related multimodal data; processing of the multimodal data according to modality to obtain multiple single-modal feature data vectors; concatenation of these vectors into a first feature data vector in a preset order; processing of the first feature data vector to obtain a second feature data vector with a first preset number of dimensions; and determination of the current state of the smart device and the corresponding current context label based on the current multiple single-modal feature vectors. Based on a preset time window, several second feature data vectors are determined and formed into a current vector sequence. The current vector sequence is input into the intention prediction model to obtain an atomic operation confidence vector of a second preset number of dimensions corresponding to a preset atomic operation library. The position index of each component in the atomic operation confidence vector corresponds one-to-one with the identifier of an atomic operation in the preset atomic operation library. The value of each component is the confidence score of the corresponding atomic operation. The K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element of the candidate atomic operation set includes the position index of an atomic operation and its confidence score. Based on the candidate atomic operation set, the current state of the smart device, and the corresponding current context label, the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set are determined. Based on the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set and the display attributes of the previous UI element, rendering processing is performed to generate an interactive interface.

[0007] Optionally, the real-time acquisition of context-related multimodal data includes: Based on a preset frequency, real-time acquisition of context-related multimodal data is performed.

[0008] Optionally, the real-time acquisition of context-related multimodal data includes: Based on preset event triggers, real-time collection of context-related multimodal data is performed.

[0009] Optionally, the multimodal data includes at least: temporal modal data and behavioral modal data.

[0010] Optionally, the multimodal data further includes spatial modal data and environmental modal data.

[0011] Optionally, the step of processing the multimodal data according to each modality to obtain multiple single-modal feature data vectors includes: The multimodal data is preprocessed according to modality and source, and incomplete data is handled. The processed data is then vectorized to obtain multiple single-modal feature data vectors.

[0012] Optionally, the step of processing the first feature data vector to obtain a second feature data vector of a first preset number of dimensions includes: The first feature data vector is normalized and linearly mapped to obtain a second feature data vector with a first preset number of dimensions.

[0013] Optionally, the construction of the preset atomization operation library includes: Extract and abstract each atomic operation of the smart device, record the source category and availability status of each atomic operation, and set the priority and fallback relationship of each atomic operation; After deduplication, merging, and verification of all atomic operations, each atomic operation is recorded as a record to build a pre-defined atomic operation library. Each atomic operation record includes at least the following fields: identifier, execution method type, target parameters, and display information. In the preset atomization operation library, a one-to-one mapping relationship is maintained between the identifier of the atomization operation and the dimension index output by the intention prediction model.

[0014] Optionally, the extraction and abstraction of each atomic operation of the smart device includes: Extract any callable capability entry point declared by the developer and abstract it into an atomic operation of the smart device; Extract any shortcut action generated by learning the historical operation sequence under user authorization based on the operating system of the smart device, and abstract it into an atomic operation of the smart device; Extract any executable action encapsulated by the operating system of the smart device, determine and abstract it into an atomic operation of the smart device.

[0015] Optionally, the display attributes include at least: visibility, size, position, and hierarchy.

[0016] Optionally, the method for generating an interactive interface further includes: Based on the trigger operation of any UI element in the interactive interface, the execution entry point of the corresponding atomic operation is called to execute the corresponding operation.

[0017] Optionally, the method for generating an interactive interface further includes: Record feedback data of the corresponding operations performed, and determine user preference tags based on the recorded historical feedback data; The step of determining the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set, based on the candidate atomic operation set, the current state of the smart device, and the corresponding current context label, includes: Based on the candidate atomic operation set, the current state of the smart device and the corresponding current context tag, and the user preference tag, determine the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set.

[0018] Optionally, the method for generating an interactive interface further includes: Based on recorded historical feedback data, adjust the relevant attributes of the corresponding atomic operations, and / or use them to iteratively train the intent prediction model.

[0019] According to another aspect of this application, a system for generating an interactive interface is provided, deployed on a smart device, wherein the system includes: The multimodal data acquisition and processing layer unit is used to acquire context-related multimodal data in real time, process the multimodal data according to modality to obtain multiple single-modal feature data vectors, and concatenate them into a first feature data vector in a preset order. The first feature data vector is also processed to obtain a second feature data vector with a first preset number of dimensions. Based on the current multiple single-modal feature vectors, the current state of the smart device and the corresponding current context label are determined. An intent prediction unit is used to determine several second feature data vectors based on a preset time window and form a current vector sequence. The current vector sequence is input into the intent prediction model to obtain an atomic operation confidence vector of a second preset number of dimensions corresponding to a preset atomic operation library. The position index of each component in the atomic operation confidence vector corresponds one-to-one with the identifier of an atomic operation in the preset atomic operation library. The value of each component is the confidence score of the corresponding atomic operation. The K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element of the candidate atomic operation set includes the position index of an atomic operation and its confidence score. The interactive interface generation unit is used to determine the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set based on the candidate atomic operation set, the current context label and the current state of the smart device, and to perform rendering processing based on the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set and the display attributes of the previous UI element to generate an interactive interface.

[0020] Optionally, the system for generating an interactive interface further includes: The atomic operation library maintenance unit is used to extract and abstract each atomic operation of the smart device, record the source category and availability status of each atomic operation, and set the priority and fallback relationship of each atomic operation. After deduplication, merging and verification of all atomic operations, each atomic operation is recorded as a record to construct a preset atomic operation library. Each atomic operation record includes at least the following fields: identifier, category, target parameter and display information. In the preset atomic operation library, a one-to-one mapping relationship is maintained between the identifier of the atomic operation and the dimension index output by the intent prediction model.

[0021] According to another aspect of this application, a computer-readable medium is provided, wherein computer-readable instructions are stored on the medium, which are executed by a processor to implement part or all of any of the above methods.

[0022] According to another aspect of this application, a device for generating an interactive interface is provided, wherein the device includes: One or more processors; and a memory storing computer-readable instructions that, when executed, cause the processor to perform some or all of the operations described above.

[0023] Compared with existing technologies, this application provides a method, system, medium, and device for generating interactive interfaces. The method includes: real-time acquisition of context-related multimodal data; processing the multimodal data according to modality to obtain multiple single-modal feature data vectors, concatenating them in a preset order to form a first feature data vector; processing the first feature data vector to obtain a second feature data vector of a first preset number of dimensions; and determining the current state of the smart device and the corresponding current context label based on the current multiple single-modal feature vectors; determining several second feature data vectors based on a preset time window and forming a current vector sequence; inputting the current vector sequence into an intent prediction model to obtain an atomic operation confidence vector of a second preset number of dimensions corresponding to a preset atomic operation library, wherein the position index of each component in the atomic operation confidence vector is... Each component corresponds one-to-one with the identifier of an atomic operation in a preset atomic operation library. The value of each component is the confidence score of the corresponding atomic operation. The K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element in the candidate atomic operation set includes the position index of an atomic operation and its confidence score. Based on the candidate atomic operation set, the current state of the smart device, and the corresponding current context label, the display attributes of the current UI elements corresponding to each atomic operation in the candidate atomic operation set are determined. Based on the display attributes of the current UI elements corresponding to each atomic operation in the candidate atomic operation set and the display attributes of the previous UI elements, rendering processing is performed to generate an interactive interface. This application can process multimodal data of the real-time perceived environment on the smart device side to determine the current state and current context label of the smart device, and obtain a vector sequence for prediction. An intent prediction model deployed on the device side is used to predict the user's atomic operation intent, determine the display attributes of the UI elements corresponding to each atomic operation, and combine the display attributes of the previous UI elements for rendering processing to generate the interactive interface of the smart device. It can adaptively generate atomic-level interactive interfaces that match the current state and context without requiring manual user triggering, greatly improving the efficiency and adaptability of smart devices in dynamically generating interactive interfaces and enhancing the user experience. Attached Figure Description

[0024] Other features, objectives, and potential technical effects of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A schematic diagram is shown of a method for generating an interactive interface for a smart device according to one aspect of this application; Figure 2 A schematic diagram of a system for generating an interactive interface deployed on a smart device according to another aspect of this application is shown. The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0025] The present application will now be described in further detail with reference to the accompanying drawings.

[0026] In a typical configuration of various embodiments of this application, the method execution entity, each trusted party of the system, and / or each module of the device may include one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0027] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0028] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0029] To further illustrate the technical means adopted and the effects achieved in this application, the technical solution of this application will be clearly and completely described below in conjunction with the accompanying drawings and embodiments and / or optional embodiments.

[0030] Figure 1 The diagram illustrates a method for generating an interactive interface according to one aspect of this application, applied to a smart device, wherein one embodiment of the method includes: S101 collects context-related multimodal data in real time, processes the multimodal data according to modality to obtain multiple single-modal feature data vectors, and concatenates them into a first feature data vector in a preset order. The first feature data vector is then processed to obtain a second feature data vector with a first preset number of dimensions. Based on the current multiple single-modal feature vectors, the current state of the smart device and the corresponding current context label are determined. S102, based on a preset time window, determines several second feature data vectors and forms a current vector sequence. The current vector sequence is input into the intention prediction model to obtain an atomic operation confidence vector of a second preset number dimension corresponding to a preset atomic operation library. The position index of each component in the atomic operation confidence vector corresponds one-to-one with the identifier of an atomic operation in the preset atomic operation library. The value of each component is the confidence score of the corresponding atomic operation. The K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element of the candidate atomic operation set includes the position index of an atomic operation and its confidence score. S103 determines the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set, the current state of the smart device, and the corresponding current context label, and performs rendering processing based on the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set and the display attributes of the previous UI element to generate an interactive interface.

[0031] This application provides a method for generating an interactive interface, which is applied to a smart device 10 and implemented through related software / firmware deployed on the smart device 10. The smart device 10 can be a mobile smart terminal, wearable device, smart vehicle system, smart home device, etc., based on the Android platform. This is merely an example; other existing or future devices and / or resource platforms that are applicable to this application should also be included within the scope of protection of this application.

[0032] In this embodiment, in step S101, the smart device 10 can collect context-related multimodal data in real time, process the multimodal data according to modality to obtain multiple single-modal feature data vectors, and concatenate them into a first feature data vector in a preset order. The first feature data vector is processed to obtain a second feature data vector with a first preset number of dimensions. Based on the current multiple single-modal feature vectors, the current state of the smart device and the corresponding current context label are determined.

[0033] With authorization from the user and / or the Android operating system, the smart device 10 can collect multimodal data related to its current context in real time. Then, according to modality, it processes the data of the same modality within the obtained multimodal data to obtain multiple single-modal feature data vectors of different modalities. These single-modal feature data vectors are then concatenated into a first feature data vector in a preset order and further processed to obtain a second feature data vector of a first preset number of dimensions. Based on the current single-modal feature data vectors of different modalities, the current state of the smart device 10 and its corresponding current context label are determined. The current state and corresponding current context label of the smart device 10 can be determined based on the obtained single-modal feature data vectors. The possible states of the smart device 10 are related to its type and can be predefined, including at least screen lock / screen on, main screen / foreground interface state, battery level, charging status, network type and connection status, Bluetooth / WiFi connection status, silent / vibration / do not disturb status, etc. If the smart device 10 is a smart vehicle system, it may also include driving mode. The context labels can be predefined based on possible scenarios. If the smart device 10 is a smart vehicle, then it can at least include commuting, driving, parking, home, office, sports, and nighttime rest. When processing the collected multimodal data by mode, for non-periodic discrete numerical data, one-hot encoding can be used. Encoding methods include hot hashing, hash buckets, clustering, and embedding, or combinations thereof. For periodic data, sine / cosine (sin / cosine) encoding or bucketing encoding can be used. To facilitate subsequent fusion using gating weighting mechanisms (e.g., when fusing different modal features, appropriate activation functions can be used to dynamically adjust the contributions of different modal features, ensuring that the accuracy and robustness of the subsequent intention prediction model output improve with the number of modalities collected), the dimension of each single-modal feature data vector can be preset to a fixed value. For incomplete data, missing data processing can be performed to ensure that the dimension of the single-modal feature data vector obtained from the missing data is the corresponding preset value.

[0034] Continuing in this embodiment, in step S102, the intelligent device 10 can determine several second feature data vectors based on a preset time window and form a current vector sequence. The current vector sequence is input into the intention prediction model to obtain an atomic operation confidence vector of a second preset number of dimensions corresponding to a preset atomic operation library. The position index of each component in the atomic operation confidence vector corresponds one-to-one with the identifier of an atomic operation in the preset atomic operation library. The value of each component is the confidence score of the corresponding atomic operation. The K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element of the candidate atomic operation set includes the position index of an atomic operation and its confidence score.

[0035] The intelligent device 10 can repeat step S101 according to a preset time window to obtain several second feature data vectors, and combine these several second feature data vectors into a current vector sequence. Then, the current vector sequence is input into the intent prediction model deployed in the intelligent device 10 to obtain an atomic operation confidence vector of the second preset number dimension corresponding to the preset atomic operation library deployed in the intelligent device 10. The position index of each component in the atomic operation confidence vector corresponds one-to-one with the identifier of an atomic operation in the preset atomic operation library, and the value of each component is the confidence score of the corresponding atomic operation. Then, the K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element of the candidate atomic operation set includes the position index of an atomic operation and its confidence score.

[0036] The intent prediction model deployed in the smart device 10 can be obtained by training a lightweight neural network (such as MobileNet, TFLite, etc.). A sample Python code snippet of a lightweight neural network built on the Keras framework is shown below: ... # Model Definition def build_intent_prediction_model(time_steps=8, input_dim=256, num_actions=500): inputs = tf.keras.Input(shape=(time_steps, input_dim)) # Feature Extraction Layer x = tf.keras.layers.TimeDistributed( tf.keras.layers.Dense(128, activation='relu') (inputs) x = tf.keras.layers.Dropout(0.2)(x) x = tf.keras.layers.TimeDistributed( tf.keras.layers.Dense(64, activation='relu') (x) x = tf.keras.layers.Flatten()(x) # Output layer (multi-label classification) outputs = tf.keras.layers.Dense(num_actions, activation='sigmoid')(x) model = tf.keras.Model(inputs=inputs, outputs=outputs) return model ... Continuing with the above example, the Java code for the predictive inference part of the Android-based smart device 10 is as follows: ... private Interpreter tfliteInterpreter; private static final int TIME_STEPS = 8; private static final int FEATURE_DIM = 256; private static final int TOP_K = 5; public List <atomicaction>predict(ContextWindow contextWindow) { / / Prepare input: [1, TIME_STEPS, FEATURE_DIM] float[][][] input = new float[1][TIME_STEPS][FEATURE_DIM]; input[0] = contextWindow.to2DArray(); / / Preparing for output float[][]output = new float[1]

[500] ; / / Predictive Inference tfliteInterpreter.run(input, output); / / Top-K selection return selectTopK(output[0], TOP_K); } public class ContextWindow { private float[][]sequenceFeatures; / / [TIME_STEPS][FEATURE_DIM] public float[][]to2DArray() { return sequenceFeatures; } } private List <atomicaction>selectTopK(float[] scores, int k) { / / Use a priority queue to select Top-K PriorityQueue <actionscore>pq = new PriorityQueue<>(k); for (int i = 0; i <scores.length; i++) { if (scores[i]>0.3) { / / Confidence score threshold pq.offer(new ActionScore(i, scores[i])); If (pq.size()>k) pq.poll(); } } / / Query specific atomication operations from the preset atomication operation library List <atomicaction>actions = new ArrayList<>(); while (!pq.isEmpty()) { ActionScore as = pq.poll(); AtomicAction action = actionDatabase.getAction(as.actionId); action.setConfidence(as.score); actions.add(0, action); / / Reverse insertion } return actions; } } …… The constructed lightweight neural network, after training, serves as an intent prediction model. It can be converted to a .tflite file and deployed to the smart device 10 for low-latency inference of the current vector sequence on the edge. Training samples can be constructed from edge logs and multimodal data collected with relevant authorization. Continuing the above example, several contextual vector sequences prior to a certain moment can be obtained as input based on a preset time window. One or more atomic operations (recorded in the atomic operation library) actually triggered by the user at that moment (considering a certain degree of redundancy, this moment and the preset time range thereafter can be considered) are used as supervision labels to construct training samples. If the user does not trigger any atomic operations, empty-label samples or negative samples can be constructed for training to improve the model's ability to recognize "no recommendation / only rollback presentation". The preset time window size is an empirical value, typically covering behavior from a few seconds to a few minutes ago. Several training samples can be obtained by sliding the preset time window based on a preset step size. The preset step size is also an empirical value, typically small (on the order of seconds), while the preset time window is large. The loss function can be a multi-label binary cross-entropy loss function, combined with sample weights (meaning that the loss function can be pre-weighted according to the frequency of each atomic operation, because the frequency of user behavior is extremely unbalanced; for example, browsing TikTok is very frequent, while checking the calendar is infrequent. Therefore, the number of samples with high-frequency atomic operations (or high-frequency samples) is often greater than that with low-frequency atomic operations (or low-frequency samples) in the sample set. During training, low-frequency atomic operation samples need to be given higher weights to increase the model's attention to low-frequency samples). Alternatively, a focal loss function can be used to reduce the contribution of high-frequency samples to the total loss, highlighting low-frequency samples. Furthermore, a pre-defined sampling strategy can be combined to oversample low-frequency samples or undersample high-frequency samples. By using one or more of the above-mentioned multi-label binary cross-entropy loss function, focal loss function, and pre-defined sampling strategies, the problem of the trained model being biased towards high-frequency samples due to the long-tail distribution of atomic operation samples can be alleviated. Regarding training acceptance and completion criteria, at least one or a combination of the following metrics can be used as validation set metrics: Top-K hit rate (Hit@K), MRR, macro / micro average F1, and coverage (the percentage of recommended atomic operations). On-device performance metrics of the smart device 10 can also be set (e.g., inference latency, peak memory usage, power consumption, etc.). When the validation set metrics no longer significantly improve in several consecutive training rounds, and the on-device performance metrics meet preset thresholds (e.g., inference latency does not exceed the preset latency limit, peak memory usage does not exceed the preset memory usage limit, etc.), training can be considered complete, and the intent prediction model can be obtained.

[0037] In addition, to ensure the feasibility of deployment on the edge, the trained model can be quantized and compressed (e.g., FP16 quantization or integer quantization) and then used as an intent prediction model for deployment on the edge, running with a local inference framework such as TFLite.

[0038] Another example is that if the lightweight neural network uses a multilayer perceptron (MLP), training samples can be constructed based on the logs from the smart device's edge: The context vector at a certain moment is obtained as input, and one or more atomic operations (recorded in the atomic operation library) actually triggered by the user at that moment (considering a certain degree of redundancy, this can be considered as that moment and a preset time range thereafter) are used as supervision labels to construct training samples. Negative samples and sample weights can be set by combining factors such as no-triggering and usage frequency. The same loss function, training acceptance, and completion criteria as the examples above can be used.

[0039] Continuing in this embodiment, in step S103, the smart device 10 can determine the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set based on the candidate atomic operation set, the current state of the smart device, and the corresponding current context label, and perform rendering processing based on the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set and the display attributes of the previous UI element to generate an interactive interface.

[0040] In this process, the smart device 10 can determine the current UI element corresponding to each atomic operation in the candidate atomic operation set obtained in step S102. Based on the current state of the smart device 10 and the corresponding current context label determined in step S101, and referring to a preset layout strategy, it can determine the display attributes of the current UI element. Then, based on the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set and the display attributes of the previous UI element, and using the UI framework deployed in the smart device 10, it performs differential updates (to reduce interface jumps, the UI framework deployed in the smart device 10 maintains the UI element information of the previous period; after determining the current UI element, only the UI elements that have changed are updated) and performs rendering processing to generate a current interactive interface that matches the current state and corresponding current context of the smart device 10. Throttle and / or debounce mechanisms can also be used to limit the number of times the interactive interface is rearranged in a short period of time, and stable anchor points can be used. Anchors, in the process of layout or state reconstruction, serve as deterministic, non-transient reference points for position or behavior. They do not depend on runtime states such as animation frames or temporary variables, but are based on business logic or structured data to ensure predictable interface reconstruction. The principle of minimum displacement (when the container size changes, anchored child elements should maintain their relative positions as much as possible, adjusting their size or orientation only when necessary to reduce visual jumps) determines the migration path of UI elements, so as to realize the dynamic reconstruction and smooth display of the interactive interface of smart device 10, dynamically matching the current state and current context of smart device 10, and improving the user experience.

[0041] Here is an example of a portion of Kotlin code using Jetpack Compose as the UI implementation framework: ... [Composable](cci:4: / / file: / / Composable:0:0-0:0) fun AdaptiveHomeScreen(viewModel: AdaptiveUIViewModel) { val predictedActions by viewModel.predictedActions.collectAsState() val uiState by viewModel.uiState.collectAsState() Box(modifier = Modifier.fillMaxSize()) { / / Background layer AdaptiveBackground(context = uiState.context) / / Dynamic card layer LazyColumn( modifier = Modifier.fillMaxSize(), contentPadding = PaddingValues(16.dp), verticalArrangement = Arrangement.spacedBy(12.dp) ) { items( items = predictedActions, key = { it.actionId} ) { action -> ActionCard( action = action, onClick = { viewModel.executeAction(action)}, onDismiss = { viewModel.dismissAction(action)}, modifier = Modifier.animateItemPlacement( animationSpec = spring( dampingRatio = Spring.DampingRatioMediumBouncy, stiffness = Spring.StiffnessLow ) ) ) } } / / Quick switch button FloatingActionButton( onClick = { viewModel.switchToTraditionalView()}, modifier = Modifier .align(Alignment.BottomEnd) .padding(16.dp) ) { Icon(Icons.Default.Apps, "Toggle View") } } } [Composable](cci:4: / / file: / / Composable:0:0-0:0) fun ActionCard( action: AtomicAction, onClick: () ->Unit, onDismiss: () ->Unit, modifier: Modifier = Modifier ) { val cardHeight = when { action.confidence > 0.7 -> 120.dp / / High confidence: large cards action.confidence>0.4 ->80.dp / / Medium confidence: Medium card else -> 60.dp / / Low confidence: small card } Card modifier = modifier .fillMaxWidth() .height(cardHeight) .clickable(onClick = onClick), elevation = CardDefaults.cardElevation(defaultElevation = 4.dp), shape = RoundedCornerShape(16.dp) ) { Row( modifier = Modifier .fillMaxSize() .padding(16.dp), verticalAlignment = Alignment.CenterVertically ) { / / Application icon Image( painter = rememberAsyncImagePainter ... In this application, an atomic operation is the smallest directly executable unit in the smart device 10, and does not necessarily correspond to an atomic operation of an app. The execution target of the atomic operation in this application can be an atomic operation of a specific app, or it can be an executable operating system capability of the smart device 10. For example, a smart device 10 based on the Android platform may have an application installed on it that corresponds to multiple atomic operations based on deeplink / shortcut / widget, such as "scan", "payment code", "travel", "open a page", etc.; atomic operations implemented through operating system capabilities, such as "open system camera", "open a settings page", "dial / navigate", etc.; and atomic operations with "cross-application semantic equivalence", such as "play music", may correspond to the entry points of different music apps.

[0042] Optionally, in step S101, the real-time acquisition of context-related multimodal data includes: Based on a preset frequency, real-time acquisition of context-related multimodal data is performed.

[0043] In this optional embodiment, the smart device 10 can preset the collection frequency to collect multimodal data related to the current situation of the smart device 10 in real time, for example, once per second.

[0044] Optionally, in step S101, the real-time acquisition of context-related multimodal data includes: Based on preset event triggers, real-time collection of context-related multimodal data is performed.

[0045] In this optional embodiment, the smart device 10 can preset event triggering rules. For example, when a change in the motion state of the smart device 10 is detected, multimodal data related to the current situation of the smart device 10 is collected in real time.

[0046] Optionally, in step S101, the multimodal data includes at least: temporal modal data and behavioral modal data.

[0047] In this optional embodiment, the multimodal data collected in real time by the smart device 10 in relation to the current context of the smart device 10 includes at least: time modal data and behavioral modal data.

[0048] The time modal data may include system time and system event timestamps. Specifically, system time may include date, hour, minute, second, day of the week, and whether it is a weekday / weekend. System event timestamps may be based on the timestamps of relevant interaction events in the system interaction event log (e.g., unlocking / screen on / last interaction / last foreground switch, etc., and may be combined with calendar event data with user and / or system authorization).

[0049] Among them, behavioral modal data can be obtained from application usage and interaction event logs. For example, it can be obtained from interaction event logs such as foreground application switching, application startup / exit, dwell time, click / swipe, etc., as well as execution logs (trigger / success / failure records, etc.) of atomic operations in a preset atomic operation library.

[0050] Optionally, the multimodal data further includes spatial modal data and environmental modal data.

[0051] In this optional embodiment, the multimodal data collected in real time by the smart device 10 in relation to the current context of the smart device 10 may further include: spatial modal data and environmental modal data.

[0052] Spatial modal data may include location, proximity, and mobility data of the smart device 10. For example, location and speed data obtained through GPS / BeiDou / base station / WiFi and other related positioning technologies of the smart device 10; proximity connection data obtained through WiFi, Bluetooth and other related technologies; mobility status (stationary / walking / running or driving, etc.) obtained through inertial measurement and other technologies, and the context can be further clarified by combining geofencing semantics (such as home / workplace, etc.).

[0053] The environmental modal data may include: light intensity (which can be collected by the light sensor integrated in the smart device 10), power / charging status (which can be collected by the power management unit integrated in the smart device 10), network type / connection status, and other data from the relevant environmental sensors / system status of the smart device 10. It may also include other system status data, such as screen brightness, do-not-disturb / driving mode, etc.

[0054] Optionally, in step S101, processing the multimodal data according to each modality to obtain multiple single-modal feature data vectors includes: The multimodal data is preprocessed according to modality and source, and incomplete data is handled. The processed data is then vectorized to obtain multiple single-modal feature data vectors.

[0055] In this optional embodiment, the intelligent device 10 can first process the acquired multimodal data by performing noise reduction, truncation, normalization, binning, or encoding according to modality and source. The log / clipping processing can be used for long-tailed distributed data. Normalizing long-tailed data directly would reduce the discriminative power of its head / recent data. The following truncation method can be used: setting an upper limit threshold, and uniformly setting the excess portion of the long-tailed data to a preset value. While this process loses the excess data, it retains the most important head / recent data, improving data processing efficiency without significantly affecting accuracy, which is generally advantageous. Among these methods, binning / bucketing is suitable for continuous data. The accuracy of the data itself is not the most important factor; rather, the range it falls within (i.e., which bucket it belongs to) is more crucial. For example, the battery level [0, 100%] can be binned into [0, 10%] (low), (10%, 30%] (medium-low), (30%, 80%] (normal), and (80%, 100%] (full). After binning, one-hot encoding and vectorization are performed, making it more suitable for subsequent processing.

[0056] If incomplete data exists, missing data processing is performed first. The missing data processing method can be based on the data source or type, using default values, the last valid value, or a mask. Then, vectorization is performed to obtain multiple single-modal feature data vectors corresponding to different dimensions for different modalities. The default value method can be used for features with a clear physical meaning of "zero state," such as "number of currently connected Bluetooth devices." If reading fails, a default value of 0 can be used. The last valid value method can be used for continuously and smoothly changing data, such as "GPS location." If the signal is lost at a certain sampling time, the location data collected at the previous sampling time can be used, which is more reasonable than using the default value. The mask method can be used when the missing data still contains information, such as "whether WiFi is connected." If the WiFi module is off, making it impossible to obtain connection status information, using the default value of 0 may be confusing. Adding a mask (such as is_wifi_missing=1) can clearly indicate that the data is unavailable.

[0057] For example, for real-time acquired time modal data, time data corresponding to periodic fields such as seconds, minutes, hours, dates, and weeks can be processed using sine and cosine periodic encoding; timestamp data related to interactive events can be processed using a truncation and normalization method. If incomplete data exists, missing data can be handled. Then, after vectorization processing, they are integrated into a single-modal feature data vector corresponding to the time dimension.

[0058] For example, for real-time collected behavioral modal data, the most recent preset number of application / operation sequences can be mapped into vectors and then concatenated or pooled sequentially. Usage frequency (e.g., number of unlocks, number of clicks) and interaction statistics (e.g., screen usage time) can be statistically analyzed and normalized within a preset-sized sliding window. If incomplete data exists, missing data can be handled. Then, each data is vectorized (discrete variables such as application categories can be directly vectorized, such as using one-hot encoding or embedding) and integrated into a single-modal feature data vector corresponding to the behavioral dimension.

[0059] For example, for real-time acquired spatial modal data, latitude and longitude location data can be geographically rasterized or clustered to obtain location cluster IDs, and then vectorized (e.g., embedded) for encoding; continuous value data such as velocity can be normalized; near-field connectivity data can be bucketed or clustered based on a preset hash function and then vectorized; fence hit data can determine the corresponding Boolean features. If there is incomplete data, missing data can be handled. Then, after each data is vectorized, they are integrated into a single-modal feature data vector corresponding to the spatial dimension.

[0060] For example, for real-time acquired environmental modal data, continuous values ​​such as light intensity and battery power can be normalized or binned to convert them into discrete values. For instance, light intensity can be normalized to [0,1]. Then, one-hot encoding is applied to the converted discrete values, as well as discrete values ​​acquired for network type / connection status, charging status, etc. If incomplete data exists, missing data can be handled. After each is vectorized, they are integrated into a single-modal feature data vector corresponding to the environmental dimension.

[0061] In some optional embodiments, in step S101, the intelligent device 10 may concatenate the obtained single-modal feature data vectors corresponding to the time dimension and the single-modal feature data vectors corresponding to the behavior dimension into a first feature data vector in a preset order. In other optional embodiments, in step S101, the intelligent device may concatenate the obtained single-modal feature data vectors corresponding to the time dimension, the single-modal feature data vectors corresponding to the behavior dimension, the single-modal feature data vectors corresponding to the spatial dimension, and / or the single-modal feature data vectors corresponding to the environmental dimension into a first feature data vector in a preset order. The specific dimensions used can be selected according to the specific application scenario and are not limited here.

[0062] The following is a partial Java code example of an exemplary multimodal data acquisition service: ... / / Contextual Data Acquisition Service public class ContextCollectorService extends Service { private SensorManager sensorManager; private LocationManager locationManager; private ActivityRecognitionClient activityClient; @Override public void onCreate() { / / Initialize Sensor Manager sensorManager = (SensorManager) getSystemService(SENSOR_SERVICE); locationManager = (LocationManager) getSystemService(LOCATION_SERVICE); / / Register sensor monitoring registerSensors(); / / Start location listener startLocationTracking() / / Listen for system events registerSystemListeners(); } private ContextVector collectContextData() { ContextVector vector = new ContextVector(); / / Time modal feature data vector.addTimeFeatures(getCurrentTimeFeatures()); / / Spatial modal feature data vector.addLocationFeatures(getLocationFeatures()); / / Behavioral modality feature data vector.addActivityFeatures(getActivityFeatures()); / / Environmental modal feature data vector.addEnvironmentFeatures(getEnvironmentFeatures()); return vector.normalize(); } } ... V_context = [ / / Time modal data hour_sin, hour_cos, / / Hour period encoding day_of_week_sin, day_of_week_cos, is_weekend, is_workday time_since_last_use, has_calendar_event, minutes to next event, event_type_embedding[4], / / Spatial modal data location_cluster_id_embedding[8], / / Semantic location after clustering movement_speed, movement_type_onehot[4], / / stationary / walking / driving / public transport wifi_hash_embedding[4], is_home, is_work, is_commuting, / / Behavioral modal data app_usage_history

[10] , / / Embeddings of the 10 most recently used applications unlock_frequency, screen_time_today, last_app_category, user_interaction_rate, / / Environmental modal data light_level_normalized, noise_level_normalized, battery_level, is_charging, network_type_onehot[4] ... Optionally, in step S101, processing the first feature data vector to obtain a second feature data vector of a first preset number of dimensions includes: The first feature data vector is normalized and linearly mapped to obtain a second feature data vector with a first preset number of dimensions.

[0063] In this optional embodiment, in step S101, the obtained first feature data vector can be normalized. For example, layer normalization or L2 normalization can be used to make the overall distribution of the vector more stable, which is more conducive to convergence when training the intent prediction model. After normalization, a linear mapping can be performed to project the vector onto a first preset number of dimensions, so as to compress the high-dimensional features in the vector into a more compact low-dimensional space, and feature fusion can be performed to obtain a second feature data vector of the first preset number of dimensions. The second feature data vector of fixed dimensions can determine the stability of the network structure of the intent prediction model, thereby improving the training and prediction efficiency. The first preset number can be determined according to the hardware and software resource constraints of the smart device 10.

[0064] In this application, atomic operations obtained in different ways can be extracted and abstracted into a unified description structure, and after deduplication, merging and priority-based processing, an atomic operation library can be constructed.

[0065] Optionally, in step S102, the construction of the preset atomization operation library includes: Extract and abstract each atomic operation of the smart device, record the source category and availability status of each atomic operation, and set the priority and fallback relationship of each atomic operation; After deduplication, merging, and verification of all atomic operations, each atomic operation is recorded as a record to build a pre-defined atomic operation library. Each atomic operation record includes at least the following fields: identifier, execution method type, target parameters, and display information. In the preset atomization operation library, a one-to-one mapping relationship is maintained between the identifier of the atomization operation and the dimension index output by the intention prediction model.

[0066] In this optional embodiment, each atomic action of the smart device 10 is pre-included in a pre-deployed atomic action library to construct the pre-deployed atomic action library. First, each atomic action of the smart device 10 is extracted and abstracted, and the source type and availability status of each atomic action are recorded, along with the priority and fallback relationship of each atomic action. Then, after deduplication, merging, and verification of all obtained atomic actions, each atomic action is abstracted into a unified description structure as a record in the atomic action library, resulting in the pre-deployed atomic action library. Each atomic action record includes at least the following fields: identifier (e.g., action_id), execution method type (e.g., action_type), target parameter (e.g., target_param), and display information (e.g., display). The identifier field uniquely identifies the atomic action and serves as a mapping index between the intent prediction model output dimension and the atomic action; the execution method type field indicates the action type corresponding to the atomic action; and the target parameter field locates and executes the executable entry point and its parameters corresponding to the atomic action. Data (e.g., application identifier / application package name, and its corresponding execution payload (e.g., shortcutId / widgetId / system capability identifier on the Android platform, etc.)); Display information fields can be used for interface rendering, typically including at least a title and icon, and may also include permission and availability information (e.g., permission requirements, preconditions, availability / failure reason), source and deduplication and merging information (e.g., developer declaration / learning / automation source, deduplication key, priority / fallback), context and scenario information (e.g., applicable scenario tags, category tags), quality and cost information (e.g., success rate, time consumption, risk level), and feedback statistics information (e.g., recent use, frequency, user pinning / hiding, etc.) to achieve filtering, sorting, stable display, and closed-loop optimization functions on the smart device 10. This pre-defined atomic operation library also maintains a one-to-one mapping between the identifiers of atomic operations and the dimension indexes (e.g., index_id) output by the intent prediction model. The number of dimensions output by the intent prediction model is the same as the number of atomic operation records in this pre-defined atomic operation library, allowing the output of the intent prediction model to be directly mapped to the corresponding atomic operations. An exemplary pre-defined atomic operation library structure using an SQL database is as follows: ... CREATE TABLE atomic_actions ( action_id INTEGER PRIMARY KEY, action_type TEXT,-- 'APP_LAUNCH', 'DEEP_LINK', 'SHORTCUT' package_name TEXT, target_action TEXT, parameters TEXT, -- JSON format display_title TEXT, display_icon BLOB, usage_count INTEGER, last_used_timestamp INTEGER ); CREATE INDEX idx_package ON atomic_actions(package_name); CREATE INDEX idx_usage ON atomic_actions(usage_count DESC); ... Optionally, the extraction and abstraction of each atomic operation of the smart device includes: Extract any callable capability entry point declared by the developer and abstract it into an atomic operation of the smart device; Extract any shortcut action generated by learning the historical operation sequence under user authorization based on the operating system of the smart device, and abstract it into an atomic operation of the smart device; Extract any executable action encapsulated by the operating system of the smart device, determine and abstract it into an atomic operation of the smart device.

[0067] In this optional embodiment, each atomic operation of the smart device 10 can be extracted and abstracted in the following three ways: First, extract any callable capability entry point declared by the developer and abstract it as an atomic operation of the smart device 10. The atomic operation obtained by this method has high reliability. The developer declaration usually includes a formal explanation from the application developer regarding the integrated third-party SDK functions, data processing behavior, permission usage, etc., which can be used to improve transparency and ensure users' right to know and compliance requirements. The callable capability entry point can be a fine-grained function interface provided by a specific App on the smart device 10, which can usually be implemented through deep link technology, shortcuts, or functions provided by the operating system (such as Widgets / Intents on the Android platform), such as the scan function of the WeChat App and the payment code of the Alipay App. Second, extract any shortcut action generated by the operating system of the smart device 10 based on the high-frequency historical operation sequence learned by the user under authorization, and abstract it as an atomic operation of the smart device 10. Third, any executable action encapsulated by the operating system of the smart device 10 is extracted and abstracted into an atomic operation of the smart device. If an atomic operation lacks a corresponding direct executable entry point, it can be encapsulated into an executable entry point and extracted as an atomic operation in the preset atomic operation library through the automation / accessibility capabilities of the operating system of the smart device 10.

[0068] This system allows for deduplication and merging of all atomic operations extracted and abstracted using different methods. Each processed atomic operation is stored with a unified description structure, forming a pre-defined atomic operation library. The merging process primarily targets atomic operations with equivalent functionality but different extraction methods. For example, the atomic operation implementing the function "Open WeChat to scan" can be achieved through a single atomic operation: using the shortcut declared by the official developer. However, if the operating system detects frequent user calls to the path "Open WeChat" -> "Click to scan" based on historical operation logs, it can generate a script simulating the click and encapsulate it into an executable entry point. Executing the atomic operation of this executable entry point can also achieve the same function. Therefore, the merging process prioritizes retaining the former atomic operation, while using the latter as a supplement or for verification. Through deduplication and / or merging, redundant atomic operations can be avoided in the pre-defined atomic operation library.

[0069] Optionally, in step S103, the display attributes include at least: visibility, size, position, and hierarchy.

[0070] In this optional embodiment, the display attributes of UI elements include at least visibility, size, position, and hierarchy. The UI framework deployed in the smart device 10 combines the candidate atomic operation set determined by the intent prediction model output, the current state determined by the current single-modal feature vector, and the corresponding current context label to calculate the display priority for each atomic operation in the candidate atomic operation set. The display priority of each atomic operation is related to the confidence score of the atomic operation and the degree of matching with the current state and the corresponding current context label, and can be determined according to a preset layout strategy. Then, according to the preset layout strategy, the corresponding current UI element is generated and the layout solution is performed to obtain the visibility, size, position, and hierarchy of each UI element in the current UI element, so as to avoid occlusion and achieve the first screen accessibility of the interface.

[0071] In interface design, common UI elements include controls, navigation elements, information display elements, containers and layout elements, temporary windows, etc. In this application, the UI elements on the interactive interface of the smart device 10 can dynamically adapt to changes based on the current state and corresponding current context of the smart device 10. Specifically, based on a preset layout strategy and combined with the output of an intent prediction model, the Top-K atomic operations corresponding to the current state and corresponding current context can be determined. Furthermore, based on the confidence score of each atomic operation, the visibility, size, position, and hierarchy of the UI element corresponding to each atomic operation can be determined. For example, a confidence score threshold can be preset. For the visibility of a UI element, if the confidence score of the atomic operation corresponding to the UI element is greater than a first preset threshold, it is visible; otherwise, it is invisible / hidden. The size template can be determined based on the confidence score of the atomic operation corresponding to the UI element. The display position and hierarchy can be sorted in descending order according to the confidence scores of the Top-K atomic operations, or expanded around the visual center.

[0072] This application may also employ a fallback mechanism (typically referring to the operating system automatically switching to a preset backup plan to ensure service continuity when the main process or main service fails). In this application, if the confidence scores of the corresponding Top-K atomic operations are all very low, to avoid a poor user experience, the operating system may trigger a fallback mechanism and execute a fallback process: instead of displaying the UI elements corresponding to these atomic operations with low confidence scores, a fallback display may be used, such as displaying a general search box or application drawer entry for users to manually search, or displaying persistent items, such as only displaying frequently used apps manually selected by the user, i.e., falling back to the traditional interaction mode to ensure the usability of the interactive interface.

[0073] Optionally, the method for generating an interactive interface further includes: S104, based on the trigger operation of any UI element in the interactive interface, calls the execution entry point of the corresponding atomic operation to execute the corresponding operation.

[0074] In this optional embodiment, in step S104, the smart device 10 can call the execution entry of the atomic operation corresponding to the UI element according to the user's trigger operation on any UI element in the displayed interactive interface, so as to execute the corresponding operation. The mapping relationship between the identifier of each atomic operation in the preset atomic operation library and its corresponding execution entry parameter can be maintained in the preset atomic operation library or in the local database on the device side.

[0075] Optionally, the method for generating an interactive interface further includes: S105 records the feedback data of the corresponding operation and determines the user preference tags based on the recorded historical feedback data; In step S103, determining the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set based on the candidate atomic operation set, the current state of the smart device, and the corresponding current context label includes: Based on the candidate atomic operation set, the current state of the smart device and the corresponding current context tag, and the user preference tag, determine the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set.

[0076] In this optional embodiment, in step S105, feedback data corresponding to the execution of the corresponding operation can be recorded, such as usage duration, cancellation, execution success / failure, etc. A historical behavior table (such as the UserInteractionHistory table) can be maintained in a local database on the device side, or corresponding fields can be configured for recording each atomic operation record in a preset atomic operation library. User preference tags can be determined based on the recorded historical feedback data. Then, in step S103, the smart device 10 can determine the current UI element corresponding to each atomic operation in the candidate atomic operation set obtained in step S102. Based on the current state of the smart device 10 and the corresponding current context tag determined in step S101, and the user preference tag determined in step S105, the display attributes of the current UI element can be determined with reference to the preset layout strategy, which can further improve the matching degree and user experience.

[0077] Optionally, the method for generating an interactive interface further includes: S106 adjusts the relevant attributes of the corresponding atomic operations based on the recorded historical feedback data, and / or uses them to iteratively train the intent prediction model.

[0078] In this optional embodiment, in step S106, the recorded historical feedback data can also be used to adjust the relevant attributes of the corresponding atomic operations. For example, the availability and confidence score threshold of each atomic operation in the preset atomic operation library can be dynamically adjusted (each atomic operation record can be configured with a confidence score threshold field to specify the minimum confidence score threshold at which the atomic operation can be selected into the candidate atomic operation set). Historical feedback data can also be used to construct new samples for iterative training of the aforementioned intent prediction model. This makes the subsequently generated interactive interface more suitable and further improves the user experience. Federated learning, distillation, or privacy-preserving aggregation can be used for cross-device model iterative training to reduce the risk of uploading raw data. Federated learning, distillation, or privacy-preserving aggregation are existing technologies commonly used for model iterative training and will not be elaborated upon here.

[0079] The specific values ​​shown in the examples of the above embodiments and / or optional embodiments of this application are only examples, and the actual values ​​should be configured / set in combination with the actual application scenario.

[0080] The executable entry points for the Android platform shown in the above embodiments and / or optional embodiments of this application, such as components like Intent / Widget / Accessibility, Shortcut functionality, and the deployed lightweight neural network FTLite based on the Keras framework, are merely examples. The technical solutions of this application are not limited to the Android platform; other operating system platforms can also utilize corresponding executable entry points and related frameworks to implement the technical solutions of this application.

[0081] Figure 2 This invention illustrates a system for generating an interactive interface according to another aspect of this application, deployed on a smart device, wherein, in one embodiment, the system includes: The multimodal data acquisition and processing layer unit 210 is used to acquire context-related multimodal data in real time, process the multimodal data according to modality to obtain multiple single-modal feature data vectors, and concatenate them into a first feature data vector in a preset order, and process the first feature data vector to obtain a second feature data vector of a first preset number of dimensions, and determine the current state of the smart device and the corresponding current context label based on the current multiple single-modal feature vectors; The intent prediction unit 220 is used to determine several second feature data vectors based on a preset time window and form a current vector sequence. The current vector sequence is input into the intent prediction model to obtain an atomic operation confidence vector of a second preset number of dimensions corresponding to a preset atomic operation library. The position index of each component in the atomic operation confidence vector corresponds one-to-one with the identifier of an atomic operation in the preset atomic operation library. The value of each component is the confidence score of the corresponding atomic operation. The K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element of the candidate atomic operation set includes the position index of an atomic operation and its confidence score. The interactive interface generation unit 230 is used to determine the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set based on the candidate atomic operation set, the current context label and the current state of the smart device, and to perform rendering processing based on the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set and the display attributes of the previous UI element to generate an interactive interface.

[0082] In this system embodiment, the system is deployed in a smart device, and the hardware and software environment of the smart device is the same as that of the aforementioned smart device 10.

[0083] In this system embodiment, with authorization from the user and / or the Android operating system, the multimodal data acquisition and processing layer unit 210 of the system can collect multimodal data related to the current context in real time. Then, according to the modality, the data of the same modality in the obtained multimodal data are processed to obtain multiple single-modal feature data vectors of different modalities. These single-modal feature data vectors of different modalities are then concatenated into a first feature data vector according to a preset order, and further processed to obtain a second feature data vector of a first preset number of dimensions. Furthermore, based on the current single-modal feature data vectors of multiple different modalities, the current state of the smart device 10 and the corresponding current context label are determined. For non-periodic discrete numerical data, one-hot (one-time) data processing can be used. Encoding methods include hot hashing, hash buckets, clustering, and embedding, or combinations thereof. For periodic data, sine / cosine (sin / cosine) encoding or bucketing encoding can be used. To facilitate subsequent fusion using gating weighting mechanisms (e.g., when fusing different modal features, appropriate activation functions can be used to dynamically adjust the contributions of different modal features, ensuring that the accuracy and robustness of the subsequent intention prediction model output improve with the number of modalities collected), the dimension of each single-modal feature data vector can be preset to a fixed value. For incomplete data, missing data processing can be performed to ensure that the dimension of the single-modal feature data vector obtained from the missing data is the corresponding preset value.

[0084] Continuing in this system embodiment, the intent prediction unit 220 of the system can obtain several second feature data vectors according to a preset time window through the multimodal data acquisition and processing layer unit 210, and combine these several second feature data vectors into a current vector sequence. Then, the current vector sequence is input into the intent prediction model deployed in the smart device 10 to obtain an atomic operation confidence vector of a second preset number of dimensions corresponding to the preset atomic operation library deployed in the smart device 10. The position index of each component in the atomic operation confidence vector corresponds one-to-one with the identifier of an atomic operation in the preset atomic operation library, and the value of each component is the confidence score of the corresponding atomic operation. Then, the K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element of the candidate atomic operation set includes the position index of an atomic operation and its confidence score.

[0085] Continuing in this system embodiment, the system's interface generation unit 230 can determine the current UI element corresponding to each atomic operation in the candidate atomic operation set obtained by the intent prediction unit 220. Based on the current state of the smart device 10 and the corresponding current context label determined by the multimodal data acquisition and processing layer unit 210, and referring to a preset layout strategy, the display attributes of the current UI element can be determined. Furthermore, based on the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set and the previous UI element... The display attributes, based on the UI framework deployed in the smart device 10, undergo differential updates (to reduce interface jumps, the UI framework deployed in the smart device 10 maintains the UI element information of the previous period's interface; after determining the current UI elements, only the UI elements that have changed are updated) before rendering processing to generate the current interactive interface that matches the current state and corresponding current context of the smart device 10. Throttle and / or debounce mechanisms can also be used to limit the number of times the interactive interface is rearranged in a short period of time, and stable anchor points are employed. Anchors, in the process of layout or state reconstruction, serve as deterministic, non-transient reference points for position or behavior. They do not depend on runtime states such as animation frames or temporary variables, but are based on business logic or structured data to ensure predictable interface reconstruction. The principle of minimum displacement (when the container size changes, anchored child elements should maintain their relative positions as much as possible, adjusting their size or orientation only when necessary to reduce visual jumps) determines the migration path of UI elements, so as to realize the dynamic reconstruction and smooth display of the interactive interface of smart device 10, dynamically matching the current state and current context of smart device 10, and improving the user experience.

[0086] Optionally, the system for generating an interactive interface further includes: The atomic operation library maintenance unit 240 is used to extract and abstract each atomic operation of the smart device, record the source category and availability status of each atomic operation, and set the priority and fallback relationship of each atomic operation. After deduplication, merging and verification of all atomic operations, each atomic operation is recorded as a record to build a preset atomic operation library. Each atomic operation record includes at least the following fields: identifier, category, target parameter and display information. In the preset atomic operation library, a one-to-one mapping relationship is maintained between the identifier of the atomic operation and the dimension index output by the intent prediction model.

[0087] In this optional system embodiment, the atomic operation library maintenance unit 240 of the system can extract and abstract each atomic operation of the smart device 10, record the source type and availability status of each atomic operation, and set the priority and fallback relationship of each atomic operation. Then, after deduplication, merging, and verification of all obtained atomic operations, each atomic operation is abstracted into a unified description structure as a record in the atomic operation library, resulting in the preset atomic operation library. Each atomic operation record includes at least the following fields: identifier (e.g., action_id), execution method type (e.g., action_type), target parameter (e.g., target_param), and display information (e.g., display). The identifier field can be used to uniquely identify the atomic operation and serve as a mapping index between the intent prediction model output dimension and the atomic operation; the execution method type field can be used to indicate the action type corresponding to the atomic operation; and the target parameter field can be used to locate and execute the executable entry point and its parameters corresponding to the atomic operation. Data (e.g., application identifier / application package name, and its corresponding execution payload (e.g., shortcutId / widgetId / system capability identifier on the Android platform, etc.)); Display information fields can be used for interface rendering, typically including at least a title and icon, and may also include permission and availability information (e.g., permission requirements, preconditions, availability / failure reason), source and deduplication and merging information (e.g., developer declaration / learning / automation source, deduplication key, priority / fallback), context and scenario information (e.g., applicable scenario tags, category tags), quality and cost information (e.g., success rate, time consumption, risk level), and feedback statistics information (e.g., recent use, frequency, user pinning / hiding, etc.) to achieve filtering, sorting, stable display, and closed-loop optimization functions on the smart device 10. In this preset atomic operation library, a one-to-one mapping relationship is maintained between the identifier of the atomic operation and the dimension index (such as index_id) output by the above intention prediction model. The number of dimensions output by the above intention prediction model is the same as the number of atomic operation records in the preset atomic operation library, which allows the output of the above intention prediction model to be directly mapped to the corresponding atomic operation.

[0088] In the above system embodiments, any method steps that are not mentioned in the executable steps of each component of the system are the same as those in the aforementioned related method embodiments and / or optional embodiments, and will not be repeated here.

[0089] According to another aspect of this application, a computer-readable medium is also provided, the computer-readable medium storing computer-readable instructions that can be executed by a processor to implement some or all of the foregoing method embodiments and / or optional embodiments.

[0090] It should be noted that the method embodiments and / or optional embodiments in this application do not strictly limit the order of execution of each step, as long as the method embodiments and / or optional embodiments can solve the defects existing in the prior art, achieve the inventive purpose of this application, and obtain beneficial effects. The method embodiments and / or optional embodiments in this application can be implemented in software and / or combinations of software and hardware. The software program involved in this application can be executed by a processor to implement the steps or functions of the above embodiments. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium.

[0091] Furthermore, part or all of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions.

[0092] According to another aspect of this application, an apparatus for generating an interactive interface is also provided. The apparatus includes: a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run part or all of the methods and / or technical solutions of the foregoing embodiments.

[0093] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0094] In this application, when terms such as "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" are used, the indicated orientation and / or positional relationship is based on the orientation and / or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation. Furthermore, some of the above terms, in addition to indicating orientation or positional relationship, can also be used to indicate other meanings; for example, the term "upper" can also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application according to the specific circumstances. Furthermore, the terms "installation," "setting," "equipped with," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to an internal connection between two devices, components, or constituent parts. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0095] Furthermore, the terms "first," "second," etc., are primarily used to distinguish different devices, units, modules, elements, circuits, or components (which may be the same or different in specific type and construction), and are not intended to indicate or imply the relative importance, order, and / or quantity of the indicated devices, units, modules, elements, circuits, or components. Unless otherwise stated, "a plurality of" means two or more.

[0096] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the device claims may also be implemented by a single unit or device through software and / or hardware.< / atomicaction> < / actionscore> < / atomicaction> < / atomicaction>

Claims

1. A method for generating an interactive interface, characterized in that, Applied to smart devices, the method includes: Real-time acquisition of context-related multimodal data; processing of the multimodal data according to modality to obtain multiple single-modal feature data vectors; concatenation of these vectors into a first feature data vector in a preset order; processing of the first feature data vector to obtain a second feature data vector with a first preset number of dimensions; and determination of the current state of the smart device and the corresponding current context label based on the current multiple single-modal feature vectors. Based on a preset time window, several second feature data vectors are determined and formed into a current vector sequence. The current vector sequence is input into the intention prediction model to obtain an atomic operation confidence vector of a second preset number of dimensions corresponding to a preset atomic operation library. The position index of each component in the atomic operation confidence vector corresponds one-to-one with the identifier of an atomic operation in the preset atomic operation library. The value of each component is the confidence score of the corresponding atomic operation. The K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element of the candidate atomic operation set includes the position index of an atomic operation and its confidence score. Based on the candidate atomic operation set, the current state of the smart device, and the corresponding current context label, the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set are determined. Based on the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set and the display attributes of the previous UI element, rendering processing is performed to generate an interactive interface.

2. The method according to claim 1, characterized in that, The real-time acquisition of context-related multimodal data includes: Based on a preset frequency, real-time acquisition of context-related multimodal data is performed.

3. The method according to claim 1, characterized in that, The real-time acquisition of context-related multimodal data includes: Based on preset event triggers, real-time collection of context-related multimodal data is performed.

4. The method according to claim 1, characterized in that, The multimodal data includes at least: temporal modal data and behavioral modal data.

5. The method according to claim 4, characterized in that, The multimodal data also includes spatial modal data and environmental modal data.

6. The method according to claim 1, characterized in that, The process of processing the multimodal data according to each modality yields multiple single-modal feature data vectors, including: The multimodal data is preprocessed according to modality and source, and incomplete data is handled. The processed data is then vectorized to obtain multiple single-modal feature data vectors.

7. The method according to claim 1, characterized in that, The step of processing the first feature data vector to obtain a second feature data vector of a first preset number of dimensions includes: The first feature data vector is normalized and linearly mapped to obtain a second feature data vector with a first preset number of dimensions.

8. The method according to claim 1, characterized in that, The construction of the preset atomization operation library includes: Extract and abstract each atomic operation of the smart device, record the source category and availability status of each atomic operation, and set the priority and fallback relationship of each atomic operation; After deduplication, merging, and verification of all atomic operations, each atomic operation is recorded as a record to build a pre-defined atomic operation library. Each atomic operation record includes at least the following fields: identifier, execution method type, target parameters, and display information. In the preset atomization operation library, a one-to-one mapping relationship is maintained between the identifier of the atomization operation and the dimension index output by the intention prediction model.

9. The method according to claim 8, characterized in that, The extraction and abstraction of each atomic operation of the smart device includes: Extract any callable capability entry point declared by the developer and abstract it into an atomic operation of the smart device; Extract any shortcut action generated by learning the historical operation sequence under user authorization based on the operating system of the smart device, and abstract it into an atomic operation of the smart device; Extract any executable action encapsulated by the operating system of the smart device, determine and abstract it into an atomic operation of the smart device.

10. The method according to claim 1, characterized in that, The display attributes include at least: visibility, size, position, and hierarchy.

11. The method according to claim 1, characterized in that, The method further includes: Based on the trigger operation of any UI element in the interactive interface, the execution entry point of the corresponding atomic operation is called to execute the corresponding operation.

12. The method according to claim 1, characterized in that, The method further includes: Record feedback data of the corresponding operations performed, and determine user preference tags based on the recorded historical feedback data; The step of determining the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set, based on the candidate atomic operation set, the current state of the smart device, and the corresponding current context label, includes: Based on the candidate atomic operation set, the current state of the smart device and the corresponding current context tag, and the user preference tag, determine the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set.

13. The method according to claim 12, characterized in that, The method further includes: Based on recorded historical feedback data, adjust the relevant attributes of the corresponding atomic operations, and / or use them to iteratively train the intent prediction model.

14. A system for generating interactive interfaces, characterized in that, Deployed on smart devices, the system includes: The multimodal data acquisition and processing layer unit is used to acquire context-related multimodal data in real time, process the multimodal data according to modality to obtain multiple single-modal feature data vectors, and concatenate them into a first feature data vector in a preset order. The first feature data vector is also processed to obtain a second feature data vector with a first preset number of dimensions. Based on the current multiple single-modal feature vectors, the current state of the smart device and the corresponding current context label are determined. An intent prediction unit is used to determine several second feature data vectors based on a preset time window and form a current vector sequence. The current vector sequence is input into the intent prediction model to obtain an atomic operation confidence vector of a second preset number of dimensions corresponding to a preset atomic operation library. The position index of each component in the atomic operation confidence vector corresponds one-to-one with the identifier of an atomic operation in the preset atomic operation library. The value of each component is the confidence score of the corresponding atomic operation. The K components with the highest confidence scores are determined to obtain a candidate atomic operation set. Each element of the candidate atomic operation set includes the position index of an atomic operation and its confidence score. The interactive interface generation unit is used to determine the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set based on the candidate atomic operation set, the current context label and the current state of the smart device, and to perform rendering processing based on the display attributes of the current UI element corresponding to each atomic operation in the candidate atomic operation set and the display attributes of the previous UI element to generate an interactive interface.

15. The system according to claim 14, characterized in that, The system also includes: The atomic operation library maintenance unit is used to extract and abstract each atomic operation of the smart device, record the source category and availability status of each atomic operation, and set the priority and fallback relationship of each atomic operation. After deduplication, merging and verification of all atomic operations, each atomic operation is recorded as a record to construct a preset atomic operation library. Each atomic operation record includes at least the following fields: identifier, category, target parameter and display information. In the preset atomic operation library, a one-to-one mapping relationship is maintained between the identifier of the atomic operation and the dimension index output by the intent prediction model.

16. A computer-readable medium, characterized in that, It stores computer-readable instructions that are executed by a processor to implement the method as described in any one of claims 1 to 13.

17. A smart device for generating an interactive interface, characterized in that, The intelligent device includes: One or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform the method as described in any one of claims 1 to 13.

Images