Multimodal end-to-side model-driven intelligent application interaction method and system

Abstract

This invention provides a multimodal edge-side model-driven intelligent application interaction method and system, relating to the field of model interaction technology. The method includes acquiring user interaction commands and determining the task type, selecting a corresponding lightweight deep learning model; processing multimodal information using an attention mechanism and optimizing the model output through Monte Carlo tree search; constructing a dynamic response policy tree and calculating the interaction utility score, using bundle search pruning to select the optimal interaction path; and generating and executing interaction response commands. This invention improves the efficiency of multimodal information processing, the accuracy of interaction responses, and the degree of personalization, enhancing the user experience.

Description

Technical Field

[0001] This invention relates to model interaction technology, and more particularly to a multimodal edge-driven intelligent application interaction method and system. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal human-computer interaction has become an important development direction for intelligent applications. Multimodal interaction technology integrates multiple information channels such as voice, image, and text to provide users with a more natural and efficient interactive experience. Especially on terminal devices such as smartphones and smart home devices, users' demands for real-time, accurate, and personalized interaction are increasing. Currently, multimodal interaction technology mainly relies on large-scale cloud models for processing, transmitting user input to cloud servers for analysis and response generation, and then returning the results to the terminal device.

[0003] However, existing multimodal interaction technologies face the following shortcomings: First, cloud-based interaction methods suffer from network latency, making it difficult to meet users' demands for real-time interaction, especially in environments with unstable network conditions, where the user experience is severely compromised. Second, existing interaction methods lack effective multimodal information fusion mechanisms, often simply splicing or processing data from different modalities separately, failing to fully exploit the complementarity and correlation of multimodal information, resulting in low accuracy in interaction understanding. Finally, existing technologies generally employ fixed response strategies, unable to dynamically adjust interaction paths based on users' personalized needs and interaction context, lacking intelligent decision-making optimization capabilities, and failing to provide a personalized and coherent interactive experience. Summary of the Invention

[0004] The embodiments of the present invention provide a multimodal end-side model-driven intelligent application interaction method and system, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a multimodal edge-side model-driven intelligent application interaction method, comprising:

[0006] Obtain user-inputted interaction commands, determine the type of interaction task to be executed based on the interaction commands, and select a corresponding lightweight deep learning model from a preset multimodal model library based on the interaction task type.

[0007] An attention mechanism is used to extract and fuse features from the multimodal information in the interaction command to generate a unified feature vector. The unified feature vector is then input into the lightweight deep learning model. The Monte Carlo tree search method is used to sample and optimize the model output to obtain the interaction response result with the highest confidence.

[0008] A dynamic response strategy tree is constructed based on the interaction response results. Multiple candidate branch nodes are set in the dynamic response strategy tree. An interaction utility score is calculated for each candidate branch node. The candidate branch nodes are pruned using a bundle search algorithm. The branch node with the highest interaction utility score is selected as the optimal interaction path.

[0009] Generate corresponding interactive response instructions based on the optimal interaction path, and execute the interactive response instructions to complete the intelligent application interaction.

[0010] Determining the type of interactive task to be executed based on the interactive instruction, and selecting a corresponding lightweight deep learning model from a preset multimodal model library based on the interactive task type, includes:

[0011] Extract the feature vector from the interaction instruction, construct the interaction task feature matrix based on the feature vector, perform feature decomposition on the interaction task feature matrix to obtain the interaction task feature decomposition result, calculate the similarity between the interaction task feature decomposition result and the preset task type feature template, and determine the type of interaction task to be executed corresponding to the interaction instruction.

[0012] Based on the type of interactive task to be executed, a task processing requirement description is generated. Based on the task processing requirement description, a deep learning model scoring matrix is ​​established according to the performance parameters, resource consumption parameters, and computational complexity parameters of each deep learning model in the preset multimodal model library. By performing feature matching operations on the deep learning model scoring matrix, a lightweight deep learning model that best matches the task processing requirement description is selected.

[0013] The unified feature vector is input into the lightweight deep learning model, and the model output is sampled and optimized using the Monte Carlo tree search method to obtain the interactive response results with the highest confidence, including:

[0014] A unified feature vector is input into a lightweight deep learning model to obtain an initial interaction response result; an initial confidence score is calculated for the initial interaction response result, and a search root node is generated based on the initial confidence score;

[0015] The search root node is expanded using a depth-first traversal strategy to generate multiple candidate response branches. Response parameters are configured for each candidate response branch, and the initial interaction response result is transformed based on the response parameters to generate diverse candidate interaction response results.

[0016] The candidate interaction response results are constructed as child nodes, an initial access count value is assigned to each child node, and a sampling probability distribution is determined based on the initial access count value; multiple rounds of random sampling are performed on each child node according to the sampling probability distribution, feature extraction is performed on the sampled response results, and the confidence score of the sampled response results is calculated.

[0017] A value evaluation function is constructed based on the confidence score and the access count value. The optimal child node is selected by the calculation result of the value evaluation function. The access count value of the optimal child node is compared with a preset sampling threshold. When the access count value reaches the preset sampling threshold, the result with the highest confidence score is selected from all candidate interaction response results as the final output interaction response result.

[0018] A depth-first traversal strategy is used to expand the search root node, generating multiple candidate response branches. Response parameters are configured for each candidate response branch, and the initial interaction response result is transformed based on the response parameters to generate diverse candidate interaction response results, including:

[0019] Mark the root node of the search as the current processing node, construct a depth-first traversal stack based on the current processing node, and obtain the initial interaction response result corresponding to the current processing node based on the depth-first traversal stack;

[0020] A response branch generation function is constructed for the current processing node. The semantic feature vector of the initial interaction response result is calculated based on the response branch generation function. Based on the semantic feature vector, semantic similarity calculation, expression integrity evaluation and interaction coherence analysis are performed to generate multiple candidate response branches.

[0021] A set of response parameters is constructed for each candidate response branch, a response transformation matrix is ​​constructed based on the set of response parameters, and the initial interaction response result is linearly combined and transformed by the response transformation matrix to generate candidate interaction response results with different characteristics.

[0022] Each candidate interaction response result is constructed as a new search node, and the new search node is pushed onto the depth-first traversal stack; when the depth of the depth-first traversal stack reaches a preset traversal threshold, all generated candidate interaction response results are output.

[0023] A dynamic response strategy tree is constructed based on the interaction response results. Multiple candidate branch nodes are set in the dynamic response strategy tree. An interaction utility score is calculated for each candidate branch node. The candidate branch nodes are then pruned using a bundle search algorithm. The branch node with the highest interaction utility score is selected as the optimal interaction path, including:

[0024] Multiple candidate branch nodes are set in the dynamic response strategy tree, and an interaction intent vector is calculated for each candidate branch node. The interaction intent vector is used to extract interaction intent features through a semantic encoding function.

[0025] An interaction state transition probability matrix is ​​constructed based on the interaction intent features. An initial interaction utility score is calculated for each candidate branch node based on the interaction state transition probability matrix. The candidate branch nodes are arranged in descending order according to the initial interaction utility scores to generate a candidate node sequence. A bundle search optimization function is constructed based on the candidate node sequence.

[0026] Based on the bundle search optimization function, the historical best branch information is continuously updated by maintaining a search bundle of fixed width. A dynamic pruning threshold is calculated based on the ratio of the historical best branch information to the current interaction utility score. The candidate node sequence is iteratively pruned based on the dynamic pruning threshold by the bundle search optimization function, and the best branch information in the search bundle is updated in each iteration.

[0027] After the beam search optimization function completes a preset number of iterations, it extracts the branch node with the highest interaction utility score from the search beam and determines the path corresponding to the branch node as the optimal interaction path.

[0028] Arrange the candidate branch nodes in descending order according to the initial interaction utility score to generate a candidate node sequence. Construct a bundle search optimization function based on the candidate node sequence, including:

[0029] Record the initial interaction utility score difference between adjacent nodes in the candidate node sequence, calculate the node weight coefficient based on the initial interaction utility score difference, group the candidate node sequence according to the node weight coefficient, divide the candidate node sequence into multiple subsequences, and construct a bundle search optimization function based on the subsequences.

[0030] A second aspect of the present invention provides a multimodal edge-side model-driven intelligent application interaction system, comprising:

[0031] The first unit is used to acquire user input interaction instructions, determine the type of interaction task to be executed based on the interaction instructions, and select a corresponding lightweight deep learning model from a preset multimodal model library based on the interaction task type.

[0032] The second unit is used to extract and fuse features from the multimodal information in the interaction instructions using an attention mechanism, generate a unified feature vector, and input the unified feature vector into the lightweight deep learning model. The Monte Carlo tree search method is used to sample and optimize the model output to obtain the interaction response result with the highest confidence.

[0033] The third unit is used to construct a dynamic response strategy tree based on the interaction response results, set multiple candidate branch nodes in the dynamic response strategy tree, calculate the interaction utility score for each candidate branch node, and prune the candidate branch nodes through a bundle search algorithm to select the branch node with the highest interaction utility score as the optimal interaction path.

[0034] The fourth unit is used to generate corresponding interactive response instructions based on the optimal interactive path, and execute the interactive response instructions to complete the intelligent application interaction.

[0035] A third aspect of the present invention provides an electronic device, comprising:

[0036] processor;

[0037] Memory used to store processor-executable instructions;

[0038] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0040] The beneficial effects of this application are as follows:

[0041] This invention achieves efficient processing of intelligent application interactions by integrating a multimodal model library on the device side and dynamically selecting a lightweight deep learning model according to the type of interaction task. This significantly reduces the consumption of computing resources and improves the response speed and processing capabilities of the device side.

[0042] This invention employs an attention mechanism and a Monte Carlo tree search method to perform deep fusion processing and result optimization of multimodal information, significantly improving the accuracy and reliability of interactive responses and effectively solving the technical problems of inaccurate understanding and unreasonable responses in complex scenarios by traditional interactive methods.

[0043] This invention introduces a dynamic response strategy tree and a bundle search algorithm to construct an intelligent interactive decision-making mechanism. By calculating the interaction utility score, it automatically selects the optimal interaction path, realizing adaptive optimization and personalized customization of the interaction process. This enables the system to provide a more intelligent and smooth interactive experience based on user needs and context. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the intelligent application interaction method driven by a multimodal edge model according to an embodiment of the present invention.

[0045] Figure 2This is a flowchart of the depth-first search process based on candidate interactive responses in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0048] Figure 1 This is a flowchart illustrating the multimodal edge-side model-driven intelligent application interaction method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0049] Obtain user-inputted interaction commands, determine the type of interaction task to be executed based on the interaction commands, and select a corresponding lightweight deep learning model from a preset multimodal model library based on the interaction task type.

[0050] An attention mechanism is used to extract and fuse features from the multimodal information in the interaction command to generate a unified feature vector. The unified feature vector is then input into the lightweight deep learning model. The Monte Carlo tree search method is used to sample and optimize the model output to obtain the interaction response result with the highest confidence.

[0051] A dynamic response strategy tree is constructed based on the interaction response results. Multiple candidate branch nodes are set in the dynamic response strategy tree. An interaction utility score is calculated for each candidate branch node. The candidate branch nodes are pruned using a bundle search algorithm. The branch node with the highest interaction utility score is selected as the optimal interaction path.

[0052] Generate corresponding interactive response instructions based on the optimal interaction path, and execute the interactive response instructions to complete the intelligent application interaction.

[0053] In one optional implementation, determining the type of interactive task to be executed for the interactive instruction, and selecting a corresponding lightweight deep learning model from a preset multimodal model library based on the interactive task type includes:

[0054] Extract the feature vector from the interaction instruction, construct the interaction task feature matrix based on the feature vector, perform feature decomposition on the interaction task feature matrix to obtain the interaction task feature decomposition result, calculate the similarity between the interaction task feature decomposition result and the preset task type feature template, and determine the type of interaction task to be executed corresponding to the interaction instruction.

[0055] Based on the type of interactive task to be executed, a task processing requirement description is generated. Based on the task processing requirement description, a deep learning model scoring matrix is ​​established according to the performance parameters, resource consumption parameters, and computational complexity parameters of each deep learning model in the preset multimodal model library. By performing feature matching operations on the deep learning model scoring matrix, a lightweight deep learning model that best matches the task processing requirement description is selected.

[0056] In the process of extracting feature vectors from interactive commands, the input interactive commands are preprocessed, including word segmentation and stop word removal. Taking the voice command "Please help me translate this English text" as an example, after word segmentation, we can obtain words such as "please", "help", "I", "translate", "this text", and "English text". The system uses a pre-trained word embedding model, such as word2vec, to convert each word into a 300-dimensional feature vector. For example, the feature vector of "translate" is [0.23, 0.45, -0.12, ..., 0.67].

[0057] When constructing the feature matrix for the interactive task based on the extracted feature vectors, the feature vectors of all words in the instruction are concatenated to form a matrix. For the example above, assuming a total of 6 keywords are extracted and the feature vector dimension is 300, the constructed feature matrix will have a dimension of 6×300.

[0058] When performing eigenvalue decomposition on the feature matrix, singular value decomposition is used to decompose the feature matrix into the product of three matrices. Through eigenvalue decomposition, the core semantic information of the interactive commands can be extracted. The principal component vector obtained after eigenvalue decomposition is [0.87, 0.35, 0.12, 0.05, 0.02, 0.01], representing the importance weights of different semantic dimensions.

[0059] When calculating similarity between the feature decomposition results of the interactive task and the preset task type feature templates, a task type library is maintained, containing multiple task types such as "text translation," "image recognition," and "speech synthesis," along with their feature templates. Each task type feature template is also a vector representation obtained by feature extraction and decomposition of a large number of similar instruction samples. For example, the feature template for the "text translation" task is [0.89, 0.32, 0.15, 0.04, 0.01, 0.02]. The system calculates the cosine similarity between the feature decomposition results and each task feature template, and selects the task type with the highest similarity as the interactive task type to be executed. In the example, the similarity with the "text translation" task is 0.98, which is much higher than the similarity of 0.25 with the "image recognition" task; therefore, the interactive task type to be executed is determined to be "text translation."

[0060] After determining the type of interactive task to be performed, task processing requirement description information is generated based on the task type. This information includes attributes such as task type, input data size, response time requirement, and accuracy requirement. For the "text translation" task, the following requirement description is generated: the task type is text translation, the input data size is short text (less than 100 characters), the required response time is less than 1 second, and the translation accuracy requirement is medium.

[0061] Based on the task processing requirements, a suitable lightweight deep learning model is selected from a pre-defined multimodal model library. This library contains multiple models for various task types, such as Transformer-tiny, Transformer-small, and LSTM-based models for translation tasks. Each model records performance parameters (e.g., accuracy, recall), resource consumption parameters (e.g., memory usage, CPU / GPU utilization), and computational complexity parameters (e.g., number of FLOPs, number of parameters).

[0062] When constructing the deep learning model scoring matrix, the task processing requirements are matched with the parameters of each candidate model. For example, for the "text translation" task, three candidate models are selected: Transformer-tiny (85% accuracy, 100MB memory usage, 5M parameters), Transformer-small (92% accuracy, 300MB memory usage, 25M parameters), and LSTM-based (83% accuracy, 80MB memory usage, 3M parameters). The system scores each model based on task requirements across three dimensions: performance, resource consumption, and computational complexity, forming the scoring matrix.

[0063] By performing feature matching operations on the deep learning model scoring matrix and comprehensively considering various dimensions, a comprehensive score is calculated for each model. The calculation formula includes a weighted sum of the scores for each dimension, where the weights are dynamically adjusted according to task requirements. For the translation task example, since the response time is required to be less than 1 second and the accuracy requirement is moderate, the resource consumption weight is set to 0.5, the performance weight to 0.3, and the complexity weight to 0.2. The calculated comprehensive scores for the three models are: Transformer-tiny (0.82), Transformer-small (0.65), and LSTM-based (0.88). The system selects the LSTM-based model with the highest comprehensive score as the lightweight deep learning model for the final interactive task.

[0064] After model selection, the chosen model is loaded and initialized, ready to execute the interactive task. For translation tasks, the system receives text input by the user, performs translation processing using an LSTM-based model, and finally returns the translation result to the user, completing the entire interactive process. The entire processing flow, from receiving the interactive command to returning the result, can be controlled within 800 milliseconds on ordinary hardware devices, meeting the user's requirements for response speed.

[0065] The above implementation method, through steps such as feature extraction, task type identification, and model matching and selection, enables the automatic selection of a suitable lightweight deep learning model based on interactive commands, balancing model performance and resource consumption, and is suitable for intelligent interactive applications in resource-constrained scenarios.

[0066] In one optional implementation, the unified feature vector is input into the lightweight deep learning model, and the model output is sampled and optimized using the Monte Carlo tree search method to obtain the interactive response result with the highest confidence, including:

[0067] A unified feature vector is input into a lightweight deep learning model to obtain an initial interaction response result; an initial confidence score is calculated for the initial interaction response result, and a search root node is generated based on the initial confidence score;

[0068] The search root node is expanded using a depth-first traversal strategy to generate multiple candidate response branches. Response parameters are configured for each candidate response branch, and the initial interaction response result is transformed based on the response parameters to generate diverse candidate interaction response results.

[0069] The candidate interaction response results are constructed as child nodes, an initial access count value is assigned to each child node, and a sampling probability distribution is determined based on the initial access count value; multiple rounds of random sampling are performed on each child node according to the sampling probability distribution, feature extraction is performed on the sampled response results, and the confidence score of the sampled response results is calculated.

[0070] A value evaluation function is constructed based on the confidence score and the access count value. The optimal child node is selected by the calculation result of the value evaluation function. The access count value of the optimal child node is compared with a preset sampling threshold. When the access count value reaches the preset sampling threshold, the result with the highest confidence score is selected from all candidate interaction response results as the final output interaction response result.

[0071] A unified feature vector is input into a lightweight deep learning model, which employs a 6-layer Transformer encoder and a 4-layer decoder structure, with the number of parameters kept below 30 million. The initial interactive response result is output through a three-stage processing process: feature encoding, context fusion, and response generation.

[0072] The confidence score of the initial interaction response is calculated using a weighted combination of four dimensions: semantic consistency score, language fluency score, content relevance score, and user intent matching score. Semantic consistency is calculated using word vectors from a pre-trained language model for similarity; language fluency is assessed based on a 3-gram language model to evaluate naturalness; content relevance is extracted using a topic modeling algorithm to extract dialogue topic features; and user intent matching is determined by identifying and calculating the matching degree using an intent classifier. A search root node is generated based on the initial confidence score, containing attribute information such as the initial interaction response result, confidence score, access counter, and a list of child nodes, using a tree-like node object structure.

[0073] A depth-first search strategy is employed to expand the search root node, implemented using a recursive algorithm with a recursion depth limited to 8 levels to control search complexity. Multiple candidate response branches are generated at each node, with the number of branches dynamically adjusted based on the node's confidence score. A conditional random sampling method is used to generate candidate branches, with the sampling temperature parameter dynamically adjusted between 0.3 and 1.2 to control diversity and quality balance. Response parameters are configured for each candidate response branch, including four dimensions: language style, sentiment, detail, and interaction mode. Language style includes four types: formal, semi-formal, informal, and conversational; sentiment includes three categories: positive, neutral, and negative, with an intensity range of 0.1 to 1.0; detail ranges from 1 to 5 to control content richness; and interaction modes include four types: informative, guiding, supportive, and entertaining.

[0074] The initial interactive response is transformed based on response parameters using vocabulary replacement templates, sentence reconstruction templates, and content expansion templates. The vocabulary replacement template replaces original words with corresponding style synonyms based on language style parameters; the replacement dictionary contains 100,000 entries. The sentence reconstruction template adjusts sentence length and complexity based on detail parameters, including sentence merging, decomposition, and recombination operations. The content expansion template adds corresponding interactive elements based on interaction mode parameters, such as adding question-guided statements in guided modes. Diverse candidate interactive response results are generated, with semantic diversity, style diversity, and structural diversity control mechanisms ensuring differentiation; the semantic similarity threshold is set at 0.7.

[0075] The candidate interaction response results are constructed into child nodes, comprising three steps: node creation, attribute initialization, and relationship establishment. Node creation allocates memory and generates a unique identifier; attribute initialization stores the candidate response content and parameter information; and relationship establishment sets the pointer relationship with the parent node. Each child node is assigned an initial access count of 1 to record the number of times it is accessed and evaluated during the search process. Based on the initial access count, a sampling probability distribution is determined, calculated using a variant of the upper confidence bound algorithm, considering both the node confidence score and the access count. The exploration coefficient is set to 1.414 to balance the utilization of high-quality nodes and the exploration of unknown node relationships. Normalization ensures that the sum of the probabilities of all child nodes equals 1.

[0076] Each child node is randomly sampled multiple times based on the sampling probability distribution, with the number of sampling rounds set to 100. A Monte Carlo method is used, generating random numbers between 0 and 1 using a pseudo-random number generator, and selecting the selected node based on the cumulative probability distribution. Feature extraction is performed on the sampled response results, encompassing three levels: textual features, semantic features, and interaction features. Textual features include lexical richness, syntactic complexity, and fluency; semantic features include topic consistency, logical coherence, and information completeness; and interaction features include user-friendliness, response timeliness, and satisfaction prediction. A confidence score for the sampled response results is calculated through a weighted summation of multiple feature dimensions. The weight coefficients are determined based on machine learning training with 100,000 labeled dialogues, and the final score is adjusted considering contextual relevance.

[0077] A value assessment function is constructed based on confidence scores and access counts, comprising two parts: utilization and exploration. The utilization component assesses the expected value based on the node's historical performance, using a weighted average method to calculate the historical confidence score while considering a time decay factor. The exploration component assesses the exploration potential based on the node's access frequency, calculated using the reciprocal of the access count. The optimal child node is selected based on the value assessment function's results. A greedy algorithm directly selects the node with the highest value assessment score, while considering diversity constraints to avoid consecutively selecting the same type of node.

[0078] The optimal child node access count is compared with a preset sampling threshold of 50, which is determined based on computational resource constraints and response time requirements. When the access count reaches the preset sampling threshold, the search process terminates. The result with the highest confidence score is selected from all candidate interaction response results as the final output interaction response result. After language polishing, format adjustment, and quality inspection, post-processing optimization is performed to ensure that it meets the user's interaction needs.

[0079] In one optional implementation, a depth-first traversal strategy is used to expand the search root node, generating multiple candidate response branches. Response parameters are configured for each candidate response branch, and the initial interaction response result is transformed based on the response parameters to generate diverse candidate interaction response results, including:

[0080] Mark the root node of the search as the current processing node, construct a depth-first traversal stack based on the current processing node, and obtain the initial interaction response result corresponding to the current processing node based on the depth-first traversal stack;

[0081] A response branch generation function is constructed for the current processing node. The semantic feature vector of the initial interaction response result is calculated based on the response branch generation function. Based on the semantic feature vector, semantic similarity calculation, expression integrity evaluation and interaction coherence analysis are performed to generate multiple candidate response branches.

[0082] A set of response parameters is constructed for each candidate response branch, a response transformation matrix is ​​constructed based on the set of response parameters, and the initial interaction response result is linearly combined and transformed by the response transformation matrix to generate candidate interaction response results with different characteristics.

[0083] Each candidate interaction response result is constructed as a new search node, and the new search node is pushed onto the depth-first traversal stack; when the depth of the depth-first traversal stack reaches a preset traversal threshold, all generated candidate interaction response results are output.

[0084] like Figure 2 As shown, the method includes:

[0085] The root node is marked as the current processing node, containing initial interaction response information and relevant contextual features. A depth-first traversal stack is constructed based on the current processing node, using a last-in-first-out (LIFO) data structure. The stack's initial capacity is set to 1024 node elements, each containing a node identifier, node depth, parent node pointer, and node state information. The depth-first traversal stack is implemented using a dynamic array structure, automatically doubling its size when insufficient. The initial interaction response result corresponding to the current processing node is obtained based on the depth-first traversal stack. This result includes multi-dimensional information such as text content, semantic tags, sentiment, and interaction type.

[0086] A response branch generation function is constructed for the current processing node. This function performs deep analysis of the node content using a pre-trained language model. The response branch generation function comprises three core components: a semantic parsing module, a content expansion module, and a diversity control module. The semantic parsing module uses an attention-based encoder to semantically understand the initial interaction response result, extracting features such as keywords, syntactic structures, and semantic roles. The content expansion module generates various response variants based on the semantic parsing results, including synonym replacement, sentence reconstruction, and content supplementation. The diversity control module sets a diversity threshold to ensure that the generated response branches maintain sufficient diversity while preserving semantic consistency.

[0087] The semantic feature vector of the initial interactive response result is calculated based on the response branch generation function. This vector is transformed into a 512-dimensional dense vector representation of the text content using word embedding technology. The generation process of the semantic feature vector includes two stages: lexical-level feature extraction and sentence-level feature aggregation. Lexical-level feature extraction maps each word to a 128-dimensional vector representation using a pre-trained word vector model. Sentence-level feature aggregation merges the lexical-level vectors into a sentence-level semantic representation using a weighted average. The weighting coefficients are determined based on the importance scores of the words, which are calculated using a combination of the TF-IDF algorithm and positional weights.

[0088] Semantic similarity is calculated based on semantic feature vectors, and semantic relevance is evaluated by calculating the cosine similarity of semantic vectors between different candidate branches. The results of the semantic similarity calculation are used to control the generation strategy of candidate branches; branches with excessively high similarity are filtered to ensure response diversity. Expression integrity is evaluated by analyzing the syntactic structure and semantic integrity of candidate responses, with evaluation metrics including grammatical correctness of sentence structure, completeness of semantic roles, and coherence of logical relationships. Interaction coherence analysis is achieved by examining the consistency between candidate responses and context, with analysis dimensions including topic coherence, sentiment consistency, and matching degree of interaction style.

[0089] The process of generating multiple candidate response branches employs a sampling strategy that controls randomness, adjusting the innovativeness and stability of the generated results by setting a temperature parameter. The temperature parameter is set to 0.7, providing moderate randomness while ensuring response quality. The number of candidate response branches is dynamically adjusted based on the depth and search complexity of the current node, typically generating 3 to 8 candidate branches per node. Each candidate response branch undergoes screening by a quality assessment module, which includes three dimensions: language fluency assessment, content relevance assessment, and user intent matching assessment.

[0090] For each candidate response branch, a set of response parameters is constructed, including response strength, language style, sentiment, and interaction mode parameters. The response strength parameter controls the level of detail and intensity of the response content, ranging from 0.1 to 1.0; higher values ​​indicate a more detailed and intense response. The language style parameter defines the expression style of the response, including four style types: formal, informal, technical, and colloquial, each corresponding to different vocabulary choices and sentence structures. The sentiment parameter describes the emotional tone of the response, including three basic dimensions: positive, neutral, and negative, with each dimension ranging from 0 to 1 in intensity. The interaction mode parameter defines the interactive characteristics of the response, including four modes: information-providing, question-guided, suggestion-guiding, and emotionally supportive.

[0091] A response transformation matrix is ​​constructed based on the set of response parameters. This matrix describes the weight of the influence of different parameters on the response content. The response transformation matrix employs a sparse matrix structure to improve computational efficiency; rows correspond to different response feature dimensions, and columns correspond to different parameter types. The numerical values ​​of the matrix elements represent the degree of influence of the corresponding parameter on a specific feature dimension, which is determined through statistical analysis of historical interaction data. The construction process of the transformation matrix also considers the interaction relationships between parameters, using cross terms to describe the synergistic effect of parameter combinations.

[0092] The initial interactive response is linearly combined using a response transformation matrix. This transformation involves matrix multiplication of the initial response's feature vector with the transformation matrix to obtain the transformed feature representation. The linear combination transformation includes two steps: feature weight adjustment and feature fusion. Feature weight adjustment weights the importance of each feature dimension based on the response parameters. Feature fusion recombines the adjusted features to form a new response representation. The transformation process also includes constraint checks to ensure that the transformed result meets the requirements of semantic consistency and interaction rationality.

[0093] Candidate interactive responses with distinct features are generated, each possessing a unique expression and interactive characteristics. The generation of candidate interactive responses employs a diversity enhancement strategy, ensuring diversity by controlling the distribution of different response branches in the semantic space. Diversity evaluation is achieved by calculating the feature distance between candidate responses, which includes three dimensions: semantic distance, style distance, and structural distance. The candidate responses also undergo consistency checks to ensure that all results remain consistent with the original interactive intent.

[0094] Each candidate interaction response is used to construct a new search node. This new node contains information such as the response content, generation parameters, quality score, and hierarchical relationship. The construction process of a new search node includes three steps: node initialization, attribute assignment, and relationship establishment. Node initialization creates a node object and assigns a unique identifier. Attribute assignment stores the various feature information of the candidate response in the node attributes. Relationship establishment determines the hierarchical relationship between the new node and its parent nodes. The new search node is then pushed onto a depth-first traversal stack. The push operation includes stack capacity checking, node insertion, and stack state updating.

[0095] When the depth of the depth-first traversal stack reaches a preset traversal threshold, the traversal process terminates and all generated candidate interaction responses are output. The preset traversal threshold is determined based on computational resource constraints and response time requirements, and is typically set to a depth of 15 levels. The setting of the traversal threshold also considers the balance between search quality and efficiency; excessively deep traversals lead to excessively high computational complexity, while shallow traversals fail to discover sufficiently high-quality candidate results. The output candidate interaction responses are sorted according to their quality scores, providing an ordered candidate set for subsequent selection and optimization processes.

[0096] In one optional implementation, a dynamic response strategy tree is constructed based on the interaction response results. Multiple candidate branch nodes are set in the dynamic response strategy tree. An interaction utility score is calculated for each candidate branch node. A bundle search algorithm is used to prune the candidate branch nodes, and the branch node with the highest interaction utility score is selected as the optimal interaction path. This includes:

[0097] Multiple candidate branch nodes are set in the dynamic response strategy tree, and an interaction intent vector is calculated for each candidate branch node. The interaction intent vector is used to extract interaction intent features through a semantic encoding function.

[0098] An interaction state transition probability matrix is ​​constructed based on the interaction intent features. An initial interaction utility score is calculated for each candidate branch node based on the interaction state transition probability matrix. The candidate branch nodes are arranged in descending order according to the initial interaction utility scores to generate a candidate node sequence. A bundle search optimization function is constructed based on the candidate node sequence.

[0099] Based on the bundle search optimization function, the historical best branch information is continuously updated by maintaining a search bundle of fixed width. A dynamic pruning threshold is calculated based on the ratio of the historical best branch information to the current interaction utility score. The candidate node sequence is iteratively pruned based on the dynamic pruning threshold by the bundle search optimization function, and the best branch information in the search bundle is updated in each iteration.

[0100] After the beam search optimization function completes a preset number of iterations, it extracts the branch node with the highest interaction utility score from the search beam and determines the path corresponding to the branch node as the optimal interaction path.

[0101] In constructing the dynamic response strategy tree, a tree-like data structure is used to organize multiple candidate branch nodes, each representing a potential interaction response scheme. For each candidate branch node, its interaction intent features are extracted using a semantic encoding function to form an interaction intent vector. This semantic encoding function employs an encoder based on the Transformer architecture to convert the interaction content corresponding to the node into a high-dimensional semantic representation vector. Specifically, the textual, visual, and audio features of the interaction content are encoded separately using a multilayer perceptron network, and then fused using an attention weighting mechanism to generate a 512-dimensional interaction intent vector.

[0102] Based on the obtained interaction intent features, an interaction state transition probability matrix is ​​constructed, which describes the probability distribution of transitioning from the current interaction state to the next interaction state. The rows and columns of the matrix correspond to different interaction states, and the matrix elements represent the transition probabilities between states. The probability weights of state transitions are determined by calculating the cosine similarity between the interaction intent vector of the current node and various interaction patterns in the historical interaction pattern library. The historical interaction pattern library contains typical user interaction behavior patterns in different scenarios, with each pattern corresponding to a specific state transition path. Based on the interaction state transition probability matrix, an initial interaction utility score is calculated for each candidate branch node, reflecting the potential value of selecting that node as the next interaction response.

[0103] All candidate branch nodes are sorted in descending order of their initial interaction utility scores to form a candidate node sequence. This sequence provides the input basis for the subsequent beam search algorithm. A beam search optimization function is constructed based on the candidate node sequence, defining the evaluation criteria and optimization objective during the search process. The beam search optimization function comprises three core components: an evaluation function, a selection function, and an update function. The evaluation function calculates the overall utility score of the nodes, the selection function determines the nodes to be retained in the search beam, and the update function maintains the optimal branch information during the search process.

[0104] The bundle search optimization function controls the complexity and efficiency of the search by maintaining a fixed-width bundle. The bundle width is set to 16, meaning that a maximum of 16 optimal branch nodes are retained in each iteration. The bundle is continuously updated with historical optimal branch information, including the cumulative utility score, path length, and complete path sequence to each retained node. This historical optimal branch information is maintained using a sliding window mechanism with a window size of 32, ensuring the algorithm can memorize a sufficiently long amount of historical information for decision-making.

[0105] The dynamic pruning threshold is calculated based on the ratio of historical best branch information to the current interaction utility score. A dynamic adjustment factor is obtained by calculating the ratio of the utility score of the current best node in the search bundle to the historical average best score. When this ratio is greater than 1.2, it indicates that the current search direction is better than the historical average, and the pruning threshold is lowered to retain more potentially high-quality nodes. When the ratio is less than 0.8, it indicates that the current search effect is poor, and the pruning threshold is increased to accelerate convergence. The calculation of the dynamic pruning threshold also considers the number of search iterations, gradually increasing the strictness of pruning as iterations progress.

[0106] Based on the bundle search optimization function, the candidate node sequence is iteratively pruned according to a dynamic pruning threshold. In each iteration, each node in the search bundle is expanded, generating all its child nodes. For each newly generated child node, its interaction utility score is calculated, which comprehensively considers the node's local utility and global path utility. Local utility reflects the interaction quality of the node itself, while global path utility reflects the cumulative utility of the entire path from the root node to the node. The newly generated child node is compared with the existing nodes in the search bundle; the node with the highest utility score is retained, while nodes with scores below the dynamic pruning threshold are eliminated.

[0107] During each iteration, the optimal branch information in the search bundle is continuously updated. This update process includes recalculating the cumulative utility score of each retained node, updating the path information of the nodes, and adjusting the node's ranking position within the search bundle. The cumulative utility score is calculated using a weighted summation, with the weighting coefficients dynamically adjusted based on the path length and node depth. Updating the path information includes recording the complete path sequence from the root node to the current node, and the selection criteria for each decision point along that path.

[0108] The iteration process terminates after the search function completes a preset number of iterations. The preset number of iterations is set based on the problem's complexity and real-time requirements, typically 50 iterations. After iteration terminates, the branch node with the highest interaction utility score is extracted from the search bundle. This node represents the optimal interaction response solution found in the current search space. The complete path corresponding to this branch node is determined as the optimal interaction path, which contains the complete interaction sequence from the initial state to the target state. The optimal interaction path not only includes the specific choice at each decision point but also the confidence assessment of the choice and information on alternative solutions, providing sufficient decision support for subsequent interaction execution.

[0109] In one optional implementation, the candidate branch nodes are sorted in descending order according to the initial interaction utility scores to generate a candidate node sequence, and the bundle search optimization function is constructed based on the candidate node sequence, including:

[0110] Record the initial interaction utility score difference between adjacent nodes in the candidate node sequence, calculate the node weight coefficient based on the initial interaction utility score difference, group the candidate node sequence according to the node weight coefficient, divide the candidate node sequence into multiple subsequences, and construct a bundle search optimization function based on the subsequences.

[0111] Sort the candidate branch nodes in descending order of their initial interaction utility scores by comparing their initial interaction utility scores and ranking them from highest to lowest. For example, suppose there are five candidate branch nodes A, B, C, D, and E in the system, with initial interaction utility scores of 0.95, 0.82, 0.76, 0.65, and 0.48, respectively. The system will sort these nodes from highest to lowest score, forming a candidate node sequence [A, B, C, D, E]. The purpose of this is to prioritize nodes with higher utility scores, thereby improving the efficiency of subsequent searches.

[0112] The initial interaction utility score difference between adjacent nodes in the candidate node sequence refers to the difference in scores between adjacent nodes after sorting. In the example above, the differences between adjacent nodes are: 0.13 (0.95-0.82) between A and B, 0.06 (0.82-0.76) between B and C, 0.11 (0.76-0.65) between C and D, and 0.17 (0.65-0.48) between D and E. These differences reflect the degree of abrupt changes in utility between adjacent nodes; larger differences indicate more significant changes in node utility.

[0113] In calculating the node weight coefficients based on the initial interaction utility score differences, an adaptive weight allocation strategy is employed. Specifically, for each node, its weight coefficient is proportional to the difference between it and the next node. The larger the difference, the higher the weight coefficient, indicating a significant utility gap between the node and subsequent nodes, which should be considered a potential grouping boundary. In the example, node A has a weight coefficient of 0.28 (0.13 / 0.47, where 0.47 is the sum of all differences), node B has a weight coefficient of 0.13 (0.06 / 0.47), node C has a weight coefficient of 0.23 (0.11 / 0.47), and node D has a weight coefficient of 0.36 (0.17 / 0.47).

[0114] When grouping the candidate node sequence according to the node weight coefficients, a weight threshold is set. When the weight coefficient of a node exceeds this threshold, it is considered the dividing point of the subsequence. Assuming the weight threshold is set to 0.25, nodes with weight coefficients exceeding 0.25 are A and D. Therefore, the candidate node sequence can be divided into three subsequences: [A], [B, C], and [D, E]. This grouping method can divide nodes based on the natural breakpoints of their utility, ensuring that nodes within the same group have similar utility characteristics.

[0115] After dividing the candidate node sequence into multiple subsequences, different search resources are allocated to each subsequence. During the beam search, the system prioritizes subsequences with higher utility and performs a more refined search within each subsequence. In the example above, the first subsequence [A] contains the nodes with the highest utility, so the system allocates 50% of the computing resources; the second subsequence [B, C] has nodes with medium utility, so it allocates 30% of the resources; and the third subsequence [D, E] has nodes with low utility, so it allocates 20% of the resources.

[0116] When constructing the bundle search optimization function based on the subsequences, the characteristics and importance of the subsequences are considered. Specifically, for each subsequence, the system determines the corresponding bundle width and search depth based on the average utility of its internal nodes and the number of nodes. Subsequences with higher utility are assigned larger bundle widths, allowing the system to explore these regions more extensively. Simultaneously, the system also considers the balance between subsequences to prevent the search process from becoming overly focused on a few highly efficient nodes, ignoring potentially valuable branches.

[0117] In practical implementation, dynamic programming is used to optimize the bundle search process. For the first subsequence [A], since it has the highest utility score, the system sets a larger bundle width, such as 5; for the second subsequence [B, C], the system sets a medium bundle width, such as 3; and for the third subsequence [D, E], the system sets a smaller bundle width, such as 1. In this way, when performing the bundle search, the system will explore the high-utility region more thoroughly, while not completely ignoring the potential value of the low-utility region.

[0118] This adaptive grouping method based on utility differences allows for a more rational allocation of computational resources, improving the efficiency and effectiveness of beam search. For scenarios with a relatively uniform utility distribution, the system generates fewer subsequences; conversely, for scenarios with significant discontinuities in utility distribution, the system generates more subsequences. This flexible grouping strategy adapts to different application scenarios and data characteristics, enhancing the system's versatility and robustness.

[0119] During the beam search, the optimal branch node is selected from each subsequence for expansion based on the constructed optimization function. The optimization function considers the node's initial utility score, the importance of its subsequence, and its current search state to comprehensively evaluate the node's priority. In this way, the system can find a near-global optimal solution with limited computational resources.

[0120] A second aspect of the present invention provides a multimodal edge-side model-driven intelligent application interaction system, comprising:

[0121] The first unit is used to acquire user input interaction instructions, determine the type of interaction task to be executed based on the interaction instructions, and select a corresponding lightweight deep learning model from a preset multimodal model library based on the interaction task type.

[0122] The second unit is used to extract and fuse features from the multimodal information in the interaction instructions using an attention mechanism, generate a unified feature vector, and input the unified feature vector into the lightweight deep learning model. The Monte Carlo tree search method is used to sample and optimize the model output to obtain the interaction response result with the highest confidence.

[0123] The third unit is used to construct a dynamic response strategy tree based on the interaction response results, set multiple candidate branch nodes in the dynamic response strategy tree, calculate the interaction utility score for each candidate branch node, and prune the candidate branch nodes through a bundle search algorithm to select the branch node with the highest interaction utility score as the optimal interaction path.

[0124] The fourth unit is used to generate corresponding interactive response instructions based on the optimal interactive path, and execute the interactive response instructions to complete the intelligent application interaction.

[0125] A third aspect of the present invention provides an electronic device, comprising:

[0126] processor;

[0127] Memory used to store processor-executable instructions;

[0128] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0129] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0130] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal edge-side model-driven intelligent application interaction method, characterized in that, include: Obtain user-inputted interaction commands, determine the type of interaction task to be executed based on the interaction commands, and select a corresponding lightweight deep learning model from a preset multimodal model library based on the interaction task type. An attention mechanism is used to extract and fuse features from the multimodal information in the interactive commands, generating a unified feature vector. This unified feature vector is then input into the lightweight deep learning model. The Monte Carlo tree search method is used to sample and optimize the model output to obtain the interactive response result with the highest confidence, including: A unified feature vector is input into a lightweight deep learning model to obtain an initial interaction response result; an initial confidence score is calculated for the initial interaction response result, and a search root node is generated based on the initial confidence score; The search root node is expanded using a depth-first traversal strategy to generate multiple candidate response branches. Response parameters are configured for each candidate response branch, and the initial interaction response result is transformed based on the response parameters to generate diverse candidate interaction response results. The candidate interaction response results are constructed as child nodes, an initial access count value is assigned to each child node, and a sampling probability distribution is determined based on the initial access count value; multiple rounds of random sampling are performed on each child node according to the sampling probability distribution, feature extraction is performed on the sampled response results, and the confidence score of the sampled response results is calculated. A value evaluation function is constructed based on the confidence score and the access count value. The optimal child node is selected by the calculation result of the value evaluation function. The access count value of the optimal child node is compared with a preset sampling threshold. When the access count value reaches the preset sampling threshold, the result with the highest confidence score is selected from all candidate interaction response results as the final output interaction response result. A dynamic response strategy tree is constructed based on the interaction response results. Multiple candidate branch nodes are set in the dynamic response strategy tree. An interaction utility score is calculated for each candidate branch node. A bundle search algorithm is used to prune the candidate branch nodes, and the branch node with the highest interaction utility score is selected as the optimal interaction path. This includes: Multiple candidate branch nodes are set in the dynamic response strategy tree, and an interaction intent vector is calculated for each candidate branch node. The interaction intent vector is used to extract interaction intent features through a semantic encoding function. An interaction state transition probability matrix is ​​constructed based on the interaction intent features. An initial interaction utility score is calculated for each candidate branch node based on the interaction state transition probability matrix. The candidate branch nodes are arranged in descending order according to the initial interaction utility scores to generate a candidate node sequence. A bundle search optimization function is constructed based on the candidate node sequence. Based on the bundle search optimization function, the historical best branch information is continuously updated by maintaining a search bundle of fixed width. A dynamic pruning threshold is calculated based on the ratio of the historical best branch information to the current interaction utility score. The candidate node sequence is iteratively pruned based on the dynamic pruning threshold by the bundle search optimization function, and the best branch information in the search bundle is updated in each iteration. After the beam search optimization function completes a preset number of iterations, the branch node with the highest interaction utility score is extracted from the search beam, and the path corresponding to the branch node is determined as the optimal interaction path. Generate corresponding interactive response instructions based on the optimal interaction path, and execute the interactive response instructions to complete the intelligent application interaction.

2. The method according to claim 1, characterized in that, Determining the type of interactive task to be executed based on the interactive instruction, and selecting a corresponding lightweight deep learning model from a preset multimodal model library based on the interactive task type, includes: Extract the feature vector from the interaction instruction, construct the interaction task feature matrix based on the feature vector, perform feature decomposition on the interaction task feature matrix to obtain the interaction task feature decomposition result, calculate the similarity between the interaction task feature decomposition result and the preset task type feature template, and determine the type of interaction task to be executed corresponding to the interaction instruction. Based on the type of interactive task to be executed, a task processing requirement description is generated. Based on the task processing requirement description, a deep learning model scoring matrix is ​​established according to the performance parameters, resource consumption parameters, and computational complexity parameters of each deep learning model in the preset multimodal model library. By performing feature matching operations on the deep learning model scoring matrix, a lightweight deep learning model that best matches the task processing requirement description is selected.

3. The method according to claim 1, characterized in that, A depth-first traversal strategy is used to expand the search root node, generating multiple candidate response branches. Response parameters are configured for each candidate response branch, and the initial interaction response result is transformed based on the response parameters to generate diverse candidate interaction response results, including: Mark the root node of the search as the current processing node, construct a depth-first traversal stack based on the current processing node, and obtain the initial interaction response result corresponding to the current processing node based on the depth-first traversal stack; A response branch generation function is constructed for the current processing node. The semantic feature vector of the initial interaction response result is calculated based on the response branch generation function. Based on the semantic feature vector, semantic similarity calculation, expression integrity evaluation and interaction coherence analysis are performed to generate multiple candidate response branches. A set of response parameters is constructed for each candidate response branch, a response transformation matrix is ​​constructed based on the set of response parameters, and the initial interaction response result is linearly combined and transformed by the response transformation matrix to generate candidate interaction response results with different characteristics. Each candidate interaction response result is constructed as a new search node, and the new search node is pushed onto the depth-first traversal stack; when the depth of the depth-first traversal stack reaches a preset traversal threshold, all generated candidate interaction response results are output.

4. The method according to claim 1, characterized in that, Arrange the candidate branch nodes in descending order according to the initial interaction utility score to generate a candidate node sequence. Construct a bundle search optimization function based on the candidate node sequence, including: Record the initial interaction utility score difference between adjacent nodes in the candidate node sequence, calculate the node weight coefficient based on the initial interaction utility score difference, group the candidate node sequence according to the node weight coefficient, divide the candidate node sequence into multiple subsequences, and construct a bundle search optimization function based on the subsequences.

5. A multimodal edge-side model-driven intelligent application interaction system, used to implement the method of any one of claims 1-4, characterized in that, include: The first unit is used to acquire user input interaction instructions, determine the type of interaction task to be executed based on the interaction instructions, and select a corresponding lightweight deep learning model from a preset multimodal model library based on the interaction task type. The second unit is used to extract and fuse features from the multimodal information in the interaction instructions using an attention mechanism, generate a unified feature vector, and input the unified feature vector into the lightweight deep learning model. The Monte Carlo tree search method is used to sample and optimize the model output to obtain the interaction response result with the highest confidence. The third unit is used to construct a dynamic response strategy tree based on the interaction response results, set multiple candidate branch nodes in the dynamic response strategy tree, calculate the interaction utility score for each candidate branch node, and prune the candidate branch nodes through a bundle search algorithm to select the branch node with the highest interaction utility score as the optimal interaction path. The fourth unit is used to generate corresponding interactive response instructions based on the optimal interactive path, and execute the interactive response instructions to complete the intelligent application interaction.

6. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 4.

Images