Intelligent scene adaptive interaction method and system based on multi-modal large model

Abstract

The application discloses a wisdom scene self-adaptive interaction method and system based on a multi-modal large model, relates to the technical fields of artificial intelligence, natural language processing and the Internet of Things, and comprises the following steps: constructing a multi-modal entity understanding pre-training data set; encoding multi-modal data in the pre-training data set according to a pre-constructed multi-modal entity encoder to generate an entity input sequence with controllable length; performing multi-target joint optimization pre-training on a pre-constructed double-tower multi-modal large model according to the pre-training data set to obtain a general pre-training model; performing lightweight scene adaptation on the general pre-training model for a subdivided wisdom scene to obtain a scene adaptation model; inputting the entity input sequence and a current natural language query into the scene adaptation model to generate a thought chain reasoning conclusion; and generating an interaction instruction according to the thought chain reasoning conclusion, so that the wisdom scene self-adaptive interaction with high accuracy, low delay, low cost and feasibility is realized.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of artificial intelligence, natural language processing and Internet of Things, and in particular to a smart scene adaptive interaction method and system based on a multimodal large model. Background Technology

[0002] With the rapid development of smart city and IoT technologies, various physical scenarios such as smart transportation, smart communities, smart campuses, and smart government have generated massive amounts of multimodal entity interaction data. How to achieve accurate understanding of entity behavior and intelligent diagnosis of needs / risks based on this data, and then complete adaptive scenario interaction, has become a core pain point for the implementation of smart city applications.

[0003] In existing technologies, the technical solution represented by query anchoring user representation model has achieved good application results in online scenarios such as Internet user behavior analysis and personalized recommendation. Its core is to use natural language query as anchor point to align user behavior data with semantic space and complete the modeling of user interests. However, these solutions have significant technical limitations and cannot be directly adapted to the physical scenarios of smart city IoT: First, existing solutions are only designed for online behavioral data such as internet user clicks and browsing, focusing on solving the problem of user interest prediction. They cannot handle diagnostic and classification tasks such as risk identification and demand determination in smart scenarios, resulting in logical flaws of reversed causality. Furthermore, they do not consider the characteristics of high noise and sparse intent in physical world data, and lack an explicit causal reasoning link from behavioral results to potential needs. Second, existing solutions do not have dedicated processing mechanisms designed for the massive, high-frequency spatiotemporal trajectories, IoT sensor time series, video surveillance, and other long-sequence heterogeneous data in smart scenarios, posing engineering feasibility risks such as sequence length explosion and exceeding the context window of a large model. Third, existing solutions only stay at the entity representation generation stage and do not design dedicated architectures for the interactive needs of device linkage, real-time control, and service scheduling in smart scenarios, failing to achieve a complete interactive closed loop from representation to physical world actions. Fourth, scenario adaptation of existing solutions requires full-scale model fine-tuning, resulting in high computing costs and long adaptation cycles, making it impossible to quickly deploy in various segmented business scenarios of smart cities.

[0004] Furthermore, existing general text embedding models are only pre-trained on general text corpora and are not customized for physical multimodal data in smart scenarios. They are difficult to extract deep correlations from noisy physical behavior data and perform poorly in entity understanding tasks in smart scenarios, failing to meet the accuracy requirements for industrial-grade deployment. Summary of the Invention

[0005] To address the aforementioned shortcomings of existing technologies, this invention provides a smart scene adaptive interaction method and system based on a multimodal large model. By constructing a multimodal pre-trained dataset adapted to physical scenes, designing a modality-specific hierarchical encoder and sequence compression mechanism, building a query-anchored dual-tower model with an explicit inference module, designing a lightweight scene adaptation mechanism, and establishing a low-latency inference and closed-loop interaction architecture, this invention systematically solves the technical problems of static entity representation, lack of multimodal heterogeneous data processing capabilities, broken causal inference links, lack of a complete physical world interaction closed loop, high scene adaptation costs, and large deployment delays in existing technologies. This achieves high accuracy, low latency, low cost, and practical adaptive interaction for smart scenes.

[0006] In a first aspect, embodiments of the present invention provide a smart scene adaptive interaction method based on a multimodal large model, comprising: Construct a multimodal entity understanding pre-trained dataset covering multiple smart physics scenarios; The multimodal data in the pre-trained dataset is encoded using a pre-built hierarchical multimodal entity encoder to generate entity input sequences of controllable length. Based on the pre-trained dataset, a multi-objective joint optimization pre-training is performed on the pre-built query-anchored dual-tower multimodal large model to obtain a general pre-trained model; For specific smart scenarios, the general pre-trained model is subjected to lightweight scenario adaptation to obtain a scenario-adapted model. The entity input sequence and the current natural language query are input into the scenario adaptation model to generate a thought chain reasoning conclusion. Based on the reasoning conclusions of the thought chain, standardized interactive instructions are generated and sent to physical devices or business systems for execution, forming an interactive closed loop.

[0007] In a preferred embodiment, the construction of a multimodal entity understanding pre-trained dataset covering multiple smart physics scenarios includes: Collect end-to-end multimodal interaction data of various interactive entities within a set historical time window to construct entity profiles; A future behavior summary is generated based on the entity file, and the future behavior summary is concatenated with a general template query to form a scenario entity time-series behavior prediction dataset. A topic seed pool is constructed through unsupervised clustering and generative models. Natural language query and structured answer samples are generated based on the entity archives. A multimodal scenario entity question answering dataset is constructed based on the natural language query and structured answer samples. Quantifiable physical world truth values ​​are automatically extracted from the entity files, and a physical world truth value dataset is constructed based on the physical world truth values, the entity files, and the natural language query. The pre-training dataset is composed of the scene entity temporal behavior prediction dataset, the multimodal scene entity question answering dataset, and the physical world truth dataset.

[0008] In a preferred embodiment, natural language query and structured answer samples are generated based on the entity files, and a multimodal scenario entity question-answering dataset is constructed based on the natural language query and structured answer samples, including: For each entity file, topics related to that entity file are retrieved from the topic seed pool, and each topic is instantiated into a natural language query based on the real multimodal data of the entity file. The entity files and natural language queries are input into the large language model to generate structured answer samples corresponding to the natural language queries, thus obtaining natural language query-structured answer sample pairs. The natural language query-structured answer sample pairs are validated in two dimensions, and the sample pairs that pass the validation are retained. The sequence length of the validated sample pairs is controlled to ultimately form a multimodal scenario entity question-answering dataset.

[0009] In a preferred embodiment, multimodal data in the pre-trained dataset is encoded using a pre-constructed hierarchical multimodal entity encoder to generate entity input sequences of controllable length, including: For different data modalities, a pre-built hierarchical multimodal entity encoder is used to extract the initial embedding, and the event-level embedding is output through a multimodal-specific adapter; Aggregate all event-level embeddings within the same modality to generate a unified modality-level embedding; All modalities are uniformly embedded and fused at the modal level to generate a global entity-level embedding; By using a fixed token limit and a self-attention mechanism, events within each modality are weighted and sorted, and core event-level embeddings are retained according to a preset limit. The modal-level unified embedding, global entity-level embedding, and core event-level embedding are structurally concatenated, and the length of the concatenated sequence is controlled to generate the entity input sequence.

[0010] In a preferred embodiment, the step of weighting events within each modality using a fixed token cap and a self-attention mechanism, and retaining core event-level embeddings according to a preset cap, includes: Within each modality, the attention weights of each event-level embedding are calculated using a self-attention layer; The event-level embeddings are sorted in descending order based on the attention weight of each event-level embedding; Based on the fixed token limit preset for each modality, a corresponding number of event-level embeddings are reserved as core event-level embeddings.

[0011] In a preferred embodiment, the modal-level unified embedding, global entity-level embedding, and core event-level embedding are structurally concatenated, and the length of the concatenated sequence is controlled to generate the entity input sequence, including: According to the preset splicing order, the modal-level unified embedding, global entity-level embedding, and core event-level embedding are structurally spliced ​​together to generate the initial entity input sequence; The initial entity input sequence is length-checked to determine whether its total length exceeds the preset proportion threshold of the multimodal large model context window; If the preset ratio threshold is exceeded, the number of core events retained for each modality is dynamically adjusted according to the scenario business priority, and the initial entity input sequence is compressed to obtain the final entity input sequence.

[0012] In a preferred embodiment, a pre-constructed query-anchored dual-tower multimodal large model is pre-trained using the pre-trained dataset through multi-objective joint optimization to obtain a general pre-trained model, including: Construct a query-anchored dual-tower multimodal large model, wherein the dual-tower model includes an anchoring tower and a semantic tower with shared weights; The entity input sequence is concatenated with a natural language query and then input into the anchoring tower to generate a scene-adaptive entity representation; the physical world truth values ​​in the pre-trained dataset are input into the semantic tower and mapped into semantic vectors for representation alignment. The dual-tower model is pre-trained by comparing the alignment loss, autoregressive generation loss, physical consistency loss, and classification loss to form a multi-objective joint optimization.

[0013] As a preferred implementation, for specific smart scenarios, the general pre-trained model is subjected to lightweight scenario adaptation to obtain a scenario-adapted model, including: Freeze the backbone parameters of the general pre-trained model; Cluster the samples of subdivided business tasks to construct a learnable category prototype center; For each sub-task, a set of learnable soft cue tokens is designed to modulate the latent space of entity representations in a contextual manner. Design a learnable modal gating weight matrix and jointly optimize it with the soft cue token to perform scene-adaptive attention redistribution on the representation of each modality; The prototype center, soft cue token, and gating weights of the category are jointly optimized by using prototype contrast loss to output the scene adaptation model.

[0014] As a preferred implementation, standardized interactive instructions are generated based on the reasoning conclusion of the thought chain and sent to physical devices or business systems for execution, forming an interactive closed loop, including: Based on the final conclusion in the reasoning chain, a preset instruction template is matched to generate device control, service scheduling, or content push instructions, and the generated instructions are verified for permissions, security, and feasibility. The verified instructions are routed to the corresponding IoT device gateway, business service system, or content push channel for execution, and the execution results are received and sent back.

[0015] Secondly, embodiments of the present invention also provide a smart scene adaptive interaction system based on a multimodal large model, comprising: The multimodal data acquisition module is used to build a pre-trained dataset for multimodal entity understanding covering multiple smart physics scenarios; The multimodal data preprocessing module is used to encode the multimodal data in the pre-trained dataset according to the pre-built hierarchical multimodal entity encoder to generate entity input sequences with controllable length. The dual-tower multimodal large model pre-training module is used to perform multi-objective joint optimization pre-training on the pre-built query-anchored dual-tower multimodal large model based on the pre-training dataset to obtain a general pre-trained model; The scene adaptive soft prompting optimization module is used to perform lightweight scene adaptation on the general pre-trained model for specific smart scenes, so as to obtain a scene-adapted model. The online reasoning module is used to input the entity input sequence and the current natural language query into the scenario adaptation model to generate a thought chain reasoning conclusion; The scene interaction instruction generation and execution module is used to generate standardized interaction instructions based on the reasoning conclusion of the thought chain, and send them to physical devices or business systems for execution to form an interaction closed loop.

[0016] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the intelligent scene adaptive interaction method based on a multimodal large model as described in any embodiment of the present invention.

[0017] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent scene adaptive interaction method based on a multimodal large model as described in any embodiment of the present invention.

[0018] Compared with existing technologies, the present invention achieves the following beneficial effects: (1) This invention solves the problem of causal reasoning discontinuity in physical entity interaction, significantly improving the accuracy of reasoning diagnosis: The present invention adds a CoT explicit reasoning module to the query anchoring framework, and combined with the physical world consistency loss, it connects the entire link of physical behavior data - structured representation - causal reasoning diagnosis - scene interaction actions, which can simultaneously adapt to two core tasks: intent prediction and diagnostic classification, solving the pain points of high noise and sparse intent in physical world data. Experimental test results show that the present invention achieves an average area under the curve (AUC) of 0.831 in 12 smart scene benchmark tasks, which is 10.2% higher than the existing general representation model, and the accuracy of risk diagnosis tasks is improved by 11.7%.

[0019] (2) It solves the engineering feasibility problem of processing long sequence heterogeneous data and the sequence length is strictly controllable: This invention designs a modality-specific core event filtering and sequence compression algorithm for long sequence heterogeneous data such as spatiotemporal trajectories, IoT time series, and video frames. Through a fixed token upper limit and self-attention weight adaptive filtering mechanism, it strictly controls the total input sequence length to not exceed 70% of the multimodal large model context window, which completely solves the technical problems of sequence length explosion and exceeding the large model context window. It has strong engineering feasibility.

[0020] (3) It achieves extremely low scene adaptation cost and can quickly implement segmented business: Based on clustering soft prompts and gated modal attention mechanism, this invention only needs to optimize a small number of learnable parameters to complete the adaptation of new scenes, without the need for full fine-tuning of large models. The scene adaptation cycle is shortened from the monthly level to the hourly level, and the computing power cost is reduced by more than 92%. At the same time, the scene gating mechanism can allow modal attention to automatically match scene requirements, further improving the model performance of segmented scenes.

[0021] (4) It has industrial-grade high-concurrency and low-latency deployment capabilities: Through the incremental inference mechanism of KV-cache prefix sharing, this invention realizes one-time entity encoding and multi-scenario reuse. The single-scenario incremental inference latency is reduced to less than 5ms. New scenarios do not require expansion of the full GPU cluster. It can support real-time interaction of multiple scenarios for millions of entities and fully meet the industrial-grade deployment requirements of smart city scenarios. Attached Figure Description

[0022] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1This is a flowchart of a smart scene adaptive interaction method based on a multimodal large model provided in an embodiment of the present invention; Figure 2 This is a flowchart of the multimodal data hierarchical compression coding process provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of query anchoring and multi-objective joint optimization provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of KV-cache incremental inference and interactive closed loop provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a smart scene adaptive interaction system based on a multimodal large model provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0024] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as being processed sequentially, many of these operations (or steps) may be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations may be rearranged. The process may be terminated when its operation is completed, but may also have additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0025] Example 1 like Figure 1 The diagram shows a flowchart of a smart scene adaptive interaction method 100 based on a multimodal large model provided in Embodiment 1 of the present invention. The method 100 specifically includes the following steps: S110: Construct a multimodal entity understanding pre-trained dataset covering multiple smart physics scenarios.

[0026] As a preferred embodiment, the multimodal entity understanding pre-training dataset SceneU constructed by this invention covers five core smart physical scenarios: smart transportation, smart community, smart campus, smart government affairs, and smart tour guide, with a total scale of no less than 100 million pre-training sample pairs, including a scene entity temporal behavior prediction dataset. Multimodal Entity Question Answering Dataset Physical World Truth Dataset The three core subsets provide multi-dimensional supervision signals for model pre-training, including behavioral temporal patterns, causal reasoning logic, and physical world constraints. The entire dataset construction process is automated, with clearly defined quantitative verification rules. The construction of each core subset includes the following steps: Step S111: Standardized entity archive construction and basic data preprocessing (1) Collect full-link multimodal interaction data of various interactive entities in smart scenarios (including natural persons, vehicles, smart IoT devices, business entities, physical places, etc.) over the past 90 days. The data types cover six categories: text interaction, structured attributes, spatiotemporal trajectory, IoT sensor time series, video frames, and audio. The specific data dimensions for different scenarios are as follows: Smart transportation scenarios: vehicle trajectory, checkpoint capture, travel payment, public transport card swiping, navigation interaction, and road condition sensor data; Smart community scenarios: access control records, key frames of surveillance videos, property maintenance records, time-series data from environmental sensors, and community service interaction data; Smart campus scenarios include: attendance records, transaction history, library borrowing data, teaching system interaction records, campus security data, and activity participation data. Smart government service scenarios include: application records, consultation and Q&A texts, material submission records, application progress query data, policy browsing history, and legal entity business data. Smart tour guide scenario: scenic area tour route, audio guide interaction records, ticket purchase data, surrounding consumption flow, travel guide browsing text, scenic area environmental data.

[0027] (2) Perform full-process preprocessing on the collected multimodal interaction data, including data cleaning, spatiotemporal alignment, data anonymization, and format standardization, and finally construct a unified structured entity archive. ,in It serves as a unique identifier for the entity.

[0028] Step S112 Scene Entity Temporal Behavior Prediction Dataset Build (1) For each entity file Using a 15-day time window, the physical interaction behavior over the following 30 days is aggregated. Core behaviors are selected based on three dimensions: behavior frequency, spatial relevance, and business impact, generating a standardized summary of future behaviors. ; (2) Use general templates for smart scenarios for querying The template content is: "What are the most likely physical interaction behaviors of this entity in the near future?" (3) Transfer the physical archives With template query Perform context splicing, the splicing operation is denoted as Summary of future behavior The training sample pairs are constructed to ultimately form a dataset for predicting the temporal behavior of scene entities. The expression is: .

[0029] Step S113 Multimodal Scene Entity Question Answering Dataset Build This dataset is used to achieve deep alignment between multimodal physical behavior and scene semantic understanding, and to provide supervised training signals for the CoT explicit reasoning module. The responses in the dataset are all structured texts containing the reasoning process and the final conclusion. The specific construction process is as follows: (1) Cold start of topic seed pool P: First, the real business history data of the five smart scenarios are represented by the bidirectional encoder model (BERT). The K-means unsupervised clustering algorithm yields 16 core topic clusters for each scenario, totaling 80 initial topics. Based on the semantics of the cluster centers, the model is further expanded using the Qwen2.5-7B-Instruct model to generate standardized and applicable query topics, forming the final topic seed pool P, ensuring that the topics closely match the distribution of real business data. (2) Query-structured answer generation: for each entity file The Qwen2.5-7B-Instruct method retrieves the top 10 most relevant topics to the entity from the topic seed pool P, and instantiates each topic into a natural language query based on real multimodal data from the entity archive. Divided into two categories: intent prediction and diagnostic classification; based on Generate structured answers using Qwen2.5-7B-Instruct By consistently using the format "Reasoning Process: XXX; Final Conclusion: XXX", we obtain... Sample; where the reasoning process is strictly based on physical archives. The traceable data is used to complete the causal deduction step by step, and the final conclusion is consistent with the query. This directly corresponds to the business objectives; (3) Two-dimensional quantization verification and correction: for the generated The samples undergo consistency checks. Samples that fail the check are corrected or discarded. Specific rules include: Keyword matching verification: extraction The core entities, events, and numerical keywords in the text The overall matching score was lower than [percentage missing] in the search and matching process. Samples are discarded directly; if the reasoning process exceeds [a certain threshold], the sample is discarded. If the content lacks supporting data, discard the sample directly; if it is lower than... If the content lacks support, truncate and correct any unfounded content, while retaining valid reasoning and conclusions. Cosine similarity (CS) verification: Calculated using a general text embedding model (gte-base). and The cosine semantic similarity is used, and samples with a similarity of less than 0.6 are discarded directly; (4) Control the sequence length of the validated samples to ensure that the length of a single sample does not exceed the context window of the multimodal large model. This ultimately forms a multimodal scenario entity question-answering dataset. The expression is: .

[0030] Step S114: Comparative learning of positive and negative samples and explicit definition To avoid representation collapse during model pre-training, specific rules for selecting positive and negative samples and hard negative samples are defined for the contrastive learning stage. All samples are drawn from... and : (1) Definition of positive samples: ① Different query-response pairs under the same entity and the same topic; ② Sample pairs corresponding to entity file slices with adjacent time windows and consistent behavioral characteristics of the same entity; ③ Sample pairs corresponding to different semantically equivalent queries of the same entity. (2) Definition of negative samples: ① All sample pairs of different entities within the same batch; ② Sample pairs corresponding to unrelated queries of the same entity but different topics; ③ Sample pairs of the same entity with significant changes in behavioral characteristics at different time windows; (3) Hard negative sample screening: Calculate the cosine semantic similarity between negative and positive samples, retain only hard negative samples with similarity between 0.4 and 0.6, filter out easy-to-distinguish simple negative samples, and improve the model's fine-grained feature discrimination ability.

[0031] Step S115 Physical World Truth Dataset Build Provides supervised truth values ​​for physical world consistency loss. All truth values ​​are automatically extracted from raw business data without manual annotation and are consistent with entity archives. Query One-to-one correspondence, the specific construction process is as follows: (1) For each entity file Extract the corresponding physical world truth values ​​based on the data modality and the type of smart scenario it belongs to. The extraction rules are as follows: quantifiable physical indicators or discrete behavioral labels. Table 1

[0032] (2) Extracted physical world truth values Standardization is performed: continuous numerical true values ​​are normalized, and discrete labeled true values ​​are one-hot encoded. (3) Transfer the physical archives Query With the truth value of the standardized physical world The training sample pairs are constructed, ultimately forming the ground truth dataset of the physical world. The expression is: .

[0033] Based on the above-mentioned entity temporal behavior prediction dataset Multimodal Entity Question Answering Dataset Physical World Truth Dataset This constitutes the pre-trained dataset.

[0034] S120: Encode the multimodal data in the pre-trained dataset according to the pre-built hierarchical multimodal entity encoder to generate an entity input sequence with controllable length.

[0035] As a preferred embodiment, this invention designs a three-level hierarchical multimodal entity encoder that progresses from coarse to fine. It designs dedicated encoding and preprocessing rules for each of the six modalities, and combines them with a core event sequence compression mechanism. By using a fixed token limit and a self-attention mechanism (SAM) to adaptively filter weights, it strictly controls the length of the input sequence, ensuring that the total sequence length does not exceed 70% of the context window of the multimodal large model (e.g., the total number of tokens is ≤2800 under a 4k window). Finally, the original multimodal data is distilled into a unified representation with multiple granularities, which is adapted to the native embedding space of the multimodal large model.

[0036] Step S121 Event-level coding and modality-specific adaptation For each data modality The modal set is denoted as Text interaction, structured attributes, spatiotemporal trajectories, IoT sensor timing, video frames, audio Complete event-level coding and modal-specific adaptation, and output event-level embeddings. The specific process is as follows: (1) For each mode The original event sequence is projected into an initial embedding using a modality-specific pre-trained encoder. ,in For each time step of the event sequence, the dedicated pre-trained encoder and preprocessing rules for each modality are shown in Table 2: Table 2

[0037] (2) For each mode Design a dedicated event adapter, implemented using a multilayer perceptron (MLP), for initial embedding. By sequentially performing layer normalization and nonlinear transformation, event-level embeddings that retain fine-grained atomic features are obtained. The calculation formula is: ; in, For modality A dedicated multilayer perceptron event adapter; LayerNorm is for layer normalization operations.

[0038] Step S122 Intramodal aggregation and modal-level unified characterization Aggregate and transform all event-level embeddings within the same modality to eliminate random noise from single events, capture behavioral trends within the modality, and output a unified modality-level embedding. The specific process is as follows: (1) Using mean pooling (MP) for mode All event-level embeddings Aggregation is performed to generate intra-modal summary vectors. ; (2) Design a modality adapter for full modality sharing, which is implemented using a multilayer perceptron (MLP) to summarize the intra-modal vectors. Layer normalization and nonlinear transformation are performed sequentially to generate a modal-level unified embedding that adapts to the input dimensions of a multimodal large model. The calculation formula is: ; in, It is a multilayer perceptron modal adapter that shares all modalities.

[0039] Step S123 Global Entity-Level Representation Fusion Design a dedicated adapter for the physical entity, implemented using a multilayer perceptron (MLP), to handle all modalities. Modal level embedding Perform global fusion to generate global entity-level embeddings that can represent the full-dimensional features of entities. It can fully capture the overall behavioral profile and cross-modal interaction characteristics of an entity.

[0040] Step S124: Multi-granularity characterization, structured splicing, and core event sequence length control This invention employs a core event retention mechanism that combines a fixed token cap with self-attention (SAM) adaptive weight filtering. It structurally concatenates global entity-level embeddings, modality-level embeddings, and core event-level embeddings to generate the final entity input token sequence. And strictly control the sequence length. The specific process is as follows: (1) Self-attention mechanism for adaptive weight selection: in each modality After the event-level encoding is completed, a single-head self-attention layer is added, with the input being all event-level embeddings of this modality. Calculate the attention weight for each event token. The calculation formula is: ; in, These are the learnable query and the key projection matrix, respectively. The dimension of the projection matrix is ​​given, and Softmax is the normalized exponential function; event tokens within the same modality are weighted according to attention. Sort in descending order and keep the highest weighted order. Each event token, with core event-level embeddings retained across all modalities. , For modality The number of core event tokens retained; (2) Fixed upper limit on the number of core event tokens: set for each modality A fixed upper limit ensures the total number of core event tokens across all modalities. The specific upper limit is: structured attributes Text interaction Spatiotemporal trajectory / IoT sensor timing Video frames / audio , (3) Multi-granularity characterization structured splicing: embedding modal level uniformly Global entity-level embedding Core event-level embeddings for each modality The initial entity input sequence is generated by splicing elements according to a fixed structure. The calculation formula is: ; (4) Dynamic length verification and compression: For the initial entity input sequence Perform a length check; if the total length exceeds the multimodal large model context window... Dynamically adjust non-core modalities based on scenario-based business priorities. Further compress the number of tokens to ensure the final The length meets the model input requirements, resulting in the final entity input sequence.

[0041] S130: Based on the pre-trained dataset, perform multi-objective joint optimization pre-training on the pre-built query-anchored dual-tower multimodal large model to obtain a general pre-trained model.

[0042] As a preferred embodiment, this invention constructs a query-anchored dual-tower multimodal large-scale model pre-training architecture. The backbone adopts the Qwen2.5-0.5B / 1.5B-Instruct parameter fine-tuning model, which is customized for smart physics scenarios. A CoT explicit reasoning module and dual-task output branches are added. At the same time, a multi-objective joint optimization objective is designed, which includes contrast alignment loss, autoregressive next token prediction loss, physical world consistency loss, and classification cross-entropy loss. This completes the general pre-training of the model and achieves the unity of semantic discriminability, reasoning logic, physical consistency, and classification accuracy of entity representation.

[0043] Step S131: Query the anchored dual-tower multimodal large model architecture design. This invention employs a weight-sharing dual-tower Transformer decoder architecture, consisting of an anchor tower and a semantic tower. The two towers share all the backbone parameters of the multimodal large model, ensuring that entity behavior features, semantic labels, and physical truth values ​​are mapped to a unified latent space. The specific architecture design is as follows: (1) Anchor Tower: The anchor tower is a query-aware feature aggregation and reasoning module. It takes the final entity input sequence as input and integrates the natural language query of the downstream scene. It is appended to the end of the entity input sequence as a semantic anchor; firstly, the entity features and query semantics are deeply fused through the backbone of a multimodal large model to output the initial representation of the entity to which the query is anchored. Secondly, a CoT (Conceptual Reasoning) explicit reasoning module was added, based on the initial representation of entities. With query A multi-step thought chain reasoning process is generated through a multimodal large model, and the supervision signal for the reasoning process comes from... The structured reasoning text in the text ultimately generates scene-adaptive entity representations based on the reasoning process. Establish a causal reasoning link from physical behavior data to scenario requirements / risk diagnosis; (2) Semantic Tower: The semantic tower is a truth-encoding module, and the input is the query. Corresponding target answer / physical truth value By sharing the weights of the anchor tower with the multimodal large model backbone, the target answer / physical truth is projected into a dense semantic vector. The semantic and physical truth values ​​used for entity representation alignment are calculated using the following formula: ; in, This is the encoding function for the backbone of a multimodal large model of semantic towers; (3) Dual-task output branch: based on scene-adaptive entity representation Two dedicated output branches are designed to adapt to two types of core query tasks in smart scenarios: Intent prediction branch: Outputs the probability distribution of an entity's future behavior, suitable for intent prediction tasks such as travel prediction and service click prediction; Diagnostic classification branch: Add a linear classification head, based on Output classification results to adapt to diagnostic classification tasks such as risk identification and demand determination.

[0044] Step S132 Multi-objective joint optimization objective design The multi-objective joint optimization objective designed in this invention includes a contrast alignment loss. Autoregressive prediction loss for the next token Loss of consistency in the physical world Classification cross-entropy loss The total loss function is obtained by weighting and summing the loss terms using learnable weights. The various loss terms work together to ensure that the model generates adaptive entity representations of the scene. It also possesses strong semantic distinguishability, rich reasoning logic, consistency with the physical world, and high classification accuracy.

[0045] Preferred, query-conditional comparison alignment loss Adaptive entity representation of a scene is achieved by employing Information Noise Contrast Estimation (InfoNCE) loss based on margin mask filtering. truth vectors generated with semantic tower The positive samples are aligned while the distance from the negative samples is increased. The calculation formula is as follows: ; in: The batch size for model training; This is the function for calculating cosine similarity. This is a temperature coefficient used to adjust the discrimination between samples; The normalization factor comprises four parts: positive samples, negative samples from the same batch, negative samples representing entities, and negative samples representing responses. The calculation formula is as follows: ; in, This is a margin mask used to filter potential false negative samples. When the cosine similarity between a negative sample and the anchor sample exceeds a threshold, it is removed from the negative sample set. The calculation formula is as follows: ; in, This is the margin threshold, with a value of 0.2, used as the critical value to determine whether a negative sample is a false negative sample. These are candidate vectors for negative samples, from the set Used with anchored samples Calculate similarity to determine whether it is a false negative sample.

[0046] Preferably, the autoregressive next token prediction loss Using autoregressive next-to-token prediction (NTP) loss, the anchor tower is autoregressively reconstructed to answer the target. In conjunction with the CoT reasoning process, it supplements fine-grained semantic modeling and reasoning logic learning, mitigating the coarse representation granularity problem caused by contrastive alignment loss. The calculation formula is as follows: ; in: Answer the target The total sequence length of the CoT inference process; For the first in the sequence One token; Answer the target The sequence consisting of the CoT inference process is the first All tokens prior to the first token (i.e., tokens 1 to 1) (one token) Predict the probability of the token based on the context for anchoring towers.

[0047] Preferably, physical world consistency loss Design a mean squared error (MSE) type physical world consistency loss for smart physics scenarios to constrain adaptive entity representation. The predictive ability of quantifiable indicators in the physical world ensures that the representation is consistent with physical laws, and solves the problem of representation distortion caused by high data noise and sparse intent in the physical world. The calculation formula is as follows: ; in: The physical prediction head is implemented using a multilayer perceptron. For the truth value of the physical world; This is the function for calculating the mean squared error.

[0048] The weight of the physical world consistency loss can be dynamically adjusted according to the scenario task type. The weight of regression scenarios such as trajectory prediction and device status prediction is increased to 1.2, while the default weight is maintained for diagnostic classification scenarios.

[0049] Preferred, classification cross-entropy loss For diagnostic classification tasks, a cross-entropy (CE) loss is designed to optimize the classification accuracy of the diagnostic classification branch. The calculation formula is as follows: ; in: The classification head is implemented using a multilayer perceptron; For diagnostic classification tasks, provide truth labels (e.g., risk / non-risk, need / no need). This is the function for calculating cross-entropy loss.

[0050] Furthermore, through learnable weights The total loss function for model pre-training is obtained by weighted summation of each loss term. The calculation formula is: ; The default weights for general smart scenarios are set as follows: The weight values ​​can be dynamically adjusted according to the business objectives of different scenarios.

[0051] Preferably, low-rank adaptation (LoRA) technology is used to fine-tune the backbone of the multimodal large model, avoiding the excessive computational cost and overfitting problems caused by full fine-tuning. The specific training parameters and implementation process are as follows: (1) Training data: fusion Three types of datasets, categorized by The proportions are divided into training set, validation set, and test set; (2) Training parameters: The rank of LoRA is set to 64, and the scaling factor is... The training batch size is set to 32, the total training steps are 50k, the optimizer is the Adaptive Moment Estimator (AdamW), and the initial learning rate is... A cosine learning rate decay strategy is adopted. (3) Hardware environment: Pre-training was performed on 64 A100-80G graphics processing units (GPUs) using a data parallel training strategy; (4) Output results: After training is completed, the pre-trained model weights of the general intelligent scene and the scene adaptive entity representation are output. The dimensions are fixed at 128.

[0052] S140: For specific smart scenarios, perform lightweight scenario adaptation on the general pre-trained model to obtain a scenario-adapted model.

[0053] As a preferred embodiment, this invention designs a soft cueing optimization and gating modal attention mechanism based on scene clustering on the basis of a general pre-trained model. It freezes all weights of the multimodal large model backbone and the hierarchical multimodal encoder, and optimizes only a small number of learnable parameters. This enables the general model to be adapted to subdivided smart scenes in a lightweight and low-cost manner, while allowing the model's attention to focus on scene-related modal features and suppressing irrelevant noise.

[0054] Step S141 Smart Scene Clustering and Category Prototype Center Construction (1) The five core scenarios of smart transportation, smart community, smart campus, smart government affairs and smart tour guide are subdivided to obtain the specific business tasks under each scenario (such as commuting travel prediction and illegal parking risk identification in smart transportation; matching of elderly home care services and management of mobile population in smart community). (2) Label and cluster the samples for each subdivided business task to construct... A learnable category prototype center To further subdivide the number of task categories, the category prototype center is used to constrain the latent space distribution of scene adaptive entity representations, allowing representations of the same category to cluster together and representations of different categories to separate.

[0055] Step S142: Scenario-Specific Learnable Soft Hint Token Design For each segmented business task in a smart scenario, a set of learnable soft cue tokens is designed as a differentiable scenario controller to adaptively represent entities within the scenario. The latent space is modulated, and the specific design rules are as follows: (1) Freeze all weights of the multimodal large model backbone and hierarchical multimodal encoder trained in step S33, and optimize only the soft cue token; (2) The number of soft hint tokens is set to 6 by default, and can be dynamically adjusted between 1 and 16 depending on the complexity of the subdivided business tasks; (3) Dimensions of soft hint tokens and scene-adaptive entity representation The dimensions remain consistent, both being 128 dimensions.

[0056] Step S143: Design of Scene Gated Modal Attention Mechanism A modal gating weight matrix is ​​designed for each segmented business task in a smart scenario to achieve scene-adaptive modal attention redistribution. This allows the model to automatically increase the attention weights of scene-related modalities and suppress noise from irrelevant modalities. The specific design is as follows: (1) The dimension of the modal gating weight matrix is ​​the same as the number of modes. Each mode corresponds to a learnable gating weight value, and the weight value range is... ; (2) Modal gating weights and scene-specific soft cue tokens are jointly optimized. Based on the scene characteristics of the subdivided business tasks, the attention weights of each modality are automatically adjusted (e.g., the weights of spatiotemporal trajectories and checkpoint data are increased in smart transportation scenarios, and the weights of IoT sensors and access control records are increased in smart community elderly care scenarios). (3) Apply modal gating weights to the modal-level unified embedding The resulting gated modal embedding is then used in subsequent representation fusion and inference.

[0057] Step S144 Joint optimization based on prototype contrastive loss Using prototype comparison loss Scene-specific soft cue tokens, modal gating weight matrix, and category prototype center Joint optimization is performed to enhance the representational discriminative power within the scene. The calculation formula is as follows: ; in: This is an adaptive entity representation of the scene after the soft cue token and modal gating weights are modulated together; The true label for the sample; The category prototype center corresponding to the truth value label; For temperature coefficient, This is the training batch size.

[0058] The training steps for joint optimization are set to 500 by default and are completed on a single A100-80G GPU. After optimization, the scene adaptation model for each subdivided business task is output.

[0059] S150: Input the entity input sequence and the current natural language query into the scenario adaptation model to generate a thought chain reasoning conclusion.

[0060] As a preferred embodiment, this invention designs a multi-scenario low-latency inference architecture based on key-value cache (KV-cache) prefix sharing, realizing one-time entity encoding and multi-scenario reuse, significantly reducing inference latency and deployment costs; at the same time, a new scenario-adaptive interactive instruction generation module is added, which transforms the entity representation and inference conclusions output by the model into executable standardized interactive instructions, connecting to IoT device platforms, business service systems and content delivery channels, and constructing a complete interactive closed loop of "representation generation-inference diagnosis-instruction output-device linkage", realizing true intelligent scenario adaptive interaction.

[0061] Step S151 Entity prefix representation pre-computation and KV-cache persistence For each entity, input the token sequence. Pre-compute to obtain its key (K) and value (V) cache values ​​in the Transformer layer, denoted as KV-cache; The pre-computed KV-cache is used as a shared semantic prefix cache for entities and stored in a distributed in-memory database (Redis Cluster) to achieve persistent storage of KV-cache. This prefix cache is independent of scenario queries and can be repeatedly used for query inference of all downstream smart scenarios after a single calculation. Design a daily incremental update mechanism: only for newly added multimodal interaction data of an entity each day, update the event-level and modal-level encoding of the corresponding modality, refresh the token of that modality and the corresponding KV-cache part, without having to reprocess the entity's full 90-day historical data; A 90-day rolling window mechanism is adopted to clean up historical data and corresponding tokens that have expired beyond the time window, ensuring the timeliness of entity representation and the stability of computational costs.

[0062] Step S152 Multi-scenario query incremental inference mechanism Multiple smart scenario queries for the same entity ( (For the number of queries in the smart scenario), an incremental inference mechanism with KV-cache prefix sharing is adopted. The specific process is as follows: (1) During inference, the entity prefix KV-cache pre-computed and persisted in step S51 is reused, and there is no need to re-encode the entity input sequence; (2) Query for each scenario Incremental processing is performed only on short text query sequences. The computational and model forward inference, with an incremental inference complexity of O(n) for each scenario query, is... ,in For query The length of the text sequence; (3) Obtain the scene-adaptive entity representation corresponding to each scene query through incremental reasoning. CoT inference conclusions and task output results enable "one-time entity encoding for reuse in multiple scenarios".

[0063] S160: Generate standardized interactive instructions based on the reasoning conclusion of the thought chain, and send them to physical devices or business systems for execution to form an interactive closed loop.

[0064] As a preferred embodiment, scene-adaptive entity representation based on model output. Based on the CoT inference conclusions, standardized executable interaction commands are generated. These commands are divided into three categories: device control, service scheduling, and content push. Simultaneously, a command compliance verification and execution mechanism is designed to achieve a closed loop of interaction with the physical world. The specific process is as follows: Interactive command generation: Based on the business needs of different smart scenarios, three types of standardized interactive commands are generated, with each command format corresponding to a specific execution object: Device control commands: For smart transportation, smart community, and smart campus scenarios, generate IoT device control commands such as traffic light control, access control permission adjustment, camera pan-tilt control, and environmental equipment switching, and connect to the IoT device control gateway; Service dispatch instructions: For smart community, smart government affairs, and smart tour guide scenarios, generate service dispatch instructions such as home-based elderly care services, government service appointments, scenic spot tour scheduling, and property maintenance order dispatch, and connect with the business service systems of each scenario; Content push instructions: For fully intelligent scenarios, generate personalized content push instructions such as policy push, travel planning, campus notices, and scenic spot guides, and connect with content delivery channels such as apps, mini programs, and radio; Command compliance verification: Design a command compliance verification module to perform permission, security, and feasibility verification on generated interactive commands, and filter out invalid commands that do not comply with security specifications, exceed device permissions, or have no execution conditions; Command execution and result feedback: The verified interactive command is routed to the corresponding execution object (IoT device control gateway, business service system, content delivery channel) to complete the command execution; at the same time, the command execution result is fed back to the model training platform for continuous iterative optimization of the model.

[0065] Preferably, this invention designs a three-tier deployment architecture of "prefix cache pool + scene inference service pool + instruction execution gateway" to meet the high-concurrency, low-latency interaction requirements of industrial-grade smart scenarios. The specific architecture design is as follows: Prefix cache pool: Built on a distributed in-memory database (RediSCluster), it stores a pre-computed key-value cache for all entities, supports millions of concurrent reads, and ensures fast retrieval of key-value cache. Scene Inference Service Pool: Built on GPU inference cluster, it undertakes real-time interactive query requests from various smart scenes, reuses KV-cache in the prefix cache pool to complete incremental inference, and outputs scene adaptive entity representation, CoT inference conclusions and interactive instructions. Command Execution Gateway: As a unified exit point for interactive commands, it connects to IoT device platforms, business service systems, and content delivery channels to complete the verification, routing, execution, and result feedback of interactive commands.

[0066] Based on the above embodiments, the core beneficial effects of the present invention are as follows: (1) This invention solves the problem of causal reasoning discontinuity in physical entity interaction, significantly improving the accuracy of reasoning diagnosis: The present invention adds a CoT explicit reasoning module to the query anchoring framework, and combined with the physical world consistency loss, it connects the entire link of physical behavior data - structured representation - causal reasoning diagnosis - scene interaction actions, which can simultaneously adapt to two core tasks: intent prediction and diagnostic classification, solving the pain points of high noise and sparse intent in physical world data. Experimental test results show that the present invention achieves an average area under the curve (AUC) of 0.831 in 12 smart scene benchmark tasks, which is 10.2% higher than the existing general representation model, and the accuracy of risk diagnosis tasks is improved by 11.7%.

[0067] (2) It solves the engineering feasibility problem of processing long sequence heterogeneous data and the sequence length is strictly controllable: This invention designs a modality-specific core event filtering and sequence compression algorithm for long sequence heterogeneous data such as spatiotemporal trajectories, IoT time series, and video frames. Through a fixed token upper limit and self-attention weight adaptive filtering mechanism, it strictly controls the total input sequence length to not exceed 70% of the multimodal large model context window, which completely solves the technical problems of sequence length explosion and exceeding the large model context window. It has strong engineering feasibility.

[0068] (3) It achieves extremely low scene adaptation cost and can quickly implement segmented business: Based on clustering soft prompts and gated modal attention mechanism, this invention only needs to optimize a small number of learnable parameters to complete the adaptation of new scenes, without the need for full fine-tuning of large models. The scene adaptation cycle is shortened from the monthly level to the hourly level, and the computing power cost is reduced by more than 92%. At the same time, the scene gating mechanism can allow modal attention to automatically match scene requirements, further improving the model performance of segmented scenes.

[0069] (4) It has industrial-grade high-concurrency and low-latency deployment capabilities: Through the incremental inference mechanism of KV-cache prefix sharing, this invention realizes one-time entity encoding and multi-scenario reuse. The single-scenario incremental inference latency is reduced to less than 5ms. New scenarios do not require expansion of the full GPU cluster. It can support real-time interaction of multiple scenarios for millions of entities and fully meet the industrial-grade deployment requirements of smart city scenarios.

[0070] Example 2 This second embodiment targets the urban operating vehicle management and travel service scenario, with the entities being operating vehicles and drivers. The core queries are "Does the vehicle have a risk of fatigued driving?", "Does the driver have commuting needs during morning and evening rush hours?", and "Predicted driving trajectory of the vehicle for the next hour". The specific implementation process is as follows: Data processing: Collect vehicle trajectory data, checkpoint capture data, navigation interaction data, and driver behavior data from the past 90 days. Compress trajectory points using the Douglas-Puk algorithm, retaining core events such as inflection points, speeding points, and stopping points. Adaptively select and retain the top 64 core trajectory tokens using a self-attention mechanism, generating standardized entity profiles. Simultaneously, the GPS coordinates of the vehicle trajectory at a given moment are extracted as the true value in the physical world. .

[0071] Model pre-training and scenario adaptation: Based on the SceneU dataset, a dual-tower multimodal large model general pre-training was completed for query anchoring. Soft prompts were tuned and gating modal attention mechanism was optimized for three sub-tasks: fatigue driving risk identification, travel demand prediction, and trajectory prediction, thereby improving the modal weights of spatiotemporal trajectory and driver behavior data. Incremental reasoning: Pre-compute the entity input token sequence for the vehicle The corresponding KV-cache is persisted. Upon receiving a query request, the KV-cache is reused to complete incremental inference and generate scene-adaptive entity representations. The CoT explicit reasoning module outputs the structured conclusion: "Reasoning process: The vehicle has been driven for more than 10 hours a day for the past 7 days, there have been 3 emergency braking incidents in the past 24 hours, and there is a single continuous driving record of more than 4 hours without rest, which is consistent with the characteristics of high-risk fatigue driving; final conclusion: The vehicle has a high risk of fatigue driving."

[0072] Interactive command generation and execution: Based on reasoning conclusions, three types of standardized interactive commands are generated: ① Device control command: Send fatigue driving audible and visual warnings to the vehicle terminal, and link the vehicle camera to increase the capture frequency to 5 minutes / time; ② Service dispatch command: Push the nearest rest area and parking spot within 3 kilometers to the driver, and simultaneously push road condition warnings along the way; ③ Control command: Report high-risk vehicles to the traffic control platform and include them in the key monitoring list.

[0073] Experimental test results: The accuracy of fatigue driving risk identification warning was improved by 11.7%, the average position error of vehicle trajectory prediction was reduced by 18.7%, and the single-scene incremental inference latency was 4.2ms.

[0074] Example 3 This third embodiment targets a smart community home-based elderly care service scenario, with the entity being elderly residents of the community. The core queries are "Does this resident have a need for home-based elderly care services?" and "Does this resident have any home safety risks?" The specific implementation process is as follows: Data processing: Collect residents' access control records, home environment sensor data, property maintenance records, and community service interaction data from the past 90 days. Compress IoT time-series data using a sliding window peak extraction algorithm, retaining core events such as abnormal fluctuations and long-term unchanged events. Adaptively select and retain the top 64 IoT core tokens through a self-attention mechanism to generate standardized entity profiles. Simultaneously, the average value of the home environment sensor data for the next period is extracted as the true value of the physical world. .

[0075] Model pre-training and scenario adaptation: General model pre-training was completed based on the SceneU dataset, and scenario adaptation was carried out for two sub-tasks: identification of the needs of elderly people living alone and diagnosis of home safety risks, thereby improving the modal weights of IoT sensor and access control record data.

[0076] Incremental reasoning: The resident's KV-cache is pre-calculated and persisted. Upon receiving a query request, incremental reasoning is completed. The CoT explicit reasoning module outputs a structured conclusion: "Reasoning process: This resident only leaves early and returns late once a day in the past 90 days. There are no records of entry and exit through the access control system during the day, no records of visits by children, no property maintenance requests or community service interactions in the past 30 days, and the human body sensor in the living room has not been triggered for a long time during the day, which is consistent with the behavioral characteristics of elderly people living alone in empty nests; Final conclusion: This resident is an elderly person living alone, and there is a need for home-based elderly care services and home safety risks."

[0077] Interactive command generation and execution: Based on reasoning conclusions, three types of standardized interactive commands are generated: ① Device control commands: Increase the monitoring priority of the emergency call button in the resident's home, link the smart door lock to open abnormal door opening warning, and link the gas sensor to improve alarm sensitivity; ② Service dispatch commands: Dispatch orders to the community elderly care service station, arrange staff to visit the home within 48 hours, and simultaneously match customized cleaning and meal assistance services; ③ Content push commands: Push weekly notifications of the elderly's home status to the resident's children, and simultaneously push community elderly care service policies and emergency contact information.

[0078] Experimental test results: The response rate of community-based elderly care services for seniors living alone has increased. Improved timeliness of home safety incident early warning The single-scenario incremental inference latency is 3.8 ms.

[0079] Example 4 This fourth embodiment targets a smart government enterprise service scenario, where the entity is a small or medium-sized enterprise (SME) within the jurisdiction. The core queries are "What preferential policies does this enterprise meet?" and "Does this enterprise have any policy application needs?" The specific implementation process is as follows: Data processing: Collect the enterprise's operating data, tax data, application records, policy browsing data, and industry qualification data over the past 90 days. Use a self-attention mechanism to adaptively filter and retain the top 32 core event tokens, generating standardized entity files. Simultaneously, extract the enterprise's policy application behavior for the next cycle as the physical world truth value. .

[0080] Model pre-training and scenario adaptation: General pre-training of the model was completed based on the SceneU dataset. Scenario adaptation was carried out for two sub-tasks: matching preferential policies for enterprises and identifying policy application needs, thereby improving the modal weights of enterprise operation data and policy browsing trajectory.

[0081] Incremental Inference: The system pre-computes and persists the enterprise's key-value cache. Upon receiving a query request, it completes incremental inference and outputs a structured conclusion via the CoT explicit inference module: "Inference process: This enterprise is a technology-based SME, established for 3 years, with the previous year's R&D investment accounting for..." The company possesses two utility model patents and has repeatedly reviewed policies for cultivating specialized, refined, and innovative enterprises within the past 30 days, meeting the core requirements for applying for specialized, refined, and innovative SMEs. Final conclusion: The company meets the application requirements for specialized, refined, and innovative SMEs and has a clear need for policy application.

[0082] Interactive command generation and execution: Based on reasoning conclusions, three types of standardized interactive commands are generated: ① Content push command: Accurately push specialized and innovative enterprise application policies, application guidelines, and material templates to the enterprise legal person, and mark the core application conditions and deadlines; ② Service dispatch command: Match enterprises with dedicated government service specialists to provide one-on-one application guidance, and simultaneously open online application green channels; ③ Service guidance command: Push online application portals and processing procedures to enterprises, and simultaneously remind them of the required material list and precautions.

[0083] Experimental test results: The rate of one-stop resolution of government services for preferential policies for enterprises has increased. The single-scenario incremental inference latency is 4.5 ms.

[0084] Example 5 Figure 5 This is a structural schematic diagram of a smart scene adaptive interaction system 500 based on a multimodal large model provided in Embodiment 5 of the present invention, as shown below. Figure 5 As shown, the system includes: The multimodal data acquisition module 510 is used to construct a multimodal entity understanding pre-training dataset covering multiple smart physics scenarios; The multimodal data preprocessing module 520 is used to encode the multimodal data in the pre-trained dataset according to the pre-constructed hierarchical multimodal entity encoder to generate an entity input sequence with controllable length. The dual-tower multimodal large model pre-training module 530 is used to perform multi-objective joint optimization pre-training on the pre-built query-anchored dual-tower multimodal large model based on the pre-training dataset to obtain a general pre-trained model; The scene adaptive soft prompting optimization module 540 is used to perform lightweight scene adaptation on the general pre-trained model for specific smart scenes to obtain a scene-adapted model. Online reasoning module 550 is used to input the entity input sequence and the current natural language query into the scenario adaptation model to generate a thought chain reasoning conclusion; The scene interaction instruction generation and execution module 560 is used to generate standardized interaction instructions based on the reasoning conclusion of the thought chain, and send them to physical devices or business systems for execution to form an interaction closed loop.

[0085] The intelligent scene adaptive interaction system based on a multimodal large model provided in this embodiment of the invention can execute the intelligent scene adaptive interaction method based on a multimodal large model provided in any of the embodiments of the invention above. It has the corresponding functions and beneficial effects of executing the intelligent scene adaptive interaction method based on a multimodal large model. For detailed process, please refer to the relevant operations of the intelligent scene adaptive interaction method based on a multimodal large model in the foregoing embodiments.

[0086] Example 6 Figure 6 This is a schematic diagram of the structure of an electronic device provided in Embodiment Six of the present invention. The electronic device 10 is intended to represent various forms of digital computers, and may also represent various forms of mobile devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0087] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0088] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0089] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 executes the intelligent scene adaptive interaction method based on multimodal large models described above.

[0090] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0091] The above embodiments are merely illustrative examples and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A smart scene adaptive interaction method based on a multimodal large model, characterized in that, include: Construct a multimodal entity understanding pre-trained dataset covering multiple smart physics scenarios; The multimodal data in the pre-trained dataset is encoded using a pre-built hierarchical multimodal entity encoder to generate entity input sequences of controllable length. Based on the pre-trained dataset, a multi-objective joint optimization pre-training is performed on the pre-built query-anchored dual-tower multimodal large model to obtain a general pre-trained model; For specific smart scenarios, the general pre-trained model is subjected to lightweight scenario adaptation to obtain a scenario-adapted model. The entity input sequence and the current natural language query are input into the scenario adaptation model to generate a thought chain reasoning conclusion. Based on the reasoning conclusions of the thought chain, standardized interactive instructions are generated and sent to physical devices or business systems for execution, forming an interactive closed loop.

2. The method according to claim 1, characterized in that, The construction of a multimodal entity understanding pre-trained dataset covering multiple smart physics scenarios includes: Collect end-to-end multimodal interaction data of various interactive entities within a set historical time window to construct entity profiles; A future behavior summary is generated based on the entity file, and the future behavior summary is concatenated with a general template query to form a scenario entity time-series behavior prediction dataset. A topic seed pool is constructed through unsupervised clustering and generative models. Natural language query and structured answer samples are generated based on the entity archives. A multimodal scenario entity question answering dataset is constructed based on the natural language query and structured answer samples. Quantifiable physical world truth values ​​are automatically extracted from the entity files, and a physical world truth value dataset is constructed based on the physical world truth values, the entity files, and the natural language query. The pre-training dataset is composed of the scene entity temporal behavior prediction dataset, the multimodal scene entity question answering dataset, and the physical world truth dataset.

3. The method according to claim 2, characterized in that, Based on the entity archives, natural language query and structured answer samples are generated, and a multimodal scenario entity question-answering dataset is constructed based on the natural language query and structured answer samples, including: For each entity file, topics related to that entity file are retrieved from the topic seed pool, and each topic is instantiated into a natural language query based on the real multimodal data of the entity file. The entity files and natural language queries are input into the large language model to generate structured answer samples corresponding to the natural language queries, thus obtaining natural language query-structured answer sample pairs. The natural language query-structured answer sample pairs are validated in two dimensions, and the sample pairs that pass the validation are retained. The sequence length of the validated sample pairs is controlled to ultimately form a multimodal scenario entity question-answering dataset.

4. The method according to claim 1, characterized in that, Based on a pre-built hierarchical multimodal entity encoder, the multimodal data in the pre-trained dataset is encoded to generate entity input sequences of controllable length, including: For different data modalities, a pre-built hierarchical multimodal entity encoder is used to extract the initial embedding, and the event-level embedding is output through a multimodal-specific adapter; Aggregate all event-level embeddings within the same modality to generate a unified modality-level embedding; All modalities are uniformly embedded and fused at the modal level to generate a global entity-level embedding; By using a fixed token limit and a self-attention mechanism, events within each modality are weighted and sorted, and core event-level embeddings are retained according to a preset limit. The modal-level unified embedding, global entity-level embedding, and core event-level embedding are structurally concatenated, and the length of the concatenated sequence is controlled to generate the entity input sequence.

5. The method according to claim 4, characterized in that, The method of weighting events within each modality using a fixed token cap and a self-attention mechanism, and retaining core event-level embeddings according to a preset cap, includes: Within each modality, the attention weights of each event-level embedding are calculated using a self-attention layer; The event-level embeddings are sorted in descending order based on the attention weight of each event-level embedding; Based on the fixed token limit preset for each modality, a corresponding number of event-level embeddings are reserved as core event-level embeddings.

6. The method according to claim 5, characterized in that, The modal-level unified embedding, global entity-level embedding, and core event-level embedding are structurally concatenated, and the length of the concatenated sequence is controlled to generate the entity input sequence, including: According to the preset splicing order, the modal-level unified embedding, global entity-level embedding, and core event-level embedding are structurally spliced ​​together to generate the initial entity input sequence; The initial entity input sequence is length-checked to determine whether its total length exceeds the preset proportion threshold of the multimodal large model context window; If the preset ratio threshold is exceeded, the number of core events retained for each modality is dynamically adjusted according to the scenario business priority, and the initial entity input sequence is compressed to obtain the final entity input sequence.

7. The method according to claim 6, characterized in that, Based on the pre-trained dataset, a pre-built query-anchored dual-tower multimodal large model is pre-trained using multi-objective joint optimization to obtain a general pre-trained model, including: Construct a query-anchored dual-tower multimodal large model, wherein the dual-tower model includes an anchoring tower and a semantic tower with shared weights; The entity input sequence is concatenated with a natural language query and then input into the anchoring tower to generate a scene-adaptive entity representation; the physical world truth values ​​in the pre-trained dataset are input into the semantic tower and mapped into semantic vectors for representation alignment. The dual-tower model is pre-trained by comparing the alignment loss, autoregressive generation loss, physical consistency loss, and classification loss to form a multi-objective joint optimization.

8. The method according to claim 1, characterized in that, For specific intelligent scenarios, a lightweight scenario adaptation is performed on the general pre-trained model to obtain a scenario-adapted model, including: Freeze the backbone parameters of the general pre-trained model; Cluster the samples of subdivided business tasks to construct a learnable category prototype center; For each sub-task, a set of learnable soft cue tokens is designed to modulate the latent space of entity representations in a contextual manner. Design a learnable modal gating weight matrix and jointly optimize it with the soft cue token to perform scene-adaptive attention redistribution on the representation of each modality; The prototype center, soft cue token, and gating weights of the category are jointly optimized by using prototype contrast loss to output the scene adaptation model.

9. The method according to claim 1, characterized in that, Based on the reasoning conclusions of the aforementioned thought chain, standardized interactive instructions are generated and sent to physical devices or business systems for execution, forming an interactive closed loop, including: Based on the final conclusion in the reasoning chain, a preset instruction template is matched to generate device control, service scheduling, or content push instructions, and the generated instructions are verified for permissions, security, and feasibility. The verified instructions are routed to the corresponding IoT device gateway, business service system, or content push channel for execution, and the execution results are received and sent back.

10. A smart scene adaptive interaction system based on a multimodal large model, characterized in that, include: The multimodal data acquisition module is used to build a pre-trained dataset for multimodal entity understanding covering multiple smart physics scenarios; The multimodal data preprocessing module is used to encode the multimodal data in the pre-trained dataset according to the pre-built hierarchical multimodal entity encoder to generate entity input sequences with controllable length. The dual-tower multimodal large model pre-training module is used to perform multi-objective joint optimization pre-training on the pre-built query-anchored dual-tower multimodal large model based on the pre-training dataset to obtain a general pre-trained model; The scene adaptive soft prompting optimization module is used to perform lightweight scene adaptation on the general pre-trained model for specific smart scenes, so as to obtain a scene-adapted model. The online reasoning module is used to input the entity input sequence and the current natural language query into the scenario adaptation model to generate a thought chain reasoning conclusion; The scene interaction instruction generation and execution module is used to generate standardized interaction instructions based on the reasoning conclusion of the thought chain, and send them to physical devices or business systems for execution to form an interaction closed loop.

Images