Control method and device of intelligent device, electronic device and storage medium

Abstract

The application relates to a control method and device of an intelligent device, an electronic device and a storage medium, the method comprising: acquiring multi-source heterogeneous data; unifying time benchmarks and / or spatial coordinates of the multi-source heterogeneous data to obtain multi-source data after time and / or space alignment; performing multi-modal fusion processing on the multi-source data to generate a multi-modal fusion feature vector; inputting the multi-modal fusion feature vector into a pre-trained intention classifier to output single-time recognition confidence for each preset intention category at a current moment; combining historical cumulative confidence of each intention category at a previous moment and the single-time recognition confidence to determine current cumulative confidence of each preset intention category, and controlling the intelligent device according to a comparison result of the current cumulative confidence and a preset threshold. Through the application, the problems of poor noise resistance and high false triggering rate in the existing vehicle-mounted intention confirmation scheme are solved.

Description

Technical Field

[0001] This application relates to the field of equipment control, and more particularly to a control method and apparatus for an intelligent device, an electronic device, and a storage medium. Background Technology

[0002] Existing in-vehicle intent verification solutions often rely on instantaneous user input or environmental data at a single moment for immediate determination. This means that a single input (such as a voice command, a touch operation, or a gesture) or environmental perception data at a single moment is used to directly determine the user's intent and immediately trigger the corresponding device control action. However, control results based on instantaneous determination of user input or environmental data at a single moment are easily affected by noise, user distraction, or environmental interference, leading to a high risk of false triggering and missed triggering. This is unacceptable in a vehicle-based scenario where safety is paramount. Therefore, existing in-vehicle intent verification solutions suffer from a high risk of false triggering and missed triggering. Summary of the Invention

[0003] This application provides a control method and apparatus for intelligent devices, an electronic device, and a storage medium to solve the problem that existing vehicle intent confirmation schemes have a high risk of false triggering and missed triggering.

[0004] In a first aspect, this application provides a control method for a smart device, comprising: acquiring multi-source heterogeneous data, wherein the multi-source heterogeneous data is heterogeneous sensing data from at least two different types of data sources in the data of a smart device associated with a user; unifying the time reference and / or spatial coordinates of the multi-source heterogeneous data to obtain time- and / or spatially aligned multi-source data; performing multimodal fusion processing on the multi-source data to generate a multimodal fusion feature vector; inputting the multimodal fusion feature vector into a pre-trained intent classifier to output the single recognition confidence score for each preset intent category at the current time; combining the historical cumulative confidence score of each intent category at the previous time and the single recognition confidence score to determine the current cumulative confidence score of each preset intent category; and controlling the smart device based on the comparison result of the current cumulative confidence score and a preset threshold.

[0005] Secondly, this application provides a control device for a smart device, comprising: The system comprises four modules: an acquisition module for acquiring multi-source heterogeneous data, wherein the multi-source heterogeneous data consists of heterogeneous sensing data from at least two different types of data sources in the data of smart devices associated with the user; a first processing module for unifying the time reference and / or spatial coordinates of the multi-source heterogeneous data to obtain time- and / or spatially aligned multi-source data; a second processing module for performing multimodal fusion processing on the multi-source data to generate a multimodal fusion feature vector; a third processing module for inputting the multimodal fusion feature vector into a pre-trained intent classifier and outputting the single recognition confidence score for each preset intent category at the current time; and a fourth processing module for combining the historical cumulative confidence score of each intent category at the previous time with the single recognition confidence score to determine the current cumulative confidence score of each preset intent category, and controlling the smart device based on the comparison result of the current cumulative confidence score and a preset threshold.

[0006] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the control method of the intelligent device described in the first aspect of this application.

[0007] Fourthly, this application also provides a storage medium storing computer-executable instructions for executing the control method of the intelligent device described in the first aspect of this application.

[0008] Compared with the prior art, the above-mentioned technical solution provided in this application has the following advantages: The method provided in this application first acquires multi-source heterogeneous data, then unifies the time reference and / or spatial coordinates of the multi-source heterogeneous data to obtain multi-source data after time and / or space alignment, and then performs multi-modal fusion processing on the multi-source data to generate multi-modal fusion feature vectors. Then, the multi-modal fusion feature vectors are input into a pre-trained intent classifier, which outputs the single recognition confidence of each preset intent category at the current time. Finally, the current cumulative confidence of each preset intent category is determined by combining the historical cumulative confidence and single recognition confidence of each intent category at the previous time, and the intelligent device is controlled according to the comparison result of the current cumulative confidence and the preset threshold. Therefore, this application achieves a comprehensive transformation from multi-source data acquisition, temporal and / or spatial alignment, heterogeneous graph attention fusion, single-recognition confidence output, confidence accumulation and update to threshold comparison control. It upgrades intent recognition and device control from the existing single instantaneous judgment to a progressive, robust decision-making process, improving the accuracy of intent recognition and significantly reducing the false trigger rate. This effectively solves the technical problem of high risk of false triggering and missed triggering in existing vehicle intent confirmation schemes. Attached Figure Description

[0009] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0012] Figure 1 A flowchart illustrating a control method for a smart device provided in an embodiment of this application; Figure 2 A schematic diagram of a multi-source heterogeneous graph provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a control device for a smart device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0015] To address the high risk of false triggering and missed triggering in existing vehicle intent confirmation schemes, this application provides a control method for intelligent devices, such as... Figure 1 As shown, the steps of this method include: Step 101: Obtain multi-source heterogeneous data, wherein the multi-source heterogeneous data is heterogeneous sensing data from at least two different types of data sources in the data of smart devices associated with the user. As can be seen, the multi-source heterogeneous data in this application embodiment refers to a collection of sensing data from various types of smart devices. Because these data originate from different types of smart devices, the data in the sensing data collection may differ in data structure, sampling frequency, physical meaning, and spatiotemporal reference. Specifically, these different types of smart devices may include user physiological data collected by wearable devices, user daily activity data recorded by mobile terminals, and vehicle operating environment data collected by vehicle sensors. It should be noted that these different types of smart devices are all user-associated devices; for example, a wearable device is a user's smartwatch, a mobile terminal is a user's mobile phone, and vehicle sensors refer to the vehicle sensors on the user's vehicle. Therefore, in specific application scenarios, this data collection may include the following data: 1) User physiological data collected by wearable devices, including but not limited to heart rate, heart rate variability, skin conductance, body temperature, and accelerometer data. This type of data reflects the user's physical state, emotional arousal, and activity level. It is characterized by high sampling frequency and strong temporal continuity, but lacks semantic information that is directly related to the external environment or user plans.

[0016] 2) User schedule activity data recorded by mobile terminals, including but not limited to timestamps, geographical locations, event types, and reminders for calendar events. This type of data reflects clues about users' time arrangements and behavioral intentions. It is characterized by rich semantic information and high degree of structure, but low update frequency and time sensitivity.

[0017] 3) Vehicle operating environment data collected by onboard sensors, including but not limited to GPS coordinates, vehicle speed, steering wheel angle, accelerator pedal status, and onboard camera images. This type of data reflects the vehicle's operating status and driving context, and is characterized by its clear physical meaning and direct correlation with driving safety.

[0018] The three types of data mentioned above belong to physiological signals, semantic information, and physical signals in terms of physical attributes, and their data formats and sampling periods are different, making them typical heterogeneous data. This application aims to cross-validate user intent from three dimensions—user's physical state, user's time arrangement, and vehicle operating environment—by simultaneously acquiring these three types of data, thereby overcoming the misjudgment problem caused by insufficient information from a single data source.

[0019] Step 102: Unify the time reference and / or spatial coordinates of the multi-source heterogeneous data to obtain multi-source data that is aligned in time and / or space; Unifying the time reference and / or spatial coordinates of multi-source heterogeneous data refers to unifying the timestamps of various heterogeneous data to the same time reference and unifying the spatial coordinates of various heterogeneous data to the same spatial coordinate system. It should be noted that not all data in multi-source heterogeneous data includes both timestamps and spatial coordinates. Therefore, for data with only timestamps, only time reference unification is needed; for data with only spatial coordinates, only spatial coordinate unification is needed. If data is directly fused without time and / or spatial alignment, discrepancies in timestamps and inconsistent representation of the same physical location in different coordinate systems will occur, leading to distorted fusion results. Specifically: 1) In terms of time, the original timestamps of each data source are converted into a unified timestamp based on Coordinated Universal Time (UTC), and time interpolation is performed on data sources with different sampling frequencies. For example, the heart rate data sampling frequency of wearable devices may be once per second, while the vehicle speed data sampling frequency of vehicle sensors may be 10 times per second. Through time interpolation, all data have aligned time scales within the same time window, ensuring that the data used for subsequent fusion belongs to the perception results of the same moment or the same very short time window.

[0020] 2) In terms of spatial dimension, the local coordinate system of each data source is determined. For example, accelerometer data from wearable devices is based on the wearer's own inertial coordinate system, positioning data from mobile terminals is based on the Global Navigation Satellite System coordinate system, and vehicle-mounted sensors are based on the vehicle's own coordinate system. A spatial transformation matrix is ​​used to convert the spatial coordinates in each local coordinate system into unified spatial coordinates in the world coordinate system. This spatial transformation matrix includes rotation and translation components, enabling precise mapping between different coordinate systems.

[0021] Through the above-mentioned time and / or spatial alignment processing, the multi-source data that were originally misaligned in time and / or space are unified under a common reference framework, providing an accurate data foundation for subsequent feature fusion and intent recognition.

[0022] Step 103: Perform multimodal fusion processing on the multi-source data to generate a multimodal fusion feature vector; In a specific example of this application's embodiments, a heterogeneous graph network can be constructed based on time- and / or spatially aligned multi-source data. A multi-head attention mechanism is then used to propagate cross-modal information between nodes in the heterogeneous graph network, generating a multi-modal fusion feature vector. Here, a heterogeneous graph network refers to a graph structure data model that includes multiple node types and multiple edge types. Its essential difference from existing homogeneous graphs (where all nodes and edges are of a single type) lies in the fact that heterogeneous graphs can explicitly distinguish the different semantic relationships carried by different types of nodes and edges, and allow for the allocation of different processing strategies to different types of relationships. Furthermore, the multi-head attention mechanism refers to running multiple independent attention computation processes (i.e., multiple heads) in parallel. Each attention head learns the information transfer weights between nodes from different feature subspaces or different association perspectives, and finally, the outputs of the multiple heads are combined. Therefore, using a multi-head attention mechanism to propagate cross-modal information between nodes in a heterogeneous graph network means connecting heterogeneous data (i.e., modalities) with different physical attributes and different semantic spaces through nodes in the graph structure, exchanging key feature information, thereby achieving knowledge complementarity and fusion.

[0023] Step 104: Input the multimodal fusion feature vector into the pre-trained intent classifier and output the single recognition confidence for each preset intent category at the current time. In this embodiment, the intent classifier is a pre-trained deep learning model. Its input is the aforementioned multimodal fusion feature vector, and its output is the recognition result for each preset intent category at the current time. The preset intent categories include, but are not limited to, parking intention, health abnormality intention, urgent appointment intention, long-distance rest intention, and in-vehicle microenvironment adjustment intention.

[0024] Furthermore, the single-shot recognition confidence in this embodiment is a probability value (between 0 and 1, summing to 1) calculated by the intent classifier for each intent category in each processing cycle based on the fusion features at the current moment. This represents the instantaneous judgment of the user's intent based solely on the current data. Specifically, the calculation method is as follows: the multimodal fusion feature vector is input into the fully connected layer of the intent classifier to obtain the original scores (logits) for each preset intent category; then, the original scores are normalized using a normalized exponential function (softmax function), mapping each original score to a probability distribution between 0 and 1, with a sum of 1. Each probability value in this probability distribution represents the single-shot recognition confidence for the corresponding intent category. Taking a scenario with three preset intent categories as an example: if the original scores output by the classifier are commuting intent: 2.5, rest intent: 1.0, and dining intent: -1.0, after softmax normalization, the corresponding single recognition confidence scores are approximately 0.80, 0.15, and 0.05, respectively, indicating that in the current instant, the classifier judges the user's intent as commuting with an 80% probability.

[0025] It is evident that the confidence level of a single recognition instance reflects the immediate judgment of the user's intent based on instantaneous data at the current moment. However, in complex environments such as in-vehicle systems, a single fluctuation in a user's breathing, a single minor adjustment of the steering wheel, or a single GPS signal drift can all cause instantaneous fluctuations in the confidence level of a single recognition instance, deviating from the true intent. If control operations are triggered directly based on the single recognition result, false triggering is highly likely due to the aforementioned unintended disturbances. Therefore, this application uses the confidence level of a single recognition instance as incremental evidence input into the subsequent confidence level accumulation stage, rather than directly as the basis for control operations.

[0026] Step 105: Combine the historical cumulative confidence and single recognition confidence of each intent category at the previous moment to determine the current cumulative confidence of each preset intent category, and control the smart device based on the comparison result of the current cumulative confidence and the preset threshold.

[0027] It should be noted that the smart device in this application embodiment can be a device associated with multi-source heterogeneous data, such as wearable devices, mobile terminals, and vehicles. Furthermore, to overcome the shortcomings of single-round recognition being susceptible to noise interference and having a high false trigger rate, this application introduces a confidence accumulation mechanism. This is because user intent typically exhibits a certain degree of continuity and stability, rather than randomly changing at every moment. Utilizing this characteristic, through the gradual accumulation of evidence over time in multiple rounds of recognition, single-round noise or instantaneous disturbances can be filtered out, making the final intent judgment more robust and reliable.

[0028] Furthermore, in this embodiment, a cumulative confidence value is maintained independently for each intent category, and this value is updated with the input of each round of recognition results. The historical cumulative confidence value refers to the confidence score accumulated by each intent category at the end of the previous processing cycle, which incorporates the combined contribution of all historical recognition evidence up to the previous moment. The current cumulative confidence value refers to the updated result in the current processing cycle, after adding the current single-time recognition confidence value as incremental evidence to the historical cumulative confidence value.

[0029] Through steps 101 to 105 above, multi-source heterogeneous data is first acquired. Then, the time reference and / or spatial coordinates of the multi-source heterogeneous data are unified to obtain multi-source data with time and / or spatial alignment. Subsequently, multi-modal fusion processing is performed on the multi-source data to generate multi-modal fusion feature vectors. Then, the multi-modal fusion feature vectors are input into a pre-trained intent classifier to output the single recognition confidence of each preset intent category at the current time. Finally, the historical cumulative confidence and single recognition confidence of each intent category at the previous time are combined to determine the current cumulative confidence of each preset intent category. And the smart device is controlled based on the comparison result of the current cumulative confidence and the preset threshold. Therefore, this application achieves a comprehensive transformation from multi-source data acquisition, temporal and / or spatial alignment, heterogeneous graph attention fusion, single-recognition confidence output, confidence accumulation and update to threshold comparison control. It upgrades intent recognition and device control from the existing single instantaneous judgment to a progressive, robust decision-making process, improving the accuracy of intent recognition and significantly reducing the false trigger rate. This effectively solves the technical problem of high risk of false triggering and missed triggering in existing vehicle intent confirmation schemes.

[0030] In an optional embodiment of this application, the method of determining the current cumulative confidence of each preset intent category by combining the historical cumulative confidence and single identification confidence of each intent category at the previous moment in step 105 may further include: Step 11: Obtain the time interval between the current processing cycle and the previous processing cycle; As can be seen, in this embodiment, multi-source data is continuously collected and intent is identified using a periodic or event-driven approach. Each complete process, from data collection, temporal and / or spatial alignment, heterogeneous graph fusion, intent classification to confidence update, is called a processing cycle. Therefore, the current processing cycle refers to the cycle in which the complete processing flow described above is currently being executed; the previous processing cycle refers to the cycle immediately preceding the current processing cycle. There is a time difference between two processing cycles on the time axis; this difference is the time interval.

[0031] In practical applications, the method for obtaining this time interval depends on the system's triggering mechanism. For systems with fixed-period triggering, the time interval is the preset sampling period. For example, if the system is set to execute a complete processing round every 5 seconds, then the time interval between each round and the previous round is always 5 seconds. For event-driven triggering systems, the time interval is the actual elapsed time between the completion of the previous round and the triggering time of the current round. This value may be irregular due to the randomness of event occurrences. For example, when abnormal fluctuations in physiological indicators are detected or the GPS location is close to a preset destination, the system may be triggered to execute a new round of processing at irregular time intervals. The purpose of obtaining this time interval is to determine the effectiveness of the historical cumulative confidence based on the time distance between two recognitions. If the interval between two recognitions is short, the historical evidence still has high reference value; if the interval is long, the scenario may have changed, and the reference value of the historical evidence decreases accordingly. Therefore, the time interval is a fundamental parameter for subsequent time decay processing.

[0032] Step 12: Perform time decay processing on the historical cumulative confidence of each intent category at the previous time according to the time interval, and add the single recognition confidence as an increment to the decayed historical cumulative confidence to obtain the current cumulative confidence of each intent category.

[0033] In the embodiments of this application, time decay processing refers to applying a decay effect to the historical cumulative confidence of each intent category at the previous moment according to the length of the time interval, so that its value decreases according to a certain rule, so as to reflect the objective law that the reference value of historical evidence at the current moment decreases with the passage of time.

[0034] In this embodiment, an exponential decay function is used to implement time decay processing. Specifically, the time decay processing is implemented in the following way:

[0035] in, Indicates the current processing time. Indicates the previous processing time. That is, the time interval; Indicates the category of intent at the previous moment. Historical cumulative confidence level; For Intent Category The corresponding preset decay coefficient is a real number greater than zero, and its value controls the decay rate of the historical cumulative confidence over time. This is the attenuation factor, whose value is between 0 and 1; This represents the historical cumulative value after time decay processing.

[0036] Among them, for the attenuation factor When the time interval When it approaches zero, the decay factor This indicates that the historical cumulative confidence level from the previous moment is fully preserved; when the time interval... As the value approaches infinity, the decay factor approaches 0, indicating that the historical cumulative confidence level has almost completely decayed. In actual operation, the time interval... Since it is a finite positive number, the decay factor is always between 0 and 1, thus achieving partial retention and partial decay of the historical cumulative confidence level.

[0037] Furthermore, a preset attenuation coefficient is used. The value of determines the rate of decay. The larger the value, the smaller the decay factor over the same time interval, and the faster the historical confidence decays. The smaller the value, the closer the decay factor is to 1, and the slower the historical confidence decays. This application allows different decay coefficients to be configured for different intention categories to adapt to the timeliness characteristics of different intentions. For example, the intention to attend an urgent appointment has a strong time urgency, and once the time window is missed, the intention is no longer valid. Therefore, a larger decay coefficient can be configured to make its confidence decay rapidly. The timeliness of the intention to take a long-distance rest is relatively relaxed, and a smaller decay coefficient can be configured to maintain the reference value of historical evidence for a longer period of time.

[0038] For example, the attenuation coefficient configured for a certain intent category. Second -1 If the time interval between the current processing cycle and the previous processing cycle Seconds, then decay factor That is, approximately 78% of the historical cumulative confidence level is retained, and approximately 22% is decayed. If the historical cumulative confidence level of this intention at the previous time step... The historical cumulative value after time decay is If the time interval is extended to Seconds, then decay factor The historical cumulative value after decay is This shows that the longer the time interval, the lower the proportion of historical evidence retained, and the greater the degree of forgetting of old evidence.

[0039] After completing the time decay processing of the historical cumulative confidence, this application uses the current single-identification confidence as new incremental evidence, and superimposes it with the decayed historical cumulative confidence to obtain the updated current cumulative confidence. Here, superposition refers to a numerical addition operation. (Based on intent category) For example, at the current moment Current cumulative confidence level The calculation method is as follows:

[0040] in, This refers to the historical cumulative value after the aforementioned time decay process. The output of the intent classifier for the intent category at the current moment The confidence level of a single identification (within the range of [0,1]).

[0041] It is evident that the degree of certainty regarding a user's intent consists of two parts: the first part is the existing degree of certainty accumulated from multiple historical identifications, which has undergone time decay adjustment (i.e., The second part is the added confidence contributed by the newly acquired instantaneous recognition result at the current moment (i.e., The two parts, when added together, form the system's overall level of confidence in the intention at the current moment.

[0042] It should be noted that the above overlay operation is performed independently for each intent category. That is, for the preset intent category... For each of these categories, the system maintains its own historical cumulative confidence score and performs decay and accumulation processing according to the formula described above. At any given time, the cumulative confidence scores of multiple intent categories may be updated simultaneously; that is, one category may increase due to continued recognition support, while another category may only undergo decay and a decrease in value due to a lack of new evidence. This parallel and independent update mechanism allows the system to simultaneously track the confidence evolution trajectories of multiple candidate intents, providing comprehensive data support for subsequent intent switching judgments and hierarchical control.

[0043] As can be seen, in this embodiment, historical information is not simply discarded (which would lose the cumulative effect and degenerate into single recognition) or completely relied upon (which would lead to response lag). Instead, the two are reasonably weighted and fused along the time dimension. This mechanism utilizes the cumulative effect of historical recognition in the time dimension to smooth instantaneous noise and improve system stability, while ensuring that the system maintains sufficient responsiveness to actual changes in user intent through time decay. Furthermore, the synergistic effect of time decay and incremental superposition forms a dynamic balance between unidirectional convergence and reversible correction. When the recognition results of multiple consecutive rounds all point to the same intent, the confidence of each round's single recognition is continuously superimposed on the ever-increasing historical cumulative value, causing the cumulative confidence to show an accelerated upward trend, and the system can quickly accumulate confidence. When the user intent actually changes, the cumulative confidence of the old intent only decays (the value gradually decreases) due to the lack of new evidence, while the cumulative confidence of the new intent is built from the current evidence (the value gradually increases). The transition between these two changes achieves a smooth transition in system judgment, rather than a drastic jump.

[0044] It should be noted that, in the embodiments of this application, the time decay processing is not limited to the aforementioned exponential decay method, but can also be implemented using linear decay or segmented decay methods. Linear decay refers to the historical cumulative confidence level decreasing uniformly according to a linear function as the time interval increases. Its mathematical form is: the decayed historical cumulative value equals the original historical cumulative value multiplied by a linear decay factor. This linear decay factor decreases linearly with the time interval between the current processing cycle and the previous processing cycle. When the time interval reaches the preset maximum effective interval, the decay factor drops to zero. Segmented decay refers to dividing the time interval into multiple intervals, using different decay rates in different intervals. For example, a slower decay rate is used in shorter time intervals to retain the reference value of recent historical evidence, while a faster decay rate is used in longer time intervals to accelerate the forgetting of outdated information. The advantage of segmented decay is that it allows for flexible configuration of the decay curve according to the timeliness characteristics of different intent categories. For example, for time-sensitive intent categories, a faster decay rate can be set in the early stages to quickly respond to changes in intent; for intent categories with less timeliness, a slower decay rate can be set in the early stages to maintain the continued influence of historical evidence. Regardless of whether linear decay, piecewise decay, or the aforementioned exponential decay is used, the core objective is to allow the historical accumulated confidence to fade reasonably over time, avoiding outdated information from continuously interfering with the current intent judgment. In practical applications, the system can choose any decay method according to specific scenario requirements, or configure different decay methods for different intent categories; this application does not impose any restrictions on this.

[0045] In an optional embodiment of this application, the method of performing time decay processing on the historical cumulative confidence of each intent category according to the time interval in step 12 above, and superimposing the single identification confidence as an increment with the decayed historical cumulative confidence to obtain the current cumulative confidence of each intent category, may further include: Step 21: Multiply the historical cumulative confidence of each intent category at the previous moment by the decay factor to obtain the decayed historical cumulative value. The decay factor is negatively exponentially related to the time interval, and the decay rate of the historical cumulative confidence over time is controlled by a preset decay coefficient. The historical cumulative value is obtained from the above. From the formula, we can see that:

[0046] Among them, the preset attenuation coefficient Used to control the rate of the aforementioned decay process. In the exponential function... middle, and They act together on the exponent in a product form. This means that the attenuation coefficient... It has a physical meaning of time scale scaling, that is, a larger... This is equivalent to the timeline being compressed, and historical information can be significantly attenuated within a short actual timeframe; smaller... This is equivalent to stretching the timeline, allowing historical information to maintain its reference value over a longer period. For example, if... From 0.05 seconds - ¹Increase to 0.10 seconds - ¹, then at the same time interval of 5 seconds, the decay factor from Become The retention rate of historical confidence decreased by approximately 17 percentage points; conversely, if... Reduced to 0.02 seconds - ¹, then the attenuation factor becomes The retention rate of historical confidence levels has significantly improved.

[0047] Step 22: Determine the corresponding quality weights based on the data quality of each data source at the current moment, and multiply the single identification confidence by the quality weights to obtain the weighted increment; In this embodiment, determining the quality weight is an independent calculation step. Its purpose is to quantify the overall reliability of the data provided by each data source at the current moment, thereby effectively adjusting the confidence level of a single identification. Specifically, the process of determining the quality weight is as follows: First, obtain the quality score of each data source at the current moment. The quality score of each data source includes at least two evaluation dimensions. The first dimension is the freshness score, which is used to characterize the time delay between the data acquisition time and the current processing time. That is, the shorter the delay, the more the data reflects the user's immediate state, and the higher the freshness score; the longer the delay, the more outdated the data may be, and the lower the freshness score. The second dimension is the reliability score, which is used to characterize the completeness and reliability of the data. That is, data sources with high signal strength, no data loss, and no obvious outliers receive a higher reliability score; data sources with weak signals, missing data, or obvious outliers receive a lower reliability score.

[0048] After obtaining the quality scores of each data source, the quality scores of each data source are multiplied by a preset weight coefficient corresponding to that data source, and then summed to obtain the quality weight. This is used to include wearable devices (index). ), mobile terminals (index) ) and vehicle sensors (index) Taking the implementation examples of three types of data sources as an example, the formula for calculating quality weight is:

[0049] in, , , The three types of data sources are at time... Quality rating; , , These are the preset weight coefficients for the corresponding data sources. Each weight coefficient is a positive number, and the sum of the three is 1. The allocation of weight coefficients reflects the prior importance of different data sources in the intent recognition task. For example, in recognizing the intent to keep an appointment urgently, the schedule data of the mobile terminal may be assigned the highest weight coefficient; in recognizing the intent to have health abnormalities, the physiological data of wearable devices may be assigned the highest weight coefficient.

[0050] It is worth emphasizing the importance of quality weight. This is a dynamic value that is recalculated with each processing cycle. This means that even if the intent classifier outputs the same single-recognition confidence score across different cycles, the actual contribution of this confidence score to the cumulative confidence score will dynamically adjust if the quality scores of different data sources change. For example, when a vehicle enters a tunnel and loses its GPS signal, the reliability score of the onboard sensor decreases, and its quality weight drops accordingly, thus weakening the contribution of the current round's recognition result to the cumulative confidence score and preventing misjudgments caused by anomalies in single-source data from contaminating the cumulative result.

[0051] By Intent Category For example, the formula for calculating the weighted increment is:

[0052] in, The output of the intent classifier for the intent category at the current moment The confidence level of a single identification. The product of the two values ​​represents the current time-to-time quality weight. This is the weighted increment.

[0053] It is evident that the initial judgment of a single identification, based on the current fused features output by the intent classifier, should be constrained by data quality factors in its actual contribution to confidence accumulation. When the signals from each data source are good and the data is complete, the quality weight approaches 1, and the initial judgment is almost fully adopted as valid evidence. When the quality of any data source declines due to poor signal, device disconnection, or data anomalies, the quality weight decreases accordingly, and the initial judgment is proportionally reduced before being included in the accumulation. Based on this, identification results generated by low-quality data will not pollute the system's accumulated judgment with equal weight, thus enabling the confidence accumulation process to adapt to data quality.

[0054] Step 23: Add the decayed historical cumulative value to the weighted increment to obtain the current cumulative confidence level of each intent category.

[0055] Combining the above calculation steps, we obtain the intent category. At the present moment The complete confidence update expression:

[0056] The formula consists of two terms: the first term The cumulative historical value after decay represents the residual contribution of historical identification evidence after decay over time; the second item The weighted increment represents the new contribution of newly identified evidence in the current round after data quality adjustments. The sum of the two values ​​is the current cumulative confidence level for this intent category at the current moment.

[0057] As can be seen, this formula encompasses the complete mathematical expression of the aforementioned time decay processing and incremental superposition operations, while further defining the mechanism by which quality weights adjust the incremental contribution. Through this formula, this application achieves a dual-weighted fusion of historical and new evidence across both the time and data quality dimensions.

[0058] The following example illustrates steps 21 to 23 above. Assume the system tracks commuting intention categories and presets an attenuation coefficient. Second - ¹, The processing cycle is fixed at one round every 5 seconds (i.e. Seconds, decay factor The historical cumulative confidence level of this intention in the previous round was [value missing]. .

[0059] First, calculate the historical cumulative value after decay: Then, the quality weights are determined: Assuming the wearable device has a good signal in the current round, and both its freshness and reliability scores are high, the overall quality score is [value missing]. The mobile device's calendar data is temporarily unreliable due to user-mandated modifications; the overall quality score is [insert score here]. The vehicle-mounted sensor signals are normal, and the overall quality score is [score missing]. The preset weighting coefficients are as follows: , , Then the quality weight is .

[0060] Furthermore, calculate the weighted increment: assuming the confidence level of the single-shot identification of "commuting" output by the intent classifier at the current moment is... The weighted increment is .

[0061] Calculate the current cumulative confidence level: .

[0062] In contrast, if all data sources are in optimal condition in this round ( Then the quality weight is The weighted increment is The current cumulative confidence level is Due to the decline in mobile terminal data quality, the current cumulative confidence level has decreased from the potential optimum of 2.02 to 1.812, a drop of 0.208. This difference reasonably reflects the negative impact of declining data quality on the accumulation of system confidence.

[0063] Therefore, through the above methods, the synergistic effect of quality weight and decay factor enables dual adaptive adjustment of confidence accumulation in both the time and data quality dimensions. The decay factor controls the natural fading of historical evidence over time, ensuring the system can track changes in user intent; the quality weight controls the reasonable weakening of current evidence due to data quality issues, ensuring the system is not interfered with by low-quality data. These two factors are independent and uncoupled, mathematically achieving decoupled control of time and quality factors. By multiplying the confidence of a single identification by the quality weight, this application achieves dynamic adjustment of the contribution of a single identification result. When the overall quality of multi-source data is high, the current identification result makes a sufficient update contribution to the accumulated confidence, and the system can quickly accumulate confidence; when the quality of any data source declines, the contribution of the current identification result is automatically weakened, and the system will not experience unnecessary confidence fluctuations due to a single low-quality input. This ensures that the confidence accumulation process remains robust even when data quality is unstable.

[0064] In an optional embodiment of this application, if the intent category identified at the current moment is different from the intent category identified at the previous moment, the method of this application embodiment may further include: Step 31: The historical cumulative confidence of the old intent category identified in the previous moment is attenuated according to the attenuation factor, and the weighted increment of the current moment is not added to the old intent category. In multiple consecutive processing cycles, each cycle outputs a recognition result for the current moment; that is, the intent category with the highest confidence in a single identification is used as the recognition result for the current moment. When this result is inconsistent with the recognition result of the previous moment, it indicates that a potential switch in user intent has been detected.

[0065] It should be noted that different intent categories refer to changes in the semantic category of the intent, not changes in the confidence level within the same intent category. For example, switching from commuting to rest, or from going to an appointment to finding parking, are different intent categories; however, if the confidence level of a single recognition for a commuting intent increases from 0.70 to 0.85, although the specific numerical value of the recognition result changes, the intent category remains commuting, which does not constitute an intent switch. Intent switch detection relies on comparing the recognition result at the previous moment with the recognition result at the current moment. When the two are inconsistent, it is determined that an intent switch event exists in the current processing cycle, triggering the special processing flow defined in steps 31 to 33.

[0066] In scenarios involving intent switching, if the conventional mode is still followed, the confidence levels of the new intent and the old intent will interfere with each other. That is, the old intent will maintain a high level due to the inertia of historical values ​​even without actual evidence, while the confidence level of the new intent will need to be accumulated slowly from zero. The two may compete near the threshold, leading to delays or fluctuations in system decision-making.

[0067] To address the aforementioned issues, the confidence update rules for scenario switching were further restricted. When a user's intent has switched to a new category, the current single-time recognition confidence is a judgment made for the new intent category, reflecting the intent direction pointed to by the current multi-source fusion features. Superimposing the current judgment result for the new intent onto the old intent category is semantically unreasonable; that is, the old intent category should not derive its confidence increment from information indicating other intents. Superimposing the single-time recognition confidence for the new intent onto the old intent would incorrectly inflate the old intent's confidence, contradicting the fact that the intent has switched, thus slowing down the fading of the old intent and interfering with the system's normal intent tracking.

[0068] For example, the system previously identified the intention as commuting, but now, due to a change in the user's route, it identifies it as dining. In this case, the current classifier output gives a higher confidence score for dining and a lower confidence score for commuting. Adding this lower confidence score to commuting, while contributing little, is still a positive increment and slows down the decay of commuting intentions. More seriously, if the high confidence score for dining is mistakenly added to commuting due to operational error, it will lead to serious misjudgment. By restricting the addition of the current weighted increment to older intention categories, this risk is eliminated at its source.

[0069] Step 32: Set the historical cumulative confidence of the new intent category identified at the current moment to zero, and set the current cumulative confidence of the new intent category to the weighted increment at the current moment; For new intent categories, firstly, the historical cumulative confidence score of the new intent category is set to zero. Since the new intent was not an intent category recognized by the system at the previous moment, its historical cumulative confidence score may be zero (if it has never been recognized before) or an old value that has decayed to a low level over time (if it was previously recognized but was not the current dominant intent). Regardless of its historical value, it is set to zero when the intent switch occurs to ensure that the accumulation of the new intent starts from a zero state and is not affected by any historical data.

[0070] Then, the current cumulative confidence level of this new intent category is set as the weighted increment at the current moment. That is:

[0071] Instead of calculating according to the conventional model .

[0072] Because switching intentions signifies a substantial change in the user's goal, the confidence in the new intention should be rebuilt from scratch, rather than additively accumulating the confidence levels of the old and new intentions. If the new intention is placed within the same cumulative framework as the old intention—that is, retaining any possible trace of historical data before adding the current increment—then in certain edge cases, the new intention might gain an unfair initial advantage due to these trace historical remnants, leading to premature intention switching in non-switching scenarios. By setting the historical cumulative confidence level to zero and starting the accumulation from the current weighted increment, this application ensures that the confidence level of the new intention is entirely based on current evidence, completely decoupling it from the historical accumulation of the old intention.

[0073] Let's continue with the example of switching from commuting to dining: (e.g., current time quality weight) Confidence of single-shot identification of new dining intentions The current cumulative confidence level of the new intention is If the value is low, it indicates that the conviction about dining intention is still in the initial stage and requires multiple rounds of continuous evidence to reach the threshold; if the value is close to or exceeds the threshold, it indicates that the current fusion features have a high degree of match with dining intention and can quickly complete the confirmation of intention. In either case, the starting point for the accumulation of new intentions is clear and unambiguous, and is not affected by old historical data.

[0074] Step 33: Restore the control state of the smart device associated with the old intent category to its initial state.

[0075] When an intent changes, the previously identified intent category may have already triggered a corresponding control state. For example, a first control signal (announcement-level service) may have been generated for the old intent, and the system may be in a state of outputting perception layer interaction information to the user (such as a car voice prompt asking "Do you want to navigate to the meeting room?", a mobile terminal displaying a parking recommendation pop-up, or a wearable device vibrating to remind the user). If the above interaction state is not cleared in time after the intent changes, the system will deliver outdated or even incorrect information to the user, causing confusion in the user experience.

[0076] The aforementioned problem is solved by restoring the control state of the smart device associated with the old intent category to its initial state. Restoring to the initial state means restoring all control states associated with the old intent to their initial state before triggering. Specifically, 1) the warning-level service state associated with the old intent is revoked. If the old intent has triggered the first control signal and is outputting perception layer interaction information, the output of that interaction information is immediately terminated. For example, if the in-vehicle voice is broadcasting an inquiry, the broadcast is interrupted; if the mobile terminal is displaying a prompt pop-up, the pop-up is closed; if the wearable device is vibrating, the vibration is stopped. 2) the execution-level service state associated with the old intent is revoked. If the old intent has triggered the second control signal and the system is executing or preparing to execute the corresponding action layer task scheduling, the tasks that have not yet been executed are canceled, and the partially executed operations are rolled back or compensated. For example, if a navigation route switch has been initiated but not yet completed, the switch operation is canceled, and the navigation state before the switch is restored. 3) the system identifier and cached data corresponding to the old intent control state are cleared. This includes, but is not limited to, clearing control state flags of old intentions, turning off timers assigned to them, and releasing associated memory resources, to ensure that the internal state of the system is consistent with the external control output.

[0077] Through the above operations, the residual control output of the old intent can be cleared in a timely manner when the intent switch occurs, avoiding the transmission of outdated information to the user or the execution of control operations that do not conform to the current intent, thereby ensuring the continuity of user experience and the security of system control.

[0078] Therefore, through the above method, the confidence levels of the old and new intentions are completely decoupled and do not contaminate each other. The old intention will not receive unreasonable increments due to the current recognition results of the new intention, and the new intention independently establishes its confidence level based entirely on the current evidence. The evolutionary trajectories of the two in the confidence space are clearly separated, and the decision boundary is well-defined. In addition, the decay rate of the old intention is controllable and predictable. Through the continuous effect of the decay factor on the historical cumulative confidence level of the old intention, the confidence level of the old intention decreases smoothly according to a unified mathematical model, without oscillations or lags caused by improperly added increments. This allows the system to forget the old intention at a certain rate after intention switching, making room for the rise of the new intention. Moreover, the starting point of the accumulation of the new intention is pure and unaffected by historical interference. Setting the historical cumulative confidence level of the new intention to zero and starting from the current weighted increment ensures that the confidence level of the new intention is entirely based on the credibility of the current evidence. This avoids the risk of premature switching due to trace historical remnants in the new intention, and also avoids the risk of delayed switching due to the incorrect transmission of historical values ​​of the old intention.

[0079] In an optional embodiment of this application, the method of determining the corresponding quality weight based on the data quality of each data source at the current moment in step 22 may further include: Step 41: Obtain the quality score of each data source at the current moment. The quality score includes a freshness score, which represents the timeliness of data collection, and a reliability score, which represents the completeness of data. In this embodiment, a freshness score is used to characterize the timeliness of data collection. Different data sources independently collect data and generate timestamps. There is a time delay between the moment the data is captured by the device's sensors and the moment the system processes it in the current processing cycle. The length of this delay determines the freshness of the data. The freshness score is a quantitative evaluation of this time delay. Specifically, the smaller the time difference between the data collection moment and the current processing moment, the better the data reflects the user's true state at the current moment, and the higher the freshness score; the larger the time difference, the more likely the data can no longer accurately reflect the user's immediate state, and the lower the freshness score. In practical implementation, the freshness score can be determined by a preset time threshold and a score mapping function. For example, when the time difference between the data collection moment and the current processing moment is less than a first preset threshold, the freshness score is set to a value close to 1; when the time difference exceeds a second preset threshold, the freshness score is set to a value close to 0; and cases in between are mapped to an intermediate value through linear interpolation or a piecewise function.

[0080] Therefore, the freshness characteristics of different data sources exhibit varying features due to differences in data acquisition mechanisms. Wearable devices operate through continuous monitoring, collecting physiological data (such as heart rate and skin conductance) at a fixed sampling frequency. The delay between the acquisition time and the system processing time is typically short, resulting in generally high freshness scores. Mobile terminal calendar data, however, has different characteristics. It originates from user manual input or synchronization with third-party systems, and the acquisition time is the moment the event is created or imported, which may be much earlier than the current processing time (e.g., a meeting calendar created a week ago is already of low timeliness in the current acquisition). It's important to note that the freshness score of mobile terminal calendar data is not entirely equivalent to the absolute time delay of the data. Semantic adjustments should also be made based on the temporal attributes of the calendar events. That is, future events that are far removed from the current time, although created earlier, still possess determinism and validity within the timeframe reflected in the calendar data. Vehicle sensor data (such as GPS coordinates and vehicle speed) are real-time sensing signals. The delay between the acquisition time and the system processing time depends on the sensor's sampling frequency and the transmission delay of the communication link, typically falling within an extremely short timescale.

[0081] Furthermore, reliability scoring is used to characterize the completeness and reliability of data. During signal acquisition and transmission, various interference factors can lead to a decline in data quality. These include measurement noise from the sensor itself, packet loss or delay in the communication link, and signal attenuation caused by loose equipment. All of these can cause the acquired data to deviate from the true value or contain missing data. Reliability scoring is a quantitative evaluation of the relative proportion of useful information and noise interference in the data, as well as the degree of data completeness. Specifically, the stronger the effective signal component, the weaker the noise component, and the fewer missing data points in the data, the higher the reliability score; the more severe the signal attenuation, the greater the noise interference, and the more missing or outlier data, the lower the reliability score. In practice, the criteria for determining reliability scoring vary depending on the type of data source. For wearable devices, reliability scores can be comprehensively determined by indicators such as the effective sampling rate of heart rate signals, the kurtosis characteristics of acceleration signals, and the baseline drift of electrodermal signals. When the wearer's vigorous exercise causes a significant baseline drift, the reliability score decreases; when poor contact between the device and the skin leads to substantial data loss, the reliability score decreases significantly.

[0082] For mobile terminals, the criteria for reliability scoring differ from those for physiological signal data. Mobile terminal calendar data is semantic information; its signal-to-noise ratio primarily reflects the degree of data structuring and semantic clarity, rather than traditional signal-to-noise ratio. For example, does the title of a calendar event contain clear location information? Does the event description include valid geolocation markers? Is the participant information complete? When calendar event descriptions are vague (e.g., only containing the word "meeting" without providing location and time details), the reliability score decreases accordingly; when the calendar data clearly contains conflicting events or duplicate entries, the reliability score decreases further.

[0083] For vehicle sensors, reliability scores can be determined by indicators such as the dilution factor of GPS signal positioning accuracy, the continuity of vehicle speed sensor signals, and the smoothness of steering wheel angle signals. When the GPS receiver fails to acquire enough satellites, leading to decreased positioning accuracy, the reliability score decreases; when the vehicle is in an environment where GPS signals are obstructed, such as a tunnel or underground parking lot, the reliability score decreases significantly.

[0084] In this specific example, assuming the system is currently determining if a user has an "urgent appointment," two key data points are retrieved: one is the "9:00 AM client meeting" schedule in the mobile device's calendar, created a week ago; the other is the current location coordinates output by the vehicle's GPS, collected in real-time. It should be noted that the quality score of a single data source is obtained by multiplying its freshness score by its reliability score. The reason for using multiplication instead of addition is that freshness and reliability are two necessary conditions for data validity. A low score in either dimension will significantly lower the overall quality score, reflecting the weakest link effect. That is, even if the data is very fresh, if its reliability is insufficient, its overall quality cannot reach a high level; conversely, even if the data is very reliable, if it is severely outdated, its reference value will also decrease significantly. Based on this, in the example above, for the calendar data, its content is complete and its semantics are clear, and its reliability score can reach 0.90. However, because seven days have passed since the collection time, its freshness score is only 0.20. The combined quality score is [insert score here]. While the GPS data was collected in real time and had a freshness score as high as 0.98, the user's vehicle had just left the underground parking garage, and the signal had not yet locked, causing severe location drift. The reliability score was only 0.40, and the quality score was... It is evident that the overall quality scores of both data sets are low; the former is weakened by its outdated nature, while the latter is dragged down by its unreliability. Ultimately, the system uses their respective quality scores as weights, multiplying them by the confidence level of the corresponding data source for a single identification of the "urgent appointment" intention. This ensures a reasonable adjustment of the contribution, meaning that even if the judgment result of low-quality data is itself reliable, it will not have an excessive pulling effect on the cumulative confidence level, thus preventing outdated information or momentary interference from contaminating the system's long-term judgment.

[0085] Step 42: Multiply the quality score of each data source by the preset weight coefficient of the corresponding data source and sum them to obtain the quality weight, wherein the sum of the preset weight coefficients is 1.

[0086] Taking wearable devices, mobile terminals, and vehicle sensors as three data sources as examples, the complete formula for calculating quality weights is as follows:

[0087] in, , , Wearable devices, mobile terminals, and vehicle sensors at different times The quality scores are all integrated with information from the corresponding freshness and reliability scores. , , These are the preset weight coefficients for the corresponding data sources. Each weight coefficient is a positive number and satisfies the following conditions: For example, in scenarios involving the identification of intentions related to health abnormalities, wearable device weighting coefficients... It can be set to a higher value (e.g., 0.6), the mobile terminal weighting coefficient. Set to a low value (e.g., 0.1), vehicle sensor weighting coefficient The value is set to an intermediate value (e.g., 0.3) to reflect the dominant role of physiological data in discriminating this intention. In the scenario of recognizing an urgent appointment intention, the mobile terminal weighting coefficient... It can be set to a higher value (such as 0.5) to highlight the importance of the schedule timeliness information.

[0088] Through steps 41 and 42 above, multi-dimensional evaluation ensures the comprehensiveness and accuracy of the quality score. Specifically, the freshness score and reliability score evaluate data quality from two independent dimensions: whether the data is timely and whether it is reliable. Both encompass the main factors affecting data validity, avoiding the bias that can result from a single-dimensional evaluation. If only freshness is evaluated while ignoring the signal-to-noise ratio, data that is new but heavily noisy will still be given high weight; conversely, if only the signal-to-noise ratio is evaluated while ignoring freshness, reliable but outdated data will still receive high weight. The combination of these two dimensions effectively solves the aforementioned problems. Furthermore, by configuring preset weight coefficients for different data sources, prior knowledge can be embedded into the calculation of quality weights. That is, different intent categories can be configured with different weight coefficient allocation schemes to highlight the dependence on specific data sources in specific intent recognition.

[0089] In an optional embodiment of this application, the method of controlling the smart device based on the comparison result of the current cumulative confidence level and the preset threshold involved in step 105 above may further include: Step 51: Compare the current cumulative confidence of each intent category with a first preset threshold and a second preset threshold, wherein the first preset threshold is less than the second preset threshold; Wherein, the first preset threshold is a lower threshold, denoted as The first preset threshold is set relatively low and is used to trigger the warning level control. The second preset threshold is a higher threshold, denoted as... Its value is set relatively high to trigger execution-level control. Both satisfy... The specific values ​​of the two thresholds can be calibrated and configured based on the application scenario, the security sensitivity of the intent category, and the system's tolerance for false triggering rate.

[0090] Step 52: When the current cumulative confidence level of any intent category is greater than or equal to a first preset threshold and less than a second preset threshold, a first control signal corresponding to the intent is generated and the smart device is controlled based on the first control signal; wherein, the first control signal is used to control the smart device to output perception layer interaction information, and the perception layer interaction information includes at least one of visual cues, auditory cues or tactile cues. In this embodiment, the first control signal is a warning-level control command, positioned as control at the information output level rather than at the physical action level. Specifically, the first control signal is used to control the smart device to output perception layer interactive information.

[0091] Among them, the perception layer interaction information refers to the prompting information that acts on the user's senses. Its function is to convey the system's predicted results to the user, so that the user is aware of the system's current intention recognition status, and at the same time provide the user with an opportunity to confirm or correct. The perception layer interaction information includes at least one of the following forms: visual prompts, such as flashing icons on the vehicle display screen, text prompts on the dashboard, pop-up notifications on mobile terminals, or changes in the dial interface of wearable devices; auditory prompts, such as in-vehicle voice inquiries, prompts on mobile terminals, or beeps on wearable devices; tactile prompts, such as vibration feedback from wearable devices.

[0092] The essential characteristics of the first control signal are non-intrusiveness and revocability. Non-intrusiveness means that the control signal's effect is limited to the human-machine interface and does not involve any substantial changes to the smart device's actuators, system operating parameters, or physical state. Revocability means that the perception layer interaction information triggered by the first control signal can be actively canceled or ignored by the user without having a lasting impact on the subsequent operating state of the smart device. This feature grants the user the right to confirm or deny the system's prediction during the warning phase, preventing the system from rashly executing substantial operations that could affect user experience or security without user confirmation.

[0093] Step 53: When the current cumulative confidence level is greater than or equal to the second preset threshold, generate a second control signal corresponding to the intention and control the smart device based on the second control signal. The second control signal is used to control the smart device to perform action layer task scheduling. Action layer task scheduling includes at least one of triggering the actuator in the smart device to change the physical state, or switching the system operation mode of the smart device, or sending an execution instruction to a third-party device.

[0094] The second control signal is an execution-level control command, positioned at the action execution level, directly driving the intelligent device to complete substantial task scheduling or state changes. The second control signal differs fundamentally from the first control signal: the first control signal only acts on the perception layer (information output) and does not change the device's physical state or operating mode; the second control signal acts on the action layer (task scheduling), directly triggering substantial changes in the intelligent device's actuators or system configuration.

[0095] Action layer task scheduling refers to the sequence of instructions that drives intelligent devices to complete substantive operations, and its specific forms include at least one of the following: 1) Triggering physical state changes in actuators within smart devices. This is the most typical operation in action-layer task scheduling, involving driving actuators within smart devices that have the ability to move or change their physical state. For example, in a vehicle, a second control signal can trigger a navigation system to switch routes, drive a seat adjustment motor to change seat posture, or control an air conditioning compressor to adjust the interior temperature; in wearable devices, a second control signal can trigger a vibration motor to enter a specific vibration mode or activate a photoelectric sensor's continuous monitoring mode.

[0096] 2) Switching the system operating mode of a smart device. This refers to changing the software operating state or system configuration parameters of a smart device. It does not involve the driving of physical actuators, but it is still a substantive operation. For example, the second control signal can control a mobile terminal to switch to Do Not Disturb mode, control a wearable device to switch from normal monitoring mode to emergency monitoring mode, or control an in-vehicle infotainment system to switch from the navigation interface to other function interfaces.

[0097] 3) Sending execution commands to third-party devices. This refers to controlling a smart device to send commands to other smart devices to achieve cross-device collaborative operation. For example, the second control signal can control a mobile terminal to automatically send status update messages (such as "the meeting will be 10 minutes late") to a preset contact, control a vehicle to upload driving data to a cloud server to trigger remote services, or control a wearable device to send an emergency help signal to an associated mobile terminal.

[0098] The three forms mentioned above are not mutually exclusive and can be used in combination within the same execution level control. For example, when the second control signal corresponds to a health abnormality intent, it can simultaneously control the vehicle navigation to switch to the nearest medical institution (first form), control the wearable device to switch to continuous heart rate monitoring mode (second form), and control the mobile terminal to send a help message to a preset emergency contact (third form).

[0099] As can be seen, the above boundary conditions ensure that all values ​​between the two thresholds have a clear classification. When the cumulative confidence level is exactly equal to the first preset threshold, it falls into the trigger range of the first control signal, generating a warning-level control; when the cumulative confidence level is exactly equal to the second preset threshold, it falls into the trigger range of the second control signal, generating an execution-level control. The two trigger ranges are continuously connected at the boundary without overlap or omission, ensuring that the control logic under any value has clear determinism. In addition, it should be noted that when the cumulative confidence level is less than the first preset threshold, neither the first nor the second control signal is generated, and the monitoring state of data acquisition, intent recognition, and confidence level updates continues until the cumulative confidence level reaches or exceeds the first preset threshold, or is decayed to a lower level by evidence.

[0100] Through steps 51 to 53 above, a two-level progressive control system of prediction and execution is achieved. Compared to the binary control mode triggered by a single threshold in the prior art, the dual-threshold architecture of this application enables the system to output only the perception layer interaction information when the accumulated confidence level reaches a low level of certainty, and to trigger the action layer task scheduling only when the accumulated confidence level reaches a high level of certainty. This hierarchical design ensures the system's proactive service capability while greatly reducing the negative impact of false triggering on user experience and driving safety. Moreover, the first control signal and the second control signal are fundamentally distinguished in terms of their level of action. The first control signal only acts on the human-machine interface, and its control content is information output, which does not involve any substantial change in the physical state or operating mode of the device, and is revocable; the second control signal directly drives the actuator, switches the system mode, or sends instructions to third-party devices, and its control content is substantial task scheduling. The clear distinction between the two ensures low intrusion of the prediction-level control at the execution level and high certainty of the execution-level control in terms of triggering conditions.

[0101] Furthermore, in a preferred embodiment of this application, the method may further include: constructing a historical intent evidence database, which stores identification evidence records for each preset intent category in historical processing cycles; each identification evidence record includes at least: a timestamp of the processing cycle, a single identification confidence level for the corresponding intent category, quality weights of each data source at the corresponding time, and a feature summary of the multimodal fusion feature vector on which the identification round is based. Moreover, after each processing cycle is completed, the single identification confidence level, quality weights, and timestamps of each intent category at the current time are stored as an identification evidence record in the historical intent evidence database, and expired evidence records are eliminated or compressed according to a preset storage period or storage capacity threshold.

[0102] Furthermore, in a preferred embodiment of this application, the method of this application embodiment further includes: establishing a user confirmation feedback path, wherein the user confirmation feedback path is used to receive confirmation or negative feedback from the user regarding the current intent recognition result and / or device control operation.

[0103] Based on this, after generating the first control signal or the second control signal, the method further includes: outputting confirmation query information through the human-computer interaction interface of the smart device, the confirmation query information being used to ask the user whether the current intent recognition result and / or device control operation is correct; receiving feedback signals input by the user through the human-computer interaction interface, the feedback signals including positive feedback or negative feedback; and adjusting the current cumulative confidence level and / or the model parameters of the intent classifier according to the feedback signals.

[0104] The adjustment of the current cumulative confidence level based on the feedback signal means that: when positive feedback is received, the current cumulative confidence level is adjusted upward by a first correction amount; when negative feedback is received, the current cumulative confidence level is adjusted downward by a second correction amount, and the intent category identified at the current moment is marked as a misidentified intent; wherein the first correction amount and the second correction amount are preset positive numbers, or dynamically determined based on the difference between the current cumulative confidence level and a preset threshold.

[0105] In optional embodiments of this application, the method of unifying the time reference and / or spatial coordinates of multi-source heterogeneous data involved in step 102 above to obtain time- / or spatially aligned multi-source data may further include: Step 61: Convert the original timestamps of the multi-source heterogeneous data into unified timestamps based on Coordinated Universal Time (UTC), and perform time interpolation on the multi-source heterogeneous data with different sampling frequencies so that the multi-source heterogeneous data have aligned time scales within the same time window. In scenarios involving the fusion of multi-source heterogeneous data, different data sources not only have inconsistent time bases (clock sources), but also different sampling frequencies. Heart rate data from wearable devices is typically collected at a frequency of 1 Hz (once per second), calendar data from mobile terminals is updated only when events are created or modified, and GPS data from vehicle sensors may be collected at a frequency of 5 Hz or 10 Hz. Without time alignment processing for these data with different sampling frequencies, the time points corresponding to the data from different data sources may differ significantly in any given processing cycle, leading to severe time misalignment when directly fused.

[0106] Based on this, the two specific operations of time alignment processing are: The first step is to convert the raw timestamps of the multi-source heterogeneous data into a unified timestamp based on Coordinated Universal Time (UTC). Each data source generates its own raw timestamp based on its device's built-in clock; wearable devices use their own crystal oscillator clock, mobile terminals use their system clock, and vehicle sensors use their GPS or vehicle bus clock. These clock sources may deviate from the UTC standard to varying degrees, and there may also be time differences between them. By uniformly converting all raw timestamps to the UTC reference, this application eliminates the time discrepancies caused by different clock sources, establishing time measurements from all data sources on a common reference frame.

[0107] The second operation involves time interpolation of multi-source heterogeneous data with different sampling frequencies, ensuring that the data has aligned time scales within the same time window. This operation resolves the issue of inconsistent sampling frequencies. Specifically, for a given target processing time... If the sampling time of a certain data source is exactly 1 If the data source is in [a specific location], then the sampled value at that moment is used directly; if the data source is [a specific location], then [the sampled value at that moment is used directly]. If no sampled value is found at a given time (due to low sampling frequency or non-overlapping sampling times), then the data source is analyzed using the data source at... Interpolate the actual sampled values ​​at the previous and next time points to obtain The estimated value of the time.

[0108] For example, the heart rate data sampling frequency of wearable devices is 1 Hz, and the sampling times are respectively Seconds; the GPS data sampling frequency of the vehicle sensors is 5 Hz, and the sampling times are respectively Seconds. System processing cycle. In the second calculation, GPS data has a sampled value at 0.5 seconds, which can be used directly; however, heart rate data has no sampled value at 0.5 seconds. Therefore, an estimate for 0.5 seconds is obtained by linear interpolation of the heart rate values ​​at 0.0 seconds and 1.0 seconds (if the heart rate is 80 beats / min at 0.0 seconds and 84 beats / min at 1.0 seconds, the estimated value for 0.5 seconds is 82 beats / min). Through this time interpolation process, all data sources are processed at the same time. Both can provide data values ​​on aligned time scales, providing time-consistent data input for subsequent feature fusion.

[0109] Through the two time alignment operations described above, multi-source data that were originally out of sync and had different frequencies were unified to a common time axis, eliminating the data misalignment problem caused by time deviation and sampling frequency differences.

[0110] Step 62: Determine the local coordinate system of the multi-source heterogeneous data, and convert the spatial coordinates under each local coordinate system into unified spatial coordinates under the world coordinate system through a spatial transformation matrix. The spatial transformation matrix includes rotation and translation components.

[0111] After time alignment is completed, spatial alignment is required, which involves determining the local coordinate system of the multi-source heterogeneous data and converting the spatial coordinates in each local coordinate system into a unified spatial coordinate system in the world coordinate system using a spatial transformation matrix.

[0112] Different data sources use different coordinate systems when collecting spatial data. Accelerometer data collected by wearable devices is based on the device's own inertial coordinate system, whose origin is usually located within the device's internal sensor chip. The direction of the coordinate axes is determined by the device's wearing orientation and changes with the wearer's posture. Positioning data from mobile terminals (such as Global Navigation Satellite System coordinates) is based on the Global Navigation Satellite System coordinate system, whose reference system is either a geocentric coordinate system or a latitude-longitude coordinate system. While it has a globally unified benchmark, its original expression is a geographical location represented by latitude, longitude, and altitude. Data collected by vehicle-mounted sensors (such as target object positions and lane line information) is based on the vehicle's own coordinate system. Its origin is usually located at the vehicle's center of mass or a fixed reference point on the vehicle. The x-axis points in the vehicle's forward direction, the y-axis points laterally, and the z-axis points perpendicularly to the vehicle, changing with the vehicle's position and orientation.

[0113] The origin, axial direction, and scale of the three coordinate systems mentioned above are all different. Directly comparing or fusing spatial coordinates from different systems will result in a fundamental spatial misalignment. Therefore, spatial alignment is necessary to unify the coordinates from all local coordinate systems to the same world coordinate system.

[0114] The choice of world coordinate system is not limited to a single fixed form and can be flexibly determined according to the application scenario. In vehicle-road cooperative scenarios, a geocentric coordinate system can be chosen as the unified world coordinate system; in vehicle-local scenarios, a world coordinate system defined based on the vehicle's position and orientation at startup can be chosen; in cross-device cooperative scenarios, a station-centric coordinate system with the user's current location as the origin can be chosen. Regardless of the world coordinate system chosen, the spatial coordinates of all data sources are based on the same coordinate system origin and the same coordinate axis direction after transformation.

[0115] Specifically, for any point in space, its homogeneous coordinates in a certain local coordinate system Through spatial transformation matrix Transformed into homogeneous coordinates in the world coordinate system The transformation relationship is as follows: Spatial transformation matrix The complete form is Homogeneous transformation matrix:

[0116] in, The rotation component (rotation matrix) is used to describe the pose difference between the local coordinate system and the world coordinate system, and to achieve rotational alignment of the coordinate axes. The translation component (translation vector) is used to describe the positional offset of the local coordinate system origin relative to the world coordinate system origin, thereby achieving translational alignment of the coordinate origin.

[0117] In addition, the rotation component and the translation component The specific values ​​can be determined through methods such as sensor calibration, relative pose estimation between devices, or coordinate registration based on known reference points. For vehicle sensors, their rotation and translation relative to the world coordinate system can be obtained through a vehicle positioning and attitude estimation system; for wearable devices, their rotation and translation relative to the user's body or relative to the vehicle can be estimated through the device's built-in inertial navigation algorithm; for mobile terminals, their Global Navigation Satellite System coordinates are themselves expressed in the world coordinate system, and only the coordinate format conversion needs to be considered during the conversion process.

[0118] Through steps 61 and 62 above, time alignment processing resolves the time misalignment problem caused by differences in clock sources and sampling frequencies among multi-source data. By unifying timestamp conversion and interpolation, each data source can provide time-aligned data values ​​at every processing moment, eliminating fusion feature distortion caused by time asynchrony and enabling subsequent heterogeneous graph attention fusion to be based on time-consistent data. Spatial alignment processing unifies the local coordinates of different devices to the same world coordinate system through rotation and translation components, fundamentally eliminating spatial misalignment caused by inconsistent coordinate systems. After spatial alignment, data originally belonging to different device spatial reference systems—such as the user's arm movement direction detected by wearable devices, destination coordinates recorded by mobile terminals, and road environment information detected by vehicle sensors—can be measured and compared spatially within the same world coordinate system.

[0119] In an optional embodiment of this application, the method of performing multimodal fusion processing on multi-source data to generate a multimodal fusion feature vector in step 103 above may further include: Step 71: Construct the corresponding node types in the heterogeneous graph network based on the different data source types in the multi-source data; In heterogeneous graph networks, nodes are the basic building blocks of the graph structure, with each node carrying the feature vector of its corresponding data source at the current moment. Constructing node types based on data source type means that data from the same data source type is mapped to nodes of the same type, and data from different types of data sources is mapped to nodes of different types. Specifically, physiological data collected by wearable devices (including heart rate, heart rate variability, skin conductance, body temperature, accelerometer data, etc.) are uniformly mapped to wearable device nodes; daily activity data recorded by mobile terminals (including timestamps, geographic location, event type, etc. of calendar events) are uniformly mapped to mobile terminal nodes; and vehicle operating environment data collected by vehicle sensors (including GPS coordinates, vehicle speed, steering wheel angle, accelerator pedal status, etc.) are uniformly mapped to vehicle sensor nodes.

[0120] The feature vectors of different types of nodes originate from different feature spaces, possessing different physical dimensions and data distribution characteristics. For example, the features of wearable device nodes are physiological time-series signals, the features of mobile terminal nodes are discrete semantic event information, and the features of vehicle sensor nodes are continuous physical quantity signals. In heterogeneous graph networks, these nodes are explicitly labeled as different types, enabling subsequent multi-head attention mechanisms to learn independent transformation matrices for different node and edge types, thereby capturing the differentiated correlation patterns between different types of data.

[0121] Step 72: For the data corresponding to each node within the same processing cycle, if the timestamp difference between any two nodes is less than a preset time threshold and / or the spatial location difference is less than a preset distance threshold, then an edge connection is established between any two nodes. The timestamp difference refers to the absolute difference between the collection times of the data corresponding to two nodes. After time alignment, the timestamps of all data sources are unified to the same time base, so the timestamps of different nodes can be directly compared on a common time axis. If the timestamp difference between two nodes is less than a preset time threshold, it indicates that the two data are close in time and may be data generated simultaneously or nearly simultaneously by the same user activity or in the same scenario. If the timestamp difference is too large, it indicates that the two data belong to different time segments, the scenario may have changed, and there is no reasonable basis for establishing a correlation.

[0122] Spatial location difference refers to the Euclidean distance or other metric distance between the spatial coordinates of two nodes. After spatial alignment, the spatial coordinates of different data sources are unified to the same world coordinate system, so the spatial coordinates of different nodes can be measured on the same spatial reference. If the spatial location difference between two nodes is less than a preset distance threshold, it indicates that the two data are spatially adjacent and may reflect activities in the same physical location or within the same spatial range. If the spatial distance is too large, it indicates that the two data reflect different spatial locations and lack a spatial basis for establishing a connection. Taking the distance between a user's mobile terminal (usually carried on the person or close to the user) and a vehicle as an example, if the spatial distance between the user's mobile terminal and the vehicle is less than 100m, it indicates that the user is more likely to use the vehicle; if the spatial distance exceeds 200m, it indicates that the user is less likely to use the vehicle. Of course, the above is just an example. In different application scenarios, the settings can be adjusted according to actual needs. The above only illustrates that the smaller the spatial distance, the closer the connection between the two, that is, there is a basis to build two different devices into different nodes with a relationship in a heterogeneous network.

[0123] Furthermore, the preset time threshold and preset distance threshold are configurable parameters preset by the system. Their specific values ​​can be set according to the spatiotemporal resolution requirements of the application scenario and the actual characteristics of the data source. For example, for vehicle application scenarios, the preset time threshold can be set to a value within the range of 1 to 5 seconds to cover a reasonable time window for the same user activity; the preset distance threshold can be set to a value within the range of 5 to 50 meters to cover a reasonable spatial range for the same vehicle or the same user activity. Based on time alignment and spatial alignment, the setting of the above thresholds gives the establishment of edge connections a clear physical meaning, that is, different source data within the same time window and the same spatial range are considered to belong to the same spatiotemporal context, and there is a necessity for information transmission and fusion between them. It can be seen that the edge structure of the heterogeneous graph network automatically adjusts with the dynamic changes in time and space. When a data source is spatially far away from other data sources (such as when a user leaves their mobile terminal at home and only carries a wearable device while driving), the spatial distance between the mobile terminal node and the vehicle sensor node exceeds the preset distance threshold, the edge connection between the two is broken, and information no longer propagates between them; when the data source returns to a nearby location, the edge connection is automatically restored.

[0124] It should be noted that establishing an edge connection between any two nodes means prioritizing the establishment of an edge connection if the difference in timestamps of the corresponding data between any two nodes is less than a preset time threshold and / or the difference in spatial location is less than a preset distance threshold. For data sources where accurate spatial coordinates cannot be obtained, edge connections can be established solely based on timestamp differences, but these edges should be assigned a lower base weight. Conversely, a similar approach applies to data sources where accurate timestamps cannot be obtained.

[0125] Step 73: Construct the edge type in the heterogeneous graph network based on the node type and the corresponding edge connection, and construct the corresponding heterogeneous graph network based on the node type, edge connection and edge type.

[0126] It's important to note that edge connection and edge type are two related but distinct concepts in the construction of heterogeneous graph networks. Edge connection determines whether a relationship exists between two nodes in the current spatiotemporal context. This is based on spatiotemporal proximity, specifically whether the difference in data acquisition times between the two nodes is less than a preset time threshold, and / or whether the difference in spatial location is less than a preset distance threshold. If these conditions are met, an edge connection is established between the nodes; otherwise, it is not. Edge type, on the other hand, determines the specific semantic category of the association between two connected nodes. This is based on the types of the nodes at both ends of the connection. For example, ... Figure 2As shown, the edges between wearable device nodes and mobile terminal nodes are categorized as the first edge type, the edges between mobile terminal nodes and vehicle sensor nodes are categorized as the second edge type, and the edges between wearable device nodes and vehicle sensor nodes are categorized as the third edge type. Specifically, the first edge type between wearable device nodes and mobile terminal nodes has a semantic category of association between physiological state and time arrangement, used to capture the consistency between the user's physical state and their schedule; the second edge type between mobile terminal nodes and vehicle sensor nodes has a semantic category of association between driving behavior and destination, used to capture the degree of matching between the vehicle's trajectory and the user's planned location; and the third edge type between wearable device nodes and vehicle sensor nodes has a semantic category of association between physiological stress and driving environment, used to capture the interaction between the user's physical response and the vehicle's driving state.

[0127] By constructing the aforementioned heterogeneous graph network, this application can explicitly model the dynamic temporal and spatial relationships between multi-source heterogeneous data, providing a structured graph data foundation for subsequent cross-modal information dissemination.

[0128] Step 74: Use the multi-head attention mechanism to propagate cross-modal information between nodes in the heterogeneous graph network and generate multi-modal fusion feature vectors. Cross-modal information propagation means that in the heterogeneous graph network, each node transmits its own feature vector to its neighboring nodes along the established edge connection and receives feature vectors from the neighboring nodes, so that the feature vectors of each node are fused with the feature vectors of other types of nodes that are connected by edges.

[0129] As can be seen, in this embodiment, cross-modal information propagation refers to the process in a heterogeneous graph network where each node transmits its own feature vector to its neighboring nodes along the established edges, and receives feature vectors from neighboring nodes, so that the feature vectors of each node are fused with the feature vectors of other types of nodes connected by edges. For ease of understanding, each node in the heterogeneous graph network can be regarded as an information processing unit, which not only tells the data source features it represents to other nodes connected by edges, but also obtains the feature information represented by other nodes. Through this bidirectional transmission and fusion process along edges, data information that was originally separated in different modal spaces can flow and converge in the network.

[0130] Taking a vehicle sensor node as an example, this node carries vehicle operating status features such as current GPS coordinates, vehicle speed, and steering wheel angle. When there is an edge connection between this node and a wearable device node, the wearable device node transmits its physiological feature vectors, such as heart rate and skin conductance, to the vehicle sensor node, while the vehicle sensor node also transmits its vehicle status features to the wearable device node. Similarly, when there is an edge connection between the vehicle sensor node and a mobile terminal node, a similar bidirectional feature transmission occurs between them. After this propagation process, the feature vector of the vehicle sensor node no longer only contains the original vehicle operating status information, but also integrates the physiological status information from the wearable device node and the schedule activity information from the mobile terminal node, becoming a fused feature vector that integrates multi-source heterogeneous data information. Similarly, the wearable device node and the mobile terminal node also receive information from other nodes, realizing cross-modal updates of their respective features. Through the mutual propagation and fusion between nodes in each processing cycle, all nodes in the entire heterogeneous graph network can obtain enhanced feature representations that integrate information from other modalities, providing input with comprehensive multi-source evidence for the subsequent intent classifier.

[0131] In an optional embodiment of this application, the method of using a multi-head attention mechanism to propagate cross-modal information between nodes in a heterogeneous graph network and generate a multimodal fusion feature vector in step 103 above may further include: Step 81: Assign a corresponding attention weight matrix to each edge type; In this application, the heterogeneous graph network contains multiple edge types, including at least a first edge type between wearable device nodes and mobile terminal nodes, a second edge type between mobile terminal nodes and vehicle sensor nodes, and a third edge type between wearable device nodes and vehicle sensor nodes. Semantically, these three edge types correspond to physiological-schedule associations, schedule-vehicle associations, and physiological-vehicle associations, respectively, and their cross-modal interaction patterns are fundamentally different. If a uniform feature transformation parameter is applied to different edge types, the unique statistical regularities and semantic connotations of each association type cannot be fully captured. Therefore, this application assigns a separate set of attention weight matrices to each edge type. The attention weight matrix is ​​a linear transformation parameter used to map the node feature vectors from the original feature space to the feature space required for attention calculation; its value is automatically learned through an optimization algorithm during model training. A first attention weight matrix is ​​assigned to the first edge type for attention calculation between wearable device nodes and mobile terminal nodes; a second attention weight matrix is ​​assigned to the second edge type for attention calculation between mobile terminal nodes and vehicle sensor nodes; and a third attention weight matrix is ​​assigned to the third edge type for attention calculation between wearable device nodes and vehicle sensor nodes. The three sets of weight matrices are independent of each other and do not affect each other. Each set learns the optimal feature mapping strategy for its corresponding edge type during training. It should be noted that the "multi-head" mechanism here refers to further configuring multiple independent attention weight matrices for parallel computation under the same edge type. Each attention head focuses on different relation patterns from different feature subspaces. Finally, the outputs of multiple heads are combined to enhance the model's feature representation ability and generalization performance.

[0132] Step 82: For any node in the heterogeneous graph network, aggregate the feature vectors of the neighboring nodes under each edge type according to each edge type, and calculate the attention contribution coefficient of each neighboring node relative to any node through the attention weight matrix corresponding to the edge type; where the attention contribution coefficient represents the importance measure of the features of the corresponding neighboring node in the information fusion of any node. After assigning the attention weight matrix, for any node in the heterogeneous graph network, its neighborhood information is processed according to the edge type. In a heterogeneous graph network, a node is typically connected to only a subset of nodes by edges; these connected nodes are called the node's neighboring nodes. Specifically, for a given edge type, firstly, the feature vectors of all neighboring nodes under that edge type are collected; then, using the attention weight matrix corresponding to that edge type, a linear transformation is performed on the feature vectors of these neighboring nodes and the node's feature vector; finally, the attention contribution coefficient of each neighboring node relative to the node is calculated using an attention mechanism.

[0133] The attention contribution coefficient is a value between 0 and 1. Its physical meaning is the importance of a neighboring node's feature information to the node's feature update within the association pattern represented by the current edge type. A larger attention contribution coefficient indicates a higher weight of the neighboring node's features in the node's information fusion, meaning a greater contribution from the neighboring information to the node; a smaller coefficient indicates a weaker influence of the neighboring node's features on the node.

[0134] Step 83: Under the same edge type, the feature vectors of each neighboring node are weighted and summed according to the corresponding attention contribution coefficients to obtain the initial fusion features under that edge type. Under the same edge type, after obtaining the attention contribution coefficients of each neighboring node relative to the current node, the system performs a weighted summation of the feature vectors of each neighboring node according to their respective attention contribution coefficients. That is, the larger the contribution coefficient of a neighboring node, the higher the proportion of its feature in the summation result; the smaller the contribution coefficient of a neighboring node, the lower the proportion of its feature in the summation result. After this weighted summation operation, all neighborhood information of the node under this edge type is aggregated into a single vector, called the initial fused feature under this edge type. This initial fused feature integrates the information of all neighboring nodes of the node under the same edge type, and its content reflects the most valuable part of the neighborhood information for the node under the cross-modal association mode represented by the current edge type.

[0135] To address this, taking a vehicle sensor node as an example, this node may simultaneously have a third-side connection (physiological-vehicle) with a wearable device node and a second-side connection (schedule-vehicle) with a mobile terminal node. Under the second-side connection, schedule features of all mobile terminal nodes connected to the vehicle sensor node are collected, and the attention contribution coefficient of each mobile terminal node relative to the vehicle sensor node is calculated. These are then weighted and summed to obtain the initial fusion feature under the schedule-vehicle side connection, which integrates the contributions of all relevant schedule information to the vehicle's operational status judgment. Under the third-side connection, physiological features of all wearable device nodes connected to the vehicle sensor node are collected, and the attention contribution coefficient of each wearable device node relative to the vehicle sensor node is calculated. These are then weighted and summed to obtain the initial fusion feature under the physiological-vehicle side connection, which integrates the contributions of all physiological information to the vehicle's operational status judgment. The two types of initial fusion features are independently calculated using different attention weight matrices, each reflecting the differentiated influence of different modalities on the node.

[0136] Step 84: Concatenate or average the multiple initial fusion features obtained for any node under different edge types to obtain the multimodal fusion feature vector of any node.

[0137] As can be seen, the same node obtains multiple initial fusion features under different edge types associated with it. Each initial fusion feature represents the representation of the node after fusing neighborhood information under a cross-modal association mode. In order to integrate these information from different edge types into a unified multimodal fusion feature vector, the system concatenates or averages the multiple initial fusion features.

[0138] The concatenation operation involves joining multiple initial fused feature vectors end-to-end along their respective feature dimensions to form a new, higher-dimensional feature vector. The advantage of concatenation is that it preserves the complete information of the initial fused features for each edge type, without losing any dimensional details, allowing the subsequent intent classifier to obtain the richest feature representation. The averaging operation calculates the arithmetic mean of multiple initial fused feature vectors along their corresponding dimensions, forming a feature vector with the same dimension as a single initial fused feature. The advantage of averaging is that it does not increase the feature dimension, effectively controlling computational complexity, and is suitable for applications sensitive to feature dimensionality.

[0139] Through steps 81 to 84 above, the multi-head attention mechanism is realized for cross-modal information propagation and the generation of multi-modal fusion feature vectors among nodes in heterogeneous graph networks. First, an independent attention weight matrix is ​​assigned to each edge type, enabling the model to learn the optimal feature mapping strategy for cross-modal associations of different semantic categories, avoiding the coarse processing of different association patterns by a uniform transformation. Second, attention contribution coefficients are calculated separately for each edge type and normalized within each category, ensuring that different edge types do not interfere with each other during information aggregation, and that key neighborhood features within each category are specifically strengthened. Finally, the initial fusion features of multiple edge types are combined through concatenation or averaging, allowing the output feature vector of any node to simultaneously incorporate complementary information from multiple data source types, providing a rich, cross-validated, high-quality input for subsequent intent classification.

[0140] In an optional embodiment of this application, the method of inputting the multimodal fusion feature vector into a pre-trained intent classifier and outputting the single recognition confidence score for each preset intent category at the current time in step 104 may further include: Step 91: Input the multimodal fusion feature vector into the fully connected layer of the intent classifier to obtain the original scores of each preset intent category; A fully connected layer is a fundamental layer structure in neural networks that establishes a learnable linear connection between each dimension of the input feature vector and each dimension of the output vector. The operation of a fully connected layer is essentially a linear transformation: it multiplies the input multimodal fusion feature vector by a predefined weight matrix and adds a bias vector to obtain an output vector with the same number of dimensions as the intent categories. Each element in this output vector is the original score for the corresponding predefined intent category.

[0141] Furthermore, the raw score refers to the unnormalized real-valued score. Each preset intent category is assigned a raw score, which has no fixed numerical range and can be positive, negative, or zero, with its absolute value being arbitrarily large. The raw score reflects the intent classifier's bias towards each intent category based on the current multimodal fusion feature vector: a higher score indicates that the intent classifier believes the current multimodal fusion feature vector matches that intent category well; a lower score indicates a lower match. However, the raw score is unnormalized, and while the score differences between different intent categories have relative comparative significance, the absolute value of a single score does not have an intuitive probabilistic interpretation. For example, the score... and It can only indicate that the classifier is more inclined towards category 1 than category 2, but it cannot directly indicate the percentage of inclination towards category 1.

[0142] Step 92: Normalize the original scores using a normalized exponential function, mapping each original score to a probability distribution between 0 and 1 with a sum of 1, and using each probability value in the probability distribution as the single recognition confidence of the corresponding intent category.

[0143] The process involves first calculating an exponential value with a base of the natural constant for each raw score based on a normalized exponential function; then, summing the exponential values ​​for all intent categories; and finally, dividing the exponential value for each intent category by this sum to obtain the corresponding probability value. Through this calculation, the probability values ​​for all intent categories are mapped to a range of 0 to 1, with the sum of all probability values ​​being 1. After obtaining the probability distribution, the probability value corresponding to each intent category is used as the single-recognition confidence level for that intent category. The single-recognition confidence level reflects the instantaneous degree of certainty the intent classifier has about that intent category at the current moment. In subsequent processing, each single-recognition confidence level is input as incremental evidence into the confidence accumulation update module, and after time decay and quality weighting, it is superimposed onto the historical cumulative confidence level of each intent category.

[0144] Therefore, the fully connected layer, as a combination of linear transformation and the normalized exponential function, as a nonlinear normalization, constitutes an end-to-end trainable intent classification module. Through optimization of the weight matrix and bias vector during the training phase, the system can automatically learn the mapping relationship from the feature space to the semantic space of intent categories from a large number of multimodal fused feature vector samples. This allows the classifier's output confidence to truly reflect the semantic matching degree between the feature vector and each intent category, providing accurate and reliable single-recognition input for subsequent confidence accumulation. Furthermore, based on the normalized exponential function, the original score in the unbounded real number domain can be transformed into a probability distribution in the bounded interval of 0 to 1, giving the single-recognition confidence a clear probabilistic semantic, i.e., the relative probability that the user's intent belongs to that category at the current moment. This numerical form allows it to be directly superimposed on the historical accumulated confidence as incremental evidence with consistent dimensions, without the need for additional numerical scaling or mapping, simplifying the mathematical expression of confidence accumulation calculation.

[0145] Corresponding to Figure 1 The method embodiments shown in this application also provide a control device for a smart device, such as... Figure 3 As shown, the device includes: The acquisition module 302 is used to acquire multi-source heterogeneous data, wherein the multi-source heterogeneous data is heterogeneous sensing data from at least two different types of data sources in the data of smart devices associated with the user. The first processing module 304 is used to unify the time reference and / or spatial coordinates of multi-source heterogeneous data to obtain multi-source data that is aligned in time and / or space. The second processing module 306 is used to perform multimodal fusion processing on multi-source data to generate a multimodal fusion feature vector; The third processing module 308 is used to input the multimodal fusion feature vector into the pre-trained intent classifier and output the single recognition confidence of each preset intent category at the current time. The fourth processing module 310 is used to combine the historical cumulative confidence and single recognition confidence of each intent category at the previous moment to determine the current cumulative confidence of each preset intent category, and to control the smart device based on the comparison result of the current cumulative confidence and the preset threshold.

[0146] In an optional embodiment of this application, the fourth processing module may further include: The acquisition unit is used to acquire the time interval between the current processing cycle and the previous processing cycle; The first processing unit is used to perform time decay processing on the historical cumulative confidence of each intent category at the previous time according to the time interval, and to add the single recognition confidence as an increment to the decayed historical cumulative confidence to obtain the current cumulative confidence of each intent category.

[0147] In an optional embodiment of this application, the first processing unit in this application embodiment may further include: The first processing subunit is used to multiply the historical cumulative confidence of each intent category at the previous moment by the decay factor to obtain the decayed historical cumulative value. The decay factor is negatively exponentially related to the time interval, and the decay rate of the historical cumulative confidence over time is controlled by a preset decay coefficient. The second processing subunit is used to determine the corresponding quality weights based on the data quality of each data source at the current time, and multiply the single identification confidence by the quality weights to obtain the weighted increment. The third processing subunit is used to add the decayed historical cumulative value to the weighted increment to obtain the current cumulative confidence level of each intent category.

[0148] In an optional embodiment of this application, if the intent category identified at the current moment is different from the intent category identified at the previous moment, the apparatus in this application embodiment further includes: The fifth processing module is used to attenuate the historical cumulative confidence of the old intent category identified in the previous moment according to the attenuation factor, and does not add the weighted increment of the current moment to the old intent category; The sixth processing module is used to set the historical cumulative confidence of the new intent category identified at the current moment to zero, and set the current cumulative confidence of the new intent category to the weighted increment at the current moment; The reset module is used to restore the control state of smart devices associated with old intent categories to their initial state.

[0149] In an optional embodiment of this application, the second processing subunit is used to perform the following steps: Obtain the quality score of each data source at the current moment. The quality score includes a freshness score, which represents the timeliness of data collection, and a reliability score, which represents the completeness of data. The quality scores of each data source are multiplied by the corresponding preset weight coefficients and then summed to obtain the quality weights, where the sum of the preset weight coefficients is 1.

[0150] In an optional embodiment of this application, the fourth processing module includes: The comparison unit is used to compare the current cumulative confidence of each intent category with a first preset threshold and a second preset threshold, wherein the first preset threshold is less than the second preset threshold; The second processing unit is configured to generate a first control signal corresponding to the intent and control the smart device based on the first control signal when the current cumulative confidence level of any intent category is greater than or equal to a first preset threshold and less than a second preset threshold; wherein the first control signal is used to control the smart device to output perception layer interaction information, and the perception layer interaction information includes at least one of visual cues, auditory cues, or tactile cues. The third processing unit is used to generate a second control signal corresponding to the intent when the current cumulative confidence level is greater than or equal to the second preset threshold, and to control the smart device based on the second control signal. The second control signal is used to control the smart device to perform action layer task scheduling. The action layer task scheduling includes at least one of triggering the actuator in the smart device to change the physical state, or switching the system operation mode of the smart device, or sending an execution instruction to a third-party device.

[0151] In an optional embodiment of this application, the first processing module in this application includes: The fourth processing unit is used to convert the original timestamps of multi-source heterogeneous data into unified timestamps based on Coordinated Universal Time, and to perform time interpolation processing on multi-source heterogeneous data with different sampling frequencies, so that the multi-source heterogeneous data have aligned time scales within the same time window. The fifth processing unit is used to determine the local coordinate system of multi-source heterogeneous data and convert the spatial coordinates under each local coordinate system into unified spatial coordinates under the world coordinate system through a spatial transformation matrix. The spatial transformation matrix includes rotation and translation components.

[0152] In an optional embodiment of this application, the second processing module includes: The first building unit is used to construct the corresponding node type in the heterogeneous graph network based on different data source types in multi-source data. The establishment unit is used to establish an edge connection between any two nodes if the timestamp difference between the data corresponding to each node in the same processing cycle is less than a preset time threshold and / or the spatial location difference is less than a preset distance threshold. The second building unit is used to construct the edge type in the heterogeneous graph network based on the node type and the corresponding edge connection, and to construct the corresponding heterogeneous graph network by the node type, edge connection and edge type; The generation unit is used to propagate cross-modal information between nodes in a heterogeneous graph network using a multi-head attention mechanism, generating multimodal fusion feature vectors. Cross-modal information propagation refers to the process in a heterogeneous graph network where each node transmits its own feature vector to its neighboring nodes along the established edge connections, and receives feature vectors from neighboring nodes, so that the feature vectors of each node can be fused with the feature vectors of other types of nodes that are connected by edges.

[0153] In an optional embodiment of this application, the second processing module includes: The allocation unit is used to assign a corresponding attention weight matrix to each edge type; The sixth processing unit is used to aggregate the feature vectors of the neighboring nodes under each edge type for any node in the heterogeneous graph network, and calculate the attention contribution coefficient of each neighboring node relative to any node through the attention weight matrix corresponding to the edge type; where the attention contribution coefficient represents the importance measure of the features of the corresponding neighboring node in the information fusion of any node. The seventh processing unit is used to weight and sum the feature vectors of each neighboring node according to the corresponding attention contribution coefficient under the same edge type to obtain the initial fusion feature under that edge type. The eighth processing unit is used to concatenate or average the multiple initial fusion features obtained by any node under different edge types to obtain the multimodal fusion feature vector of any node.

[0154] In an optional embodiment of this application, the third processing module includes: The ninth processing unit is used to input the multimodal fusion feature vector into the fully connected layer of the intent classifier to obtain the original score of each preset intent category; The tenth processing unit is used to normalize the original scores using a normalized exponential function, mapping each original score to a probability distribution between 0 and 1 with a sum of 1, and using each probability value in the probability distribution as the single recognition confidence of the corresponding intent category.

[0155] like Figure 4 As shown in the figure, this application provides an electronic device, including a processor 411, a communication interface 412, a memory 413, and a communication bus 414, wherein the processor 411, the communication interface 412, and the memory 413 communicate with each other through the communication bus 414. Memory 413 is used to store computer programs; In one embodiment of this application, when the processor 411 executes the program stored in the memory 413, it implements the control method of the smart device provided in any of the foregoing method embodiments, and its function is similar, so it will not be described again here.

[0156] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the control method for an intelligent device as provided in any of the foregoing method embodiments.

[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0159] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0160] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A control method for an intelligent device, characterized in that, include: Acquire multi-source heterogeneous data, wherein the multi-source heterogeneous data is heterogeneous sensing data from at least two different types of data sources in the data of smart devices associated with the user; The time reference and / or spatial coordinates of the multi-source heterogeneous data are unified to obtain multi-source data that is aligned in time and / or space. The multi-source data is subjected to multimodal fusion processing to generate a multimodal fusion feature vector; The multimodal fusion feature vector is input into a pre-trained intent classifier, which outputs the single recognition confidence for each preset intent category at the current time. By combining the historical cumulative confidence of each intent category at the previous moment with the single recognition confidence, the current cumulative confidence of each preset intent category is determined, and the smart device is controlled based on the comparison result between the current cumulative confidence and the preset threshold. The process of performing multimodal fusion processing on the multi-source data to generate a multimodal fusion feature vector includes: Based on the different data source types in the multi-source data, construct the corresponding node types in the heterogeneous graph network; For the data corresponding to each node within the same processing cycle, if the timestamp difference between any two nodes is less than a preset time threshold and / or the spatial location difference is less than a preset distance threshold, then an edge connection is established between the two nodes. The edge types in the heterogeneous graph network are constructed based on the node types and corresponding edge connections, and the corresponding heterogeneous graph network is constructed based on the node types, the edge connections, and the edge types. A multi-head attention mechanism is used to propagate cross-modal information among nodes in the heterogeneous graph network and generate multi-modal fusion feature vectors. The cross-modal information propagation refers to the following: in the heterogeneous graph network, each node transmits its own feature vector to its neighboring nodes along the established edge connection and receives feature vectors from the neighboring nodes, so that the feature vectors of each node are fused with the feature vectors of other types of nodes that are connected by edges. Specifically, the multi-head attention mechanism is used to propagate cross-modal information among nodes in the heterogeneous graph network, generating a multimodal fusion feature vector, including: Assign a corresponding attention weight matrix to each edge type; For any node in the heterogeneous graph network, the feature vectors of the neighboring nodes under each edge type are aggregated according to each edge type, and the attention contribution coefficient of each neighboring node relative to the given node is calculated using the attention weight matrix corresponding to the edge type; wherein, the attention contribution coefficient represents the importance measure of the features of the corresponding neighboring node in the information fusion of the given node. Under the same edge type, the feature vectors of each neighboring node are weighted and summed according to the corresponding attention contribution coefficients to obtain the initial fusion features under that edge type; The multiple initial fusion features obtained by any node under different edge types are concatenated or averaged to obtain the multimodal fusion feature vector of any node.

2. The method according to claim 1, characterized in that, Combining the historical cumulative confidence of each intent category at the previous moment with the single recognition confidence, the current cumulative confidence of each preset intent category is determined, including: Get the time interval between the current processing cycle and the previous processing cycle; The historical cumulative confidence of each intent category at the previous time step is time-decayed according to the time interval, and the single recognition confidence is added to the decayed historical cumulative confidence as an increment to obtain the current cumulative confidence of each intent category.

3. The method according to claim 2, characterized in that, The historical cumulative confidence scores of each intent category at the previous time step are time-decayed according to the time interval, and the single recognition confidence score is added to the decayed historical cumulative confidence scores as an increment to obtain the current cumulative confidence scores of each intent category, including: The historical cumulative confidence of each intent category at the previous moment is multiplied by the decay factor to obtain the decayed historical cumulative value. The decay factor is negatively exponentially correlated with the time interval, and the decay rate of the historical cumulative confidence over time is controlled by a preset decay coefficient. The corresponding quality weights are determined based on the data quality of each data source at the current moment, and the single identification confidence is multiplied by the quality weights to obtain the weighted increment; The current cumulative confidence level for each intent category is obtained by adding the decayed historical cumulative value to the weighted increment.

4. The method according to claim 3, characterized in that, If the intent category identified at the current moment is different from the intent category identified at the previous moment, the method further includes: The historical cumulative confidence of the old intent category identified in the previous moment is attenuated according to the attenuation factor, and the weighted increment of the current moment is not added to the old intent category; Set the historical cumulative confidence of the new intent category identified at the current moment to zero, and set the current cumulative confidence of the new intent category to the weighted increment at the current moment; Restore the control state of the smart device associated with the old intent category to its initial state.

5. The method according to claim 3, characterized in that, The corresponding quality weights are determined based on the data quality of each data source at the current moment, including: Obtain the quality score of each data source at the current moment, wherein the quality score includes a freshness score that represents the timeliness of data collection and a reliability score that represents the completeness of data; The quality score of each data source is multiplied by the preset weight coefficient of the corresponding data source and then summed to obtain the quality weight, wherein the sum of the preset weight coefficients is 1.

6. The method according to claim 1, characterized in that, Controlling the smart device based on a comparison between the current cumulative confidence level and a preset threshold includes: The current cumulative confidence level of each intent category is compared with a first preset threshold and a second preset threshold, wherein the first preset threshold is less than the second preset threshold; If the current cumulative confidence level of any intent category is greater than or equal to the first preset threshold and less than the second preset threshold, a first control signal corresponding to the intent is generated and the smart device is controlled based on the first control signal; wherein, the first control signal is used to control the smart device to output perception layer interaction information, and the perception layer interaction information includes at least one of visual cues, auditory cues or tactile cues. When the current cumulative confidence level is greater than or equal to the second preset threshold, a second control signal corresponding to the intention is generated and the smart device is controlled based on the second control signal. The second control signal is used to control the smart device to perform action layer task scheduling. The action layer task scheduling includes at least one of triggering the actuator in the smart device to change the physical state, or switching the system operation mode of the smart device, or sending an execution instruction to a third-party device.

7. The method according to claim 1, characterized in that, Unifying the time reference and / or spatial coordinates of the multi-source heterogeneous data to obtain time- and / or spatially aligned multi-source data includes: The original timestamps of the multi-source heterogeneous data are converted into unified timestamps based on Coordinated Universal Time (UTC), and time interpolation is performed on the multi-source heterogeneous data with different sampling frequencies so that the multi-source heterogeneous data have aligned time scales within the same time window. The local coordinate system of the multi-source heterogeneous data is determined, and the spatial coordinates under each local coordinate system are converted into unified spatial coordinates under the world coordinate system through a spatial transformation matrix, wherein the spatial transformation matrix includes rotation components and translation components.

8. The method according to claim 1, characterized in that, The multimodal fused feature vector is input into a pre-trained intent classifier, and the output of the single recognition confidence for each preset intent category at the current time includes: The multimodal fusion feature vector is input into the fully connected layer of the intent classifier to obtain the original score of each preset intent category; The original scores are normalized by a normalized exponential function, mapping each original score to a probability distribution between 0 and 1 with a sum of 1, and each probability value in the probability distribution is used as the single recognition confidence of the corresponding intent category.

9. A control device for an intelligent device, characterized in that, include: The acquisition module is used to acquire multi-source heterogeneous data, wherein the multi-source heterogeneous data is heterogeneous sensing data from at least two different types of data sources in the data of smart devices associated with the user; The first processing module is used to unify the time reference and / or spatial coordinates of the multi-source heterogeneous data to obtain multi-source data that is aligned in time and / or space. The second processing module is used to perform multimodal fusion processing on the multi-source data to generate a multimodal fusion feature vector; The third processing module is used to input the multimodal fusion feature vector into a pre-trained intent classifier and output the single recognition confidence for each preset intent category at the current time. The fourth processing module is used to combine the historical cumulative confidence of each intent category at the previous moment with the single recognition confidence to determine the current cumulative confidence of each preset intent category, and to control the smart device based on the comparison result of the current cumulative confidence with the preset threshold. The second processing module includes: The first building unit is used to construct the corresponding node type in the heterogeneous graph network based on different data source types in multi-source data; The establishment unit is used to establish an edge connection between any two nodes if the timestamp difference between the data corresponding to each node in the same processing cycle is less than a preset time threshold and / or the spatial location difference is less than a preset distance threshold. The second building unit is used to construct the edge type in the heterogeneous graph network based on the node type and the corresponding edge connection, and to construct the corresponding heterogeneous graph network by the node type, edge connection and edge type; The generation unit is used to propagate cross-modal information between nodes in a heterogeneous graph network using a multi-head attention mechanism, generating multi-modal fused feature vectors. Cross-modal information propagation refers to the process in a heterogeneous graph network where each node transmits its own feature vector to its neighboring nodes along the established edges and receives feature vectors from neighboring nodes, so that the feature vectors of each node can be fused with the feature vectors of other types of nodes that are connected by edges. The second processing module further includes: The allocation unit is used to assign a corresponding attention weight matrix to each edge type; The sixth processing unit is used to aggregate the feature vectors of the neighboring nodes under each edge type for any node in the heterogeneous graph network, and calculate the attention contribution coefficient of each neighboring node relative to any node through the attention weight matrix corresponding to the edge type; where the attention contribution coefficient represents the importance measure of the features of the corresponding neighboring node in the information fusion of any node. The seventh processing unit is used to weight and sum the feature vectors of each neighboring node according to the corresponding attention contribution coefficient under the same edge type to obtain the initial fusion feature under that edge type. The eighth processing unit is used to concatenate or average the multiple initial fusion features obtained by any node under different edge types to obtain the multimodal fusion feature vector of any node.

10. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. The memory is used to store computer programs; the processor is used to execute the computer programs to implement the control method of the intelligent device according to any one of claims 1-8.

11. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the control method of the intelligent device according to any one of claims 1-8.

Images