Intelligent device control method, electronic device, readable storage medium and computer program product

Abstract

The application discloses an intelligent device control method, an electronic device, a readable storage medium and a computer program product, and belongs to the technical field of intelligent device control. The method comprises the following steps: acquiring a plurality of sensing data of a first mode based on FTTR technology; the first mode corresponds to any mode in a plurality of modes, and the mode indicates the data type of the sensing data; performing intention recognition on the plurality of sensing data of the first mode, and determining a semantic label corresponding to each sensing data; determining a semantic conflict degree of the first mode based on the semantic label corresponding to each sensing data; the semantic conflict degree is used for representing the frequency of switching between contradictory semantics of the semantic label corresponding to each sensing data; it is determined that the first mode has a semantic conflict according to the semantic conflict degree, and sending of a control instruction to the intelligent device corresponding to the first mode is stopped.

Description

Technical Field

[0001] This application relates to the field of intelligent device control technology, and in particular to an intelligent device control method, electronic device, readable storage medium, and computer program product. Background Technology

[0002] Currently, relevant intelligent device control systems, such as smart device control systems, generally adopt a single-modal (e.g., voice, vision, environmental perception) or multi-modal simple weighted fusion strategy for environmental control. This makes it difficult to handle complex scenarios such as concurrent user intents and semantic contradictions between modalities (e.g., users repeatedly issuing contradictory control intents within a short period, such as continuously alternating commands to turn lights on / off or heat / cool). In particular, when semantic contradictions occur within a short period, the system continues to blindly output control commands, directly leading to frequent start-stop of smart home devices, shortened device lifespan, increased energy consumption, and chaotic control logic. Summary of the Invention

[0003] This application provides a smart device control method, electronic device, readable storage medium, and computer program product, which can solve the problem that related smart device control systems cannot recognize semantic conflicts, causing the system to continue outputting instructions during semantic conflicts, resulting in frequent start-stop of smart devices, increased energy consumption, and chaotic execution.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows: Firstly, a method for controlling an intelligent device is provided, including: Acquire multiple sensing data of a first modality; the first modality corresponds to any one of multiple modalities, and the modality indicates the data type of the sensing data; Intent recognition is performed on multiple perceptual data of the first modality to determine the semantic label corresponding to each perceptual data. The semantic conflict level of the first modality is determined based on the semantic labels corresponding to each perceptual data; the semantic conflict level is used to indicate the frequency with which the semantic labels corresponding to each perceptual data switch between contradictory semantics; If a semantic conflict is determined based on the degree of semantic conflict, control commands are stopped from being sent to the smart device corresponding to the first modality.

[0005] In a second aspect, an electronic device is provided, comprising a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-described intelligent device control method.

[0006] Thirdly, a readable storage medium is provided, wherein at least one computer program is stored in the readable storage medium, the computer program being loaded and executed by a processor to implement the above-described intelligent device control method.

[0007] Fourthly, a computer program product is provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor to implement the intelligent device control method provided in the various optional implementations described above.

[0008] The intelligent device control method, electronic device, readable storage medium, and computer program product provided in this application can quantify semantic conflicts based on the frequency of contradictory switching of semantic tags, achieve accurate conflict detection, and actively freeze conflict commands when it is determined that there is a semantic conflict in the modality, thereby avoiding frequent device start-up and shutdown, reducing energy consumption, extending device life, and improving usage safety.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0011] Figure 1 A flowchart of an exemplary embodiment of the present application is shown; Figure 2 A flowchart illustrating another exemplary embodiment of this application provides a method for controlling a smart device; Figure 3 A flowchart illustrating the total energy evolution of a particle system provided in an exemplary embodiment of this application is shown. Figure 4 A flowchart illustrating particle iteration convergence and decision selection provided in an exemplary embodiment of this application is shown. Figure 5 A flowchart of conflict polarization coding provided in an exemplary embodiment of this application is shown; Figure 6 A block diagram of an intelligent device control apparatus provided in an exemplary embodiment of this application is shown; Figure 7 A structural block diagram of an electronic device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0013] To address the problem that related intelligent device control systems cannot recognize semantic conflicts, causing the system to continue outputting commands during semantic conflicts, resulting in frequent start-stop of intelligent devices, increased energy consumption, and chaotic execution, this application provides an intelligent device control method.

[0014] The subject executing this method can be an edge computing node, a local control host, a cloud server, or an application (APP) running on the aforementioned devices; this application does not limit this.

[0015] For example, this method can be run on an intelligent device control system with multimodal data acquisition and edge computing capabilities. The overall environment includes three parts: hardware environment, software environment and network environment. (1) The hardware environment includes a multimodal acquisition module, an edge computing device, and local and cloud storage devices. In this embodiment, the multimodal acquisition module can be deployed in each room or device edge node to collect perception data such as voice, image, temperature, and infrared body sensing; the edge computing device is an embedded computing device used for local real-time data processing and semantic conflict analysis; the storage device includes a local solid state drive (SSD) and a cloud object storage service (OSS). (2) The software environment adopts embedded operating systems such as Linux, which supports multi-threading and real-time processing; it uses communication protocols such as HyperText Transfer Protocol (HTTP) / HyperText Transfer Protocol Secure (HTTPS), WebSockets (WS), and Message Queuing Telemetry Transport (MQTT) to realize data transmission between devices; it supports cloud-based real-time streaming media service (RTMP) and message queue (MQ) services. (3) The network environment, including both LAN and WAN, is based on Fiber To The Room (FTTR) technology to ensure low-latency transmission; the network uplink bandwidth is not less than 10Mbps, and the network latency is less than 100 milliseconds (ms).

[0016] In some embodiments, the smart device can be, but is not limited to, home devices, elderly care devices, security devices, in-vehicle devices, office devices, industrial devices, etc., and the method can be applied to, but is not limited to, the following application scenarios: (1) Smart home control: control of multimodal devices in the living room, bedroom, study and other homes; (2) Smart elderly care: health monitoring of the elderly and handling of conflicts between voice and physiological data; (3) Intelligent security: Door and window sensors are linked with cameras to prevent false alarms; (4) In-vehicle intelligent cockpit: conflict arbitration of voice commands, fatigue detection, and body sensing data; (5) Intelligent office: Multimodal control of meeting rooms and concurrent instruction processing for multiple users; (6) Industrial Internet of Things: Industrial equipment status monitoring, sensor oscillation and fault handling.

[0017] Based on the above description, the following will combine Figures 1 to 5 The intelligent device control method provided in the embodiments of this application will be explained and described in detail. These embodiments are only used to explain this application and do not constitute a limitation on this application.

[0018] In some embodiments, this application provides a smart device control method. Figure 1 A flowchart illustrating a smart device control method according to an exemplary embodiment of this application is shown. Figure 1 As shown, the intelligent device control method mainly includes the following steps (S101-S104): S101. Acquire multiple sensing data of the first mode; In some embodiments, the first modality is any one of multiple modalities. In this embodiment, the modality indicates the data type of the perceived data. Exemplary data types include voice type, visual type, infrared body temperature type, ambient temperature type, light type, etc., and may also include other data types of perceived data used for intelligent device control, which are not limited in this application.

[0019] In some embodiments, sensor data of the corresponding type is collected by sensors corresponding to each modality; for example, voice data is collected by a microphone, visual / behavioral data is collected by a camera, ambient temperature data is collected by a temperature sensor, and body surface sensory data is collected by an infrared sensor. Of course, other sensors may also be included, which are not limited in this application.

[0020] For example, the same voice modality continuously collects multiple voice data such as "turn on the light", "turn off the light", and "turn on the light" within 3 seconds; the same temperature modality continuously collects multiple ambient temperature data such as 25℃, 26℃, and 24℃.

[0021] In this embodiment, multi-channel sensing data corresponding to each modality can be collected, providing a data foundation for subsequent intent recognition and conflict judgment for any modality.

[0022] S102. Perform intent recognition on multiple perceptual data of the first modality and determine the semantic label corresponding to each perceptual data. In this embodiment, the perceived data is converted into standardized user operation intentions, i.e., semantic tags, which can be used for semantic conflict judgment. In some embodiments, semantic tags are set based on the user's operation intentions on the smart device; for example, semantic tags include at least one of the following: temperature adjustment, lighting control, security alarm control, ambient atmosphere control, and device operation mode control. Of course, other smart device control intentions may also be included, which are not limited in this application. In this embodiment, temperature adjustment includes, for example, heating, cooling, and ventilation; lighting control includes, for example, turning on the light, turning off the light, and adjusting the brightness; security alarm control includes, for example, arming, disarming, and intrusion alarm; ambient atmosphere control includes, for example, entertainment, silent, and sleep; and device operation mode control includes, for example, energy-saving mode, standard mode, and powerful mode. For example, a multi-dimensional semantic space can be established, such as an 8-dimensional (D=8) semantic space, including: heating tendency, cooling tendency, ventilation tendency, turning off the light tendency, turning on the light tendency, security alarm tendency, entertainment and relaxation tendency, and silent tendency. This example of a multi-dimensional semantic space is only illustrative and does not limit this application.

[0023] In some embodiments, performing intent recognition on multiple perceptual data of the first modality and determining the semantic label corresponding to each perceptual data includes: performing user operation intent recognition on the multiple perceptual data of the first modality respectively, and mapping each perceptual data to a corresponding semantic label based on intent matching rules.

[0024] In this embodiment, user intent recognition refers to mapping raw perceptual data such as voice, image, and temperature to control intents that the device can execute; for example, the voice "cold" is mapped to temperature adjustment, the camera recognizes "reading" as lighting control, and the infrared sensor detects "high temperature" as temperature adjustment. For example, the intent matching rule can be a preset correspondence between perceptual data and intent, or it can be a mapping relationship model learned based on machine learning models, deep learning models, or large language models. Correspondingly, the semantic space on which the semantic tags are based can be a preset fixed semantic space, or it can be a learnable, updatable, and scalable dynamic semantic space. This application does not limit the construction method or update form of the semantic space. For example, if the voice "too hot" is collected, it is mapped to the semantic tag "cooling / cooling down"; as another example, if the voice "turn on the light" is collected, it is mapped to the semantic tag "lighting control / turn on the light"; if the camera collects the user's behavior as "sleeping," it is mapped to the semantic tags "lighting control, ambient atmosphere control / turn off the light, mute."

[0025] This embodiment can unify multi-source heterogeneous perception data into standard semantic tags, providing a standardized data foundation for the standardization and quantification of subsequent semantic conflict judgment.

[0026] S103. Determine the degree of semantic conflict of the first modality based on the semantic labels corresponding to each perceptual data. In some embodiments, the degree of semantic conflict is used to represent the frequency with which the semantic labels corresponding to each piece of perceived data switch between contradictory semantics. In this embodiment, the degree of semantic conflict of a modality is determined by performing semantic conflict analysis on the semantic labels corresponding to each piece of perceived data in the same modality within a preset time window.

[0027] In some embodiments, determining the semantic conflict level of a first modality based on the semantic labels corresponding to each perceptual data includes: calculating the semantic oscillation entropy corresponding to the first modality based on the feature distribution of contradictory semantic switching occurring within a preset time interval for the semantic labels corresponding to each perceptual data. In this embodiment, the target semantic oscillation entropy is calculated based on the feature distribution of contradictory semantic switching within a preset time interval, enabling the semantic conflict level to be quantified, comparable, and definable, thus solving the problems of ambiguous and unexplainable conflict determination.

[0028] In this embodiment, semantic oscillation entropy is used to represent the degree of semantic conflict in the first modality. In some embodiments, semantic oscillation entropy includes a first semantic oscillation entropy and / or a second semantic oscillation entropy. The first semantic oscillation entropy represents the degree of temporal semantic oscillation fluctuation in a single modality, while the second semantic oscillation entropy represents the degree of influence of intermodal oscillation linkages. In this embodiment, the degree of semantic conflict in the first modality can be comprehensively analyzed from both the perspective of modal independence and intermodal dependence.

[0029] In some embodiments, the semantic oscillation entropy corresponding to the first modality is calculated based on the feature distribution of contradictory semantic switching of semantic labels corresponding to each perceptual data within a preset time interval, including at least one of the following methods: Method 1: Single-modal independent semantic conflict analysis The semantic oscillation frequency of the first modality is statistically analyzed at each preset time interval to obtain the semantic oscillation frequency distribution; the first semantic oscillation entropy is calculated based on the semantic oscillation frequency distribution, which is used to represent the degree of temporal semantic oscillation fluctuation of the semantic tags corresponding to each perceptual data of the first modality; the semantic oscillation frequency represents the number of times the semantic tags switch between contradictory semantics within a preset time period; Using this method, the number of times semantic tags of each modality switch between contradictory semantics within a preset time window can be counted at preset time intervals, i.e. semantic oscillation frequency, to obtain the frequency distribution of each modality, and further calculate the first semantic oscillation entropy of each modality.

[0030] For example, if the voice modality switches between "turn on the light / turn off the light / turn on the light" continuously within 3 seconds, the semantic flip count is 2 times, the oscillation frequency is high, which indicates a high degree of conflict.

[0031] For example, the preset time window length is T = 3 seconds, and the preset sampling time interval is Δt = 0.5 seconds. For each modality i, the number of semantic label flips within the statistical time window T is... Then the semantic oscillation frequency for: .

[0032] For example, the first semantic oscillation entropy of mode i can be calculated using the following formula:

[0033] in, This represents the first semantic oscillation entropy of the i-th mode. Let represent the semantic oscillation frequency of the i-th mode at time t, where T represents the preset time window length, and the summation operation represents the cumulative calculation of the entropy value over the entire time window.

[0034] For example, in the first modality, which is the speech modality, six semantic tags "turn on the light, turn on the light, turn off the light, turn on the light, turn on the light, turn off the light" are collected within 3 seconds. The number of semantic tag flips is [not specified]. =3, then semantic oscillation frequency =3 / 3=1Hz, the frequency distribution is concentrated and the entropy value is high, indicating that the speech modality has violent semantic oscillations in a short period of time.

[0035] Method 2: The temporal condition conflict probability between the first mode and the second mode is statistically analyzed at each preset time interval to obtain the condition conflict probability distribution; and the second semantic oscillation entropy is calculated based on the condition conflict probability distribution, wherein the second mode corresponds to any mode other than the first mode among multiple modes; the second semantic oscillation entropy is used to represent the degree of linkage oscillation influence of the first mode on the second mode.

[0036] Using this second method, the temporal linkage conflict relationship between the first mode and any other second mode can be statistically analyzed at preset time intervals to obtain the conditional conflict probability distribution. Based on the conditional conflict probability distribution, the second semantic oscillation entropy can be calculated to identify other modal linkage conflict problems caused by one mode oscillation.

[0037] For example, the second semantic oscillation entropy is calculated using the following formula:

[0038] in, The second semantic oscillation entropy represents the i-th modality. This represents the conditional probability that, at time t, mode i oscillates when mode j also oscillates; T represents the preset time window length, and the summation operation represents the cumulative calculation of the entropy value over the entire time window.

[0039] For example, the first mode is the voice mode, and the second mode is the lighting control mode; when the voice mode repeatedly oscillates between turning the lights on and off within a short period of time, and the lighting control mode also simultaneously experiences command conflicts, then the conditional probability... The higher entropy of the second semantic oscillation indicates that the speech modality has a significant linkage oscillation effect on the illumination type modality.

[0040] In this embodiment, by using the dual conflict quantification method of single-mode oscillation entropy and inter-mode linkage oscillation entropy, it is possible to identify both single-mode oscillations and multi-mode linkage conflicts, resulting in more comprehensive and robust conflict detection. Conflict quantification based on semantic oscillation entropy makes conflict judgment more accurate and provides stronger anti-interference capabilities.

[0041] S104. Based on the degree of semantic conflict, if a semantic conflict is determined to exist in the first modality, stop sending control commands to the smart device corresponding to the first modality.

[0042] In some embodiments, determining that a first modality has a semantic conflict based on the degree of semantic conflict includes: determining that a first modality has a semantic conflict when the semantic oscillation entropy is greater than a preset conflict threshold and the duration of the semantic conflict exceeds a preset minimum duration conflict threshold, wherein the duration of the semantic conflict is the duration during which the semantic oscillation entropy is greater than the preset conflict threshold. For example, when... and / or > (For example: =0.5) and continue > When the system enters a state of "continuous semantic oscillation," it is determined that... Indicates the duration of semantic conflict. A minimum duration conflict threshold is preset. In this embodiment, semantic conflicts are determined using a dual condition of entropy threshold and duration, avoiding false freezing caused by instantaneous fluctuations and noise interference, and improving the accuracy of conflict determination and the system's anti-interference capability.

[0043] In this embodiment, the above-described determination method is applicable to single-modal semantic conflicts. For example, a child repeatedly utters contradictory commands such as "turn on the light, turn off the light, turn on the light, turn off the light" within a short period of time, causing continuous semantic oscillations in a single speech modality. In practical engineering applications, depending on the actual scenario requirements, the system can be configured to determine the existence of a semantic conflict only when multiple modalities simultaneously experience semantic conflicts, further improving the reliability and accuracy of semantic conflict determination.

[0044] In some embodiments, once a persistent semantic conflict is determined, the system immediately enters a decision freeze state and suspends the issuance of control commands to the smart device corresponding to the conflicting mode.

[0045] For example, the following three implementation modes can be selected during the freeze period: Mode 1: System-level control pause (1) Edge computing devices (such as NVIDIA Jetson series, Raspberry Pi, etc.) continue to operate at full power, and the operating system, sensor acquisition, semantic analysis, network communication and other modules continue to work. Only the control command output module is frozen, specifically including: not sending control commands to the devices corresponding to the modal that has been determined to have semantic conflicts, and even if there are actions to be executed in the command queue, they are temporarily suspended. (2) Do not empty the instruction queue; mark the instructions to be executed as "pending confirmation". (3) User interaction modules such as voice prompts, application (APP) notifications, and LED status lights are working normally and can prompt users via voice or mobile APP: "Signal conflict detected, please confirm your needs." (4) Continuously monitor the semantic oscillation frequency and record the freeze duration.

[0046] Mode 2: Device Standby Mode Devices corresponding to modalities where semantic conflicts have been identified enter a shutdown state and can be woken up by voice, buttons, or other means; the core computing unit (CPU), memory, network, voice prompt module, and other devices corresponding to modalities where semantic conflicts have not occurred continue to operate normally.

[0047] Mode 3: System Light Hibernation Mode The CPU is downclocked to its lowest operating frequency (e.g., 500MHz), memory keeps data refreshed, non-critical processes are suspended, system power consumption is reduced, and conflict detection is retained.

[0048] In this embodiment, semantic conflict state is determined by both semantic oscillation entropy and duration, and decision freezing is performed in the conflict state to avoid frequent start-stop, malfunction or chaotic operation of smart devices due to contradictory commands.

[0049] In some embodiments, the method provided in this application further includes: in response to satisfying a preset unfreezing strategy, resuming the sending of control commands to the smart device corresponding to the first modality; for example, the preset unfreezing strategy includes at least one of the following: (1) receiving a confirmation instruction initiated by the user to resume sending control commands to the smart device corresponding to the first modality; (2) the semantic oscillation entropy of the first modality is less than a preset unfreezing threshold; (3) the blocking duration for stopping the sending of control commands to the smart device corresponding to the first modality reaches the unfreezing duration, the unfreezing duration being the minimum value between the predicted unfreezing time calculated based on the semantic oscillation frequency of the first modality and the user-preset unfreezing time.

[0050] In method (1), the user actively initiates a control command to the standby or hibernating device, such as waking up the voice device; in method (2), if the semantic oscillation entropy value of the modal semantic conflict decreases and is less than the preset unfreeze value, then the control of the smart device corresponding to the first modality is automatically restored; in method (3), after stopping the sending of control commands, a timer is started until the timer interruption duration reaches the value of the unfreeze duration, then the control of the smart device corresponding to the first modality is restored.

[0051] For example, the unfreezing time adopts an adaptive strategy and is calculated using the following formula:

[0052] in, This indicates the predicted thawing time. The user-preset manual defrosting time. Indicates the semantic oscillation frequency at the current moment; Indicates the preset thawing threshold; , representing the decay rate of the semantic oscillation frequency, This is a smoothing factor, a local constant used to avoid denominators of 0.

[0053] In this embodiment, the above-mentioned unfreezing strategy achieves a complete closed loop of "conflict freezing - intelligent unfreezing"; the unfreezing time is the minimum value between the predicted unfreezing time calculated based on semantic oscillation frequency and the user-preset unfreezing time, taking into account both the degree of automation and user controllability.

[0054] In some embodiments, after thawing, instead of immediately executing drastic control actions, a probing action is performed first. After observing that the user has no objection feedback, the system is gradually adjusted to the target state. For example, the air conditioner first enters the fan mode without directly heating or cooling; the lights are first adjusted to a medium brightness without being fully turned on or off; the curtains are first opened or closed to a medium position, and then fine-tuned based on user feedback to avoid triggering semantic oscillations a second time.

[0055] To further improve the control accuracy of smart devices, in some embodiments, this application also provides a smart device control method. This smart device control method models multimodal semantic conflicts as a multibody dynamics system. By assigning physical attributes such as mass, position, and velocity to each mode, it uses a mutually exclusive force field driven by Newtonian mechanics to achieve the natural separation of contradictory modes. Using the minimization of system energy as the decision criterion, it achieves interpretable, adaptive, and lightweight multimodal semantic arbitration with zero neural network and zero training data.

[0056] Figure 2 A flowchart illustrating a smart device control method according to an exemplary embodiment of this application is shown. Figure 2 As shown, the intelligent device control method further includes the following steps (S201-S202): S201. Treat each mode as a particle, and construct a multimodal particle mutual exclusion dynamics model based on the particles corresponding to each mode and the semantic tags corresponding to each mode. In this step, each modality (such as speech type, visual type, infrared body temperature type, ambient temperature type, lighting type, etc.) is treated as a particle with physical properties, moving in a multi-dimensional semantic space. Using multibody dynamics, by assigning physical properties such as mass, position, and velocity to each modality, multimodal semantic conflicts are modeled as a multimodal particle mutual exclusion dynamics model. When the semantic intents of different modalities conflict, mutual repulsion forces are generated between the particles, propelling them forward. The entire system automatically converges to the state of lowest energy as the particles move further apart, at which point the most stable particle is selected as the decision-making criterion. The entire process requires no training or weighting and is completely self-organizing.

[0057] In some embodiments, each mode is treated as a particle, and a multimodal particle mutual exclusion dynamics model is constructed based on the particles corresponding to each mode and the semantic tags corresponding to each mode, including: (1) Treat each mode as a particle, where the particle's properties include: position vector ,quality and velocity vector ; In some embodiments, the particle's position vector The coordinates of the mode in the semantic space are obtained by conflict polarization encoding of the semantic label of the corresponding mode (see detailed description below).

[0058] In some embodiments, the mass of the particle This represents the historical reliability of the corresponding modality. For example, the initial value is set to 1.0, read from the local cache, and reflects the historical decision accuracy of this modality.

[0059] In some embodiments, the particle velocity vector The initial value is a zero vector, representing the initial motion state of the particle in the semantic space.

[0060] In this embodiment, a particle i is initialized for each current mode i, and the semantic position vector of each particle is... The speed is calculated in real time by the collision polarization encoder. Initialized as a zero vector. For example, the speech modality, temperature modality, and infrared somatosensory modality are each initialized as three independent particles, each with a mass of [missing information]. = = 1.0, all speeds are 0, semantic position vector The results are given by their respective polarization encodings. In this embodiment, the multimodal conflict is transformed into a standard physical particle system to achieve quantifiable, evolvable, and interpretable conflict.

[0061] (2) Construction of vector-type mutual repulsion: Particle pairs formed by two particles i and j with contradictory semantics exhibit vector-type mutual repulsion. The direction of a vector-type repulsive force is determined by the difference in the position vectors of the two particles, that is, along... > Direction; The magnitude of the vector-type repulsive force is positively correlated with the mass of the two particles and negatively correlated with the square of the positional distance between the two particles; Wherein, the existence of contradictory semantics means that the positional distance between the particle pair is greater than or equal to a preset positional threshold; When there is no semantic contradiction between the two particles, the vector-type repulsive force between the two particles is zero vector; For example, iterate through all particle pairs (i,j) and calculate the semantic distance between particle pairs (i,j). =| |; If the semantic distance is less than the contradiction condition threshold, it indicates that the two intentions are similar but conflict exists, then the conflict coefficient =1, otherwise =0. For example, the speech modality intends to "heat", while the temperature modality intends to "cool". The two are semantically contradictory, thus generating a strong mutual repulsion force, causing the particles formed by the two modalities to move away from each other in the semantic space.

[0062] For example, the vector-type repulsive force between particle pairs can be calculated using the following formula. :

[0063] Here, G is essentially a hyperparameter, its physical meaning being the "stiffness coefficient" controlling the entire force field. A larger G indicates more intense intermodal repulsion and faster evolution; a smaller G indicates smoother evolution. This parameter is adjustable, tailored to the system's sensitivity to conflict. All forces follow vector operation rules.

[0064] In this embodiment, the automatic separation of contradictory modes is achieved through vector-type mutual exclusion force, eliminating the need for manual weight setting and making conflict resolution more natural.

[0065] (3) Calculation of total system energy: In some embodiments, the total system energy of the multimodal particle mutual repulsion dynamics model is obtained based on the properties of each particle and the vector-type mutual repulsion force. The total system energy is used to represent the overall energy state of the multimodal particle mutual repulsion dynamics model.

[0066] In this embodiment, the total system energy consists of two parts: the total kinetic energy of the particles and the total mutual repulsion potential energy of the particles. In some embodiments, Figure 3 This diagram illustrates the total energy evolution of a particle system provided in an embodiment of this application. Figure 3 As shown, based on the properties of each particle and the vector-type mutual repulsion force, the total system energy of the multimodal particle mutual repulsion dynamics model is obtained, including the following steps: S301. Summate all the vector-type repulsive forces acting on each particle to obtain the resultant force vector of each particle; For example, the resultant force vector of particle i is calculated using the following formula. : = , This indicates the calculation of the resultant force of other particles j on particle i, excluding itself.

[0067] S302. The acceleration vector of each particle is calculated based on the resultant force vector and mass of each particle. The direction of the acceleration vector is consistent with the direction of the resultant force vector. For example, the acceleration vector of particle i is calculated using the following formula. : = / ; S303. Based on the acceleration vector, calculate the velocity vector and semantic position vector of each particle; The velocity vector is updated using the following formula: (t+Δt)= (t)+ Δt; The semantic position vector is updated using the following formula: (t+Δt)= (t)+ (t) Δt. Wherein, (t) and (t) represents the velocity vector and semantic position vector of particle i at the current moment. (t+Δt) and (t+Δt) represents the velocity vector and semantic position vector of particle i at the next moment.

[0068] S304. Calculate the total kinetic energy of the particles based on their velocity vectors and masses; calculate the total mutual repulsive potential energy of the particles based on the vector-type mutual repulsive forces and positional distances between each pair of particles. For example, the total kinetic energy of a particle can be calculated using some formulas:

[0069] For example, the total mutual repulsive potential energy of particles can be calculated using some formulas: = ,in, Let be the potential function corresponding to the mutual repulsion force. ≈ .

[0070] S305. Summing the total kinetic energy of the particles with the total mutual repulsion potential energy of the particles yields the total system energy of the multimodal particle mutual repulsion dynamics model.

[0071] For example, the total energy of the system E= + .

[0072] In this embodiment, the faster the particles move and the stronger the mutual repulsion, the higher the total energy of the system; conversely, when the particles tend to be stationary and the distance is stable, the total energy is lower. Therefore, the total system energy can be used as a unified convergence criterion. The total system energy of the multimodal particle mutual repulsion dynamics model is iteratively calculated based on the vector-type mutual repulsion between particles until the total system energy is less than a preset total energy threshold or the maximum number of iterations is reached. At this point, the iteration converges, the multimodal particle mutual repulsion dynamics model reaches an energy stable state, and the self-organized resolution of multimodal conflicts is achieved, making the decision-making stable and reliable.

[0073] In this embodiment, semantic conflict is transformed into a physical particle dynamics problem, and the decision-making process is interpretable and traceable; the particle attributes (position / mass / velocity) and vector-type mutual repulsion rules are clarified, the model structure is clear and the computation is lightweight; the total energy of the system is constructed based on particle attributes and mutual repulsion forces, providing a unified judgment standard for subsequent iterative convergence.

[0074] S202. Based on the multimodal particle mutual exclusion dynamics model, select the mode with the highest historical credibility as the target decision mode, and send corresponding control commands to the corresponding intelligent device based on the target decision mode.

[0075] In this step, the system iteratively evolves to a stable state with the goal of minimizing energy, and selects the mode corresponding to the most stable and reliable particle to output control commands, thereby achieving precise decision-making.

[0076] In some embodiments, Figure 4 This diagram illustrates the particle iteration convergence and decision selection flowchart provided in an embodiment of this application. Figure 4 As shown, based on the multimodal particle mutual exclusion dynamics model, the mode with the highest historical reliability is selected as the target decision mode, and corresponding control commands are sent to the corresponding intelligent device based on the target decision mode, including the following steps (S401-S402): S401. With the goal of minimizing the total system energy of the multimodal particle mutual repulsion dynamics model, the total system energy of the multimodal particle mutual repulsion dynamics model is iteratively calculated based on the vector mutual repulsion force between particles until the total system energy is less than the preset total energy threshold or the maximum number of iterations is reached. The iteration converges and the multimodal particle mutual repulsion dynamics model reaches an energy stable state. For example, with the goal of minimizing the total system energy E, the mutual repulsion force calculation, velocity vector and semantic position vector update, and total energy update are repeatedly performed, i.e., steps S301-S305 are repeatedly executed until the following conditions are met: the change in total energy is less than a preset total energy threshold; or the number of iterations reaches the maximum number of iterations. At this point, the model reaches an energy stable state, the particles are basically stationary, and the conflict is completely resolved. In this embodiment, iterating to a stable state with the goal of minimizing the total system energy achieves self-resolution of multimodal conflicts.

[0077] S402. Under stable energy conditions, select the mode corresponding to the particle with the highest mass whose velocity vector is less than the preset velocity threshold as the target decision mode, generate control commands based on the position vector of the particle corresponding to the target decision mode, and send the control commands to the corresponding smart device.

[0078] In this embodiment, the velocity vector is selected under a stable energy state. Particles with velocity less than a preset threshold (nearly stationary); then, the mode corresponding to the particle with the largest mass is selected as the target decision mode; based on the particle's position vector... The mapping generates control commands (e.g., turning on the air conditioner's cooling mode if the location is near "cooling"), and sends them to the corresponding smart devices. In this embodiment, the mode corresponding to the most stable and highest-mass particle is selected as the decision-making basis, and the most reliable and stable control command is output.

[0079] For example, the mass of the voice modal particle is 0.9, the mass of the temperature modal particle is 1.0, and the mass of the infrared modal particle is 0.95. After stabilization, the mass of the temperature modal particle is the largest, so the temperature modal is selected as the decision basis to output cooling / heating commands.

[0080] In this embodiment, a multimodal particle mutual exclusion dynamics model is introduced to achieve multimodal reliable decision-making; the target decision mode is selected based on historical reliability, and stable and reliable control commands are output when there are contradictions in the multimodal models, thereby improving the control accuracy in complex scenarios.

[0081] In some embodiments, this application also proposes a particle mass adaptive update mechanism. The method provided in this application embodiment further includes: updating the particle mass based on user feedback information, whereby the user feedback information is a confidence confirmation result of the user's assessment of the execution result of the intelligent device corresponding to the target decision mode performing the corresponding operation according to the control command.

[0082] In this embodiment, after the smart device executes the corresponding operation indicated by the control command, the user can provide feedback on the execution effect of the decision (such as whether the decision is accurate). Since particle mass reflects the accuracy of the decision, the particle mass is updated in real time based on user feedback to achieve closed-loop adaptive optimization of particle mass. For example, if the user manually overrides the command (such as changing cooling to heating), it indicates that the decision is inaccurate and the particle mass needs to be reduced, for example, by multiplying the mass by a coefficient of 0.9. If the user explicitly affirms (such as confirming or giving a thumbs up), it indicates that the decision is accurate. For example, the particle mass can also be increased by a preset ratio, for example, by multiplying the mass by a coefficient of 1.05. The values ​​of the above coefficients are only illustrative examples and this application does not impose any limitations. In some embodiments, the updated mass is stored locally for the next conflict arbitration.

[0083] In this embodiment, the particle mass is adaptively updated based on user feedback, enabling online learning and continuous optimization of modal reliability. The adaptive update mechanism of particle mass does not require retraining the model, and can continuously improve the accuracy of subsequent decisions, forming a personalized adaptive control closed loop.

[0084] The above Figure 2 The illustrated intelligent device control method can run in real time on edge computing devices and is applicable to various scenarios such as smart homes, vehicles, elderly care, offices, and industries. In this embodiment, multimodal conflicts are transformed into particle dynamics systems, eliminating the need for neural networks and large amounts of labeled data. Furthermore, vector-type mutual repulsion forces are used to achieve automatic separation of conflicting modes, with the minimization of the total system energy as the convergence criterion, achieving self-organized stability of multimodal conflicts and making conflict resolution more natural and stable. The decision mode with the highest historical reliability is selected based on particle mass, making decisions interpretable and control more precise. At the same time, user feedback loops and adaptive quality updates are supported, enabling continuous evolution of decision-making capabilities.

[0085] In some embodiments, before constructing the multimodal particle mutual exclusion dynamics model in step S201, the semantic tags are first subjected to conflict polarization encoding to obtain the semantic position vector of each mode in a preset semantic space. In some embodiments, before constructing the multimodal particle mutual exclusion dynamics model, the method provided in this application further includes: performing conflict polarization encoding on the semantic tags of each mode to obtain the semantic position vector corresponding to each mode in a preset semantic space; the particle position vector... It is represented by the semantic position vector of the corresponding modality.

[0086] In this embodiment, conflict polarization coding is used to push modes with contradictory semantics apart in the semantic space, making the conflict relationship explicit and providing accurate and stable semantic coordinates for subsequent particle repulsion force calculations and dynamic iterations. The entire coding process requires no neural network training or pre-trained models, and is achieved solely through geometric optimization.

[0087] Figure 5 A flowchart illustrating the conflict polarization coding process provided in an embodiment of this application is shown. Figure 5 As shown, conflict polarization encoding is performed on the semantic tags of each modality to obtain the semantic position vector corresponding to each modality in the preset semantic space, including the following steps (S501-S504): S501. Map the semantic tags of the third modality to a preset semantic space according to a preset matching rule to generate the initial semantic position vector of the third modality; wherein, the third modality corresponds to any one of the multiple modalities; In this step, conflict polarization coding maps the semantic label of any modality (i.e., the third modality) to the semantic space according to a preset matching rule or a learnable mapping relationship, generating the initial semantic position vector of modality i. In this embodiment, the initial semantic location vector is directly mapped from raw perceptual data such as speech, temperature, infrared, and vision. This is used to preserve the original semantic information and avoid deviation from the user's true intent after polarization. For example, the speech "cold" is mapped to an initial semantic location vector closer to the "heating" area; and a high ambient temperature is mapped to an initial semantic location vector closer to the "cooling" area.

[0088] For example, in this step, a D-dimensional initial semantic encoding vector is generated for each modality i. ∈R D For example, the semantic space dimension D=8, which corresponds to heating tendency, cooling tendency, air supply tendency, light-on tendency, light-off tendency, security and alert tendency, entertainment and relaxation tendency, and quiet tendency, respectively.

[0089] For example, conflict polar coding uses different mapping methods depending on the modality type. The following examples illustrate this.

[0090] (1) Speech modality First, the voice command is processed by speech recognition to obtain text. Then, the text is classified according to intent. Finally, based on the intent, a lookup table is used to map the text into an 8-dimensional initial semantic location vector. .

[0091] (2) Temperature modes (using piecewise linear function mapping) For example, if the ambient temperature T < 18℃, the intent is identified as an increased tendency to heat, and the initial semantic location vector... The range is [0.9, 0.1, 0.0, ...]. For example, if the ambient temperature is 18℃≤T≤22℃, the intent is identified as a neutral state, and the initial semantic location vector is... For example, if the ambient temperature T > 24℃, the intention is to identify an increased tendency to cool down, and the initial semantic location vector is [0.3, 0.3, 0.4, ...]. [0.1, 0.9, 0.0, ...].

[0092] (3) Infrared somatosensory mode (using piecewise linear function mapping) For example, based on body surface temperature T body Perform piecewise function mapping, for example, T body <36℃, intent identified as increased heating tendency, initial semantic location vector For example, [0.98, 0.12, 0.0, ...]; and for example, T body >37℃, intent identified as increased cooling tendency, initial semantic location vector [0.12, 0.98, 0.0, ...].

[0093] (4) Camera / visual modality (using behavior recognition mapping) Behavior classification is performed through image recognition, followed by intent mapping to obtain the initial semantic location vector. For example, crossing one's arms or curling up indicates a tendency to generate heat; wiping sweat or fanning oneself indicates a tendency to cool down; reading or sewing indicates a tendency to turn on the lights; closing one's eyes or yawning indicates a tendency to turn off the lights. Based on the corresponding intention mapping, an initial semantic location vector is obtained. .

[0094] S502. Based on the current semantic position vector of the third mode and the contradictory semantic relationship between the fourth mode and the third mode, construct the total potential energy function of the third mode; In some embodiments, the total potential energy function is used to represent the semantic repulsion strength between modes with contradictory semantics and the retention constraint strength between the current semantic position vector and the initial semantic position vector; wherein, the fourth mode corresponds to any mode other than the third mode among multiple modes; In this embodiment, the current semantic position vector of the third modality The initial value is the initial semantic position vector obtained in step S501. As the repulsive forces between it and other modes are pushed away, its position changes.

[0095] In some embodiments, based on the current semantic position vector of the third mode and the contradictory semantic relationship between the fourth mode and the third mode, the total potential function of the third mode is constructed, including the following steps: S601. Determine the contradiction weights of the mode pairs based on the contradictory semantic relationships between the mode pairs formed by the third and fourth modes. Based on the contradiction weights, perform a weighted summation of the Euclidean distances between the current semantic position vectors of the mode pairs to obtain the contradiction polarization potential energy function of the third mode. The contradiction polarization potential energy function is used to represent the semantic repulsion strength between modes with contradictory semantics. S602. Based on the Euclidean distance between the current semantic position vector of the third modality and the initial semantic position vector of the third modality, construct the semantic preservation potential function of the third modality; the semantic preservation potential function is used to represent the constraint strength that limits the deviation of the current semantic position vector from the initial semantic position vector; S603. The total potential function of the third mode is obtained by weighted summation of the contradictory polarization potential function and the semantic preservation potential function.

[0096] In this embodiment, the total potential energy function is decomposed into contradictory polarization potential energy and semantic preservation potential energy, which not only ensures the separation of contradictory modes but also preserves the original semantic information, making the encoding more reasonable and more in line with the control intent.

[0097] In this embodiment, the total potential energy function Used to characterize two constraints simultaneously: (1) The repulsion strength between contradictory semantic modes, through the contradictory polarization potential energy function accomplish; (2) The strength of the retention constraint between the current semantic position and the initial semantic position is determined by the semantic retention potential function. accomplish.

[0098] Total potential energy function The design goal is to find a set of semantic location vectors. This results in: 1) Mode pairs with strong contradictions having large distances (i.e., maximizing) ); 2) Each modality does not deviate too far from its initial semantic position (i.e., minimizes) ).

[0099] In some embodiments, the total potential energy function is derived from the contradictory polarization potential energy function. With semantic preservation potential function Weighted summation yields: =-λ +μ , where λ is the polarization intensity coefficient (default 1.0) and μ is the semantic preservation coefficient (default 0.1).

[0100] In some embodiments, the contradiction weights are determined based on the contradictory semantic relationships between modal pairs. Based on the weighted summation of the Euclidean distances between mode pairs using contradictory weights, the contradictory polarization potential energy function is obtained: .

[0101] In some embodiments, a conflict weight matrix W∈R is constructed based on the actual conflict relationships between each modality pair. K×K Where K is the total number of modes, and the weights in the weight matrix W are... ∈[0,1], with the diagonal being 0.

[0102] The table below shows the conflict weights for some modal pairs and the conflict determination rules, which are for illustrative purposes only and are not intended to limit the scope of this application.

[0103]

[0104] For example, as shown in the table, if the voice says "cold" but the infrared body surface temperature is >37℃, the contradiction weight is 0.9, which is considered a strong contradiction; conversely, if the infrared image says "hot" and the camera captures an image of "crossed arms," ​​the contradiction weight is 0.3, which is considered a weak contradiction. In this embodiment, by quantifying the contradiction strength between mode pairs, the polarization process is made to better reflect the actual degree of conflict, providing contradiction weights for constructing the contradiction polarization potential energy function. The stronger the contradiction, the larger the weight, and the easier it is to resolve.

[0105] In some embodiments, a semantic preservation potential function is constructed based on the Euclidean distance between the current semantic position vector and the initial semantic position vector: In this embodiment, the potential energy is used to limit the position from deviating excessively from the initial semantics after conflict polarization coding, ensuring that the conflict polarization coding process is not distorted, so as to preserve the original intent and avoid control errors caused by over-polarization.

[0106] S503. With the goal of minimizing the total potential energy function, iteratively calculate the total potential energy function until the change in the total potential energy function is less than the preset convergence threshold or the maximum number of iterations is reached, at which point the iteration converges. In this embodiment, the gradient iterative update of the total potential energy function is performed with the goal of minimizing the total potential energy function:

[0107] in, It is the step size (learning rate). For any mode i, the gradient of the total potential energy function with respect to its position. The first term comes from the repulsive force of contradictory modes; the stronger the contradiction ( The greater the repulsive force, the greater the restoring force from the current semantic position vector. The further away from the initial semantic position vector, the greater the pulling force.

[0108] Iterate until either of the following conditions is met: the change in the total potential energy function is less than a preset convergence threshold; or, the number of iterations reaches the maximum number of iterations; or || , It can be set to 0.0001. For example, after 100 iterations, the potential energy no longer decreases, contradictory modes move away from each other, and the semantics remain stable.

[0109] S504. The semantic position vector obtained after iterative convergence is used as the semantic position vector corresponding to the third mode.

[0110] After iterative convergence, the final semantic position vector is used as the particle's position vector. .

[0111] The intelligent device control method provided in this application can quantify semantic conflicts based on the frequency of contradictory switching of semantic tags, achieve accurate conflict detection, actively freeze conflict commands when it is determined that there is a semantic conflict in the modality, avoid frequent device start-up and shutdown, reduce energy consumption, extend device life and improve safety; and adopt conflict polarization coding and particle dynamics arbitration to achieve interpretable resolution of multimodal conflicts, without the need for large AI model training, lightweight, and can run on edge devices.

[0112] An exemplary embodiment of this application provides a smart device control device 100. Figure 6 This illustration shows a structural block diagram of an intelligent device control device 100 provided in an exemplary embodiment of this application. The intelligent device control device 100 described above can achieve the following: Figures 1 to 5 All or part of the contents of any of the embodiments shown. The following is only a brief description of the structure and function of the intelligent device control device 100. For other matters not covered, please refer to the relevant descriptions in the above-described intelligent device control method. The embodiments of the intelligent device control device 100 correspond to the embodiments of the above-described intelligent device control method. All implementation processes and methods of the above-described method embodiments can be applied to the embodiments of the intelligent device control device and can achieve the same technical effects.

[0113] like Figure 6 As shown, the intelligent device control device 100 includes: an acquisition module 101, a semantic recognition module 102, a conflict determination module 103, and a control module 104. In this embodiment, the acquisition module 101 is used to acquire multiple perceptual data of a first modality; the first modality corresponds to any one of multiple modalities, and the modality indicates the data type of the perceptual data; the semantic recognition module 102 is used to perform intent recognition on the multiple perceptual data of the first modality and determine the semantic label corresponding to each perceptual data; the conflict determination module 103 is used to determine the degree of semantic conflict of the first modality based on the semantic label corresponding to each perceptual data; the degree of semantic conflict is used to indicate the frequency of switching between contradictory semantics of the semantic labels corresponding to each perceptual data; the control module 104 is used to determine that there is a semantic conflict in the first modality according to the degree of semantic conflict and stop sending control commands to the intelligent device corresponding to the first modality.

[0114] In some embodiments, the semantic recognition module 102 is used to perform intent recognition on multiple perceptual data of the first modality and determine the semantic label corresponding to each perceptual data, including: performing user operation intent recognition on the multiple perceptual data of the first modality respectively, and mapping each perceptual data to a corresponding semantic label based on intent matching rules.

[0115] In some embodiments, semantic tags are set based on the user's intention to operate the smart device, wherein the semantic tags include at least one of the following: temperature regulation, lighting control, security alert control, ambient atmosphere control, and device operating mode control.

[0116] In some embodiments, the conflict determination module 103 is used to determine the degree of semantic conflict of the first mode based on the semantic labels corresponding to each perceptual data, including: calculating the semantic oscillation entropy corresponding to the first mode based on the feature distribution of contradictory semantic switching of the semantic labels corresponding to each perceptual data within a preset time interval, wherein the semantic oscillation entropy is used to represent the degree of semantic conflict of the first mode.

[0117] In some embodiments, the semantic oscillation entropy includes: a first semantic oscillation entropy and / or a second semantic oscillation entropy; the conflict determination module 103 is used to calculate the semantic oscillation entropy corresponding to the first modality based on the feature distribution of contradictory semantic switching of the semantic labels corresponding to each perceptual data within a preset time interval, including: The semantic oscillation frequency of the first modality is statistically analyzed at preset time intervals to obtain the semantic oscillation frequency distribution; the first semantic oscillation entropy is calculated based on the semantic oscillation frequency distribution, which represents the degree of temporal semantic oscillation fluctuation of the semantic tags corresponding to each perceptual data of the first modality; the semantic oscillation frequency represents the number of times the semantic tag switches between contradictory semantics within a preset time period; and / or, The temporal condition conflict probability between the first mode and the second mode is statistically analyzed at each preset time interval to obtain the condition conflict probability distribution; and the second semantic oscillation entropy is calculated based on the condition conflict probability distribution, wherein the second mode corresponds to any mode other than the first mode among multiple modes; the second semantic oscillation entropy is used to represent the degree of linkage oscillation influence of the first mode on the second mode.

[0118] In some embodiments, the conflict determination module 103 is used to determine that a first modality has a semantic conflict based on the degree of semantic conflict, including: determining that a first modality has a semantic conflict when the semantic oscillation entropy is greater than a preset conflict threshold and the semantic conflict duration exceeds a preset minimum duration conflict threshold, wherein the semantic conflict duration is the duration during which the semantic oscillation entropy is greater than the preset conflict threshold.

[0119] In some embodiments, the control module 104 is further configured to resume sending control commands to the smart device corresponding to the first mode in response to satisfying a preset unfreezing strategy; in some embodiments, the preset unfreezing strategy includes at least one of the following: Upon receiving a user-initiated recovery request, a confirmation instruction is sent to the smart device corresponding to the first mode to confirm the control command. The semantic oscillation entropy of the first mode is less than the preset thawing threshold; The blocking time for stopping the transmission of control commands to the smart device corresponding to the first mode reaches the unfreezing time. The unfreezing time is the minimum value between the predicted unfreezing time calculated based on the semantic oscillation frequency of the first mode and the user-preset unfreezing time.

[0120] In some embodiments, such as Figure 6 As shown, the intelligent device control device 100 provided in this application embodiment further includes: a multimodal particle dynamics module 105, used to treat each mode as a particle, construct a multimodal particle mutual exclusion dynamics model based on the particles corresponding to each mode and the semantic tags corresponding to each mode; based on the multimodal particle mutual exclusion dynamics model, select the mode with the highest historical credibility as the target decision mode, and send corresponding control commands to the corresponding intelligent device through the control module 104 based on the target decision mode.

[0121] In some embodiments, the multimodal particle dynamics module 105 is used to treat each mode as a particle and construct a multimodal particle mutual exclusion dynamics model based on the particles corresponding to each mode and the semantic tags corresponding to each mode, including: Each mode is treated as a particle, and the particle's attributes include: position vector, mass, and velocity vector. The particle's position vector is obtained by conflict polarization encoding of the semantic tags of the corresponding mode, and the particle's mass represents the historical credibility of the corresponding mode. The particle's velocity vector is initially set to zero. A particle pair formed by two particles with contradictory semantics has a vector-type repulsive force. The direction of the vector-type repulsive force is determined along the difference in the position vectors of the two particles. The magnitude of the vector-type repulsive force is positively correlated with the mass of the two particles and negatively correlated with the square of the position distance between the two particles. Here, contradictory semantics means that the position distance between the particle pair is greater than or equal to a preset position threshold. When there is no semantic contradiction between the two particles, the vector-type repulsive force between the two particles is zero. Based on the properties of each particle and the vector-type mutual repulsion force, the total system energy of the multimodal particle mutual repulsion dynamics model is obtained. The total system energy is used to represent the overall energy state of the multimodal particle mutual repulsion dynamics model.

[0122] In some embodiments, the multimodal particle dynamics module 105 is used to obtain the total system energy of the multimodal particle mutual repulsion dynamics model based on the properties and vector-type mutual repulsion forces of each particle, including: summing all vector-type mutual repulsion forces acting on each particle to obtain the resultant force vector of each particle; calculating the acceleration vector of each particle based on the resultant force vector and mass, wherein the direction of the acceleration vector is consistent with the direction of the resultant force vector; calculating the velocity vector and semantic position vector of each particle based on the acceleration vector; calculating the total kinetic energy of the particles based on the velocity vector and mass of each particle; calculating the total mutual repulsion potential energy of the particles based on the vector-type mutual repulsion forces and positional distances between each pair of particles; and summing the total kinetic energy of the particles and the total mutual repulsion potential energy of the particles to obtain the total system energy of the multimodal particle mutual repulsion dynamics model.

[0123] In some embodiments, such as Figure 6 As shown, the intelligent device control device 100 provided in this application embodiment further includes: a conflict polarization coding module 106, used to perform conflict polarization coding on the semantic tags of each mode before constructing a multimodal particle mutual exclusion dynamics model, to obtain the semantic position vector corresponding to each mode in a preset semantic space; the position vector of the particle is represented by the semantic position vector of the corresponding mode.

[0124] In some embodiments, the conflict polarization coding module 106 is used to perform conflict polarization coding on the semantic tags of each modality to obtain the semantic position vector corresponding to each modality in a preset semantic space, including: The semantic labels of the third modality are mapped to a preset semantic space according to a preset matching rule to generate the initial semantic position vector of the third modality; wherein, the third modality corresponds to any modality among multiple modalities; Based on the current semantic position vector of the third mode and the contradictory semantic relationship between the fourth mode and the third mode, a total potential energy function of the third mode is constructed. The total potential energy function is used to represent the semantic repulsion strength between modes with contradictory semantics and the maintenance constraint strength between the current semantic position vector and the initial semantic position vector. Among them, the fourth mode corresponds to any mode other than the third mode among multiple modes. With the goal of minimizing the total potential energy function, the total potential energy function is iteratively calculated until the change in the total potential energy function is less than the preset convergence threshold or the maximum number of iterations is reached, at which point the iteration converges. The semantic position vector obtained after iterative convergence is used as the semantic position vector corresponding to the third mode.

[0125] In some embodiments, the conflict polarization coding module 106 is used to construct the total potential function of the third mode based on the current semantic position vector of the third mode and the contradictory semantic relationship between the fourth mode and the third mode, including: The contradiction weights of the mode pairs are determined based on the contradictory semantic relationships between the mode pairs formed by the third and fourth modes. The Euclidean distances between the current semantic position vectors of the mode pairs are then weighted and summed based on the contradiction weights to obtain the contradiction polarization potential function of the third mode. The contradiction polarization potential function is used to represent the semantic repulsion strength between modes with contradictory semantics. Based on the Euclidean distance between the current semantic position vector of the third modality and the initial semantic position vector of the third modality, a semantic preservation potential function of the third modality is constructed; the semantic preservation potential function is used to represent the constraint strength that limits the deviation of the current semantic position vector from the initial semantic position vector; The total potential function of the third mode is obtained by weighted summation of the contradictory polarization potential function and the semantic preservation potential function.

[0126] In some embodiments, the multimodal particle dynamics module 105 is used to select the mode with the highest historical reliability as the target decision mode based on a multimodal particle mutual exclusion dynamics model, and send corresponding control commands to the corresponding smart device through the control module 104 based on the target decision mode, including: With the goal of minimizing the total system energy of the multimodal particle mutual repulsion dynamics model, the total system energy of the multimodal particle mutual repulsion dynamics model is iteratively calculated based on the vector mutual repulsion force between particles until the total system energy is less than the preset total energy threshold or the maximum number of iterations is reached. The iteration converges and the multimodal particle mutual repulsion dynamics model reaches an energy stable state. In a stable energy state, the mode corresponding to the particle with the highest mass whose velocity vector is less than a preset velocity threshold is selected as the target decision mode. The control module 104 generates control commands based on the position vector of the particle corresponding to the target decision mode and sends the control commands to the corresponding smart device.

[0127] In some embodiments, the multimodal particle dynamics module 105 is further configured to update the mass of the particles based on user feedback information, wherein the user feedback information is a confidence confirmation result of the execution result of the intelligent device corresponding to the target decision mode performing the corresponding operation according to the control command.

[0128] The intelligent device control device provided in this application embodiment can quantify semantic conflicts based on the frequency of contradictory switching of semantic tags, achieve accurate conflict detection, actively freeze conflict commands when it is determined that there is a semantic conflict in the modality, avoid frequent device start-up and shutdown, reduce energy consumption, extend device life and improve safety; and adopt conflict polarization coding and particle dynamics arbitration to achieve interpretable resolution of multimodal conflicts, without the need for large AI model training, lightweight, and can run on edge devices.

[0129] Figure 7A structural block diagram of an electronic device 1000 illustrating an exemplary embodiment of this application is shown. The electronic device 1000 can be implemented as the aforementioned intelligent device control device, which can be configured in electronic devices such as smartphones, tablets, laptops, desktop computers, smartwatches, televisions, or servers.

[0130] Typically, electronic device 1000 includes a processor 1001 and a memory 1002.

[0131] Processor 1001 may include one or more processing cores, such as a quad-core processor, a deca-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0132] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one instruction, which is executed by the processor 1001 to implement all or part of the steps in the intelligent device control method illustrated in the method embodiments of this application.

[0133] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the electronic device 1000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0134] In one exemplary embodiment, a readable storage medium is also provided, which stores a program or instructions that, when executed by a processor, implement all or part of the steps in the above-described intelligent device control method. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0135] In one exemplary embodiment, a computer program product is also provided, comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the aforementioned... Figures 1 to 5 All or part of the steps of the intelligent device control method shown in any embodiment.

[0136] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0137] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for controlling an intelligent device, characterized in that, include: Acquire multiple sensory data from the first modality; The first modality corresponds to any one of multiple modalities, and the modality indicates the data type of the sensed data; Intent recognition is performed on multiple perceptual data of the first modality to determine the semantic label corresponding to each perceptual data. Based on the feature distribution of contradictory semantic switching of semantic labels corresponding to each perceptual data within a preset time interval, the semantic oscillation entropy corresponding to the first modality is calculated; wherein, the semantic oscillation entropy is used to represent the degree of semantic conflict of the first modality, and the degree of semantic conflict is used to represent the frequency of switching between contradictory semantics of semantic labels corresponding to each perceptual data. If the semantic oscillation entropy is greater than a preset conflict threshold and the semantic conflict duration exceeds a preset minimum duration conflict threshold, it is determined that the first modality has a semantic conflict, and control commands are stopped being sent to the smart device corresponding to the first modality; wherein, the semantic conflict duration is the duration during which the semantic oscillation entropy is greater than the preset conflict threshold.

2. The method according to claim 1, characterized in that, The step of performing intent recognition on multiple perceptual data points of the first modality and determining the semantic label corresponding to each perceptual data point includes: User operation intent is identified for multiple perception data of the first modality, and each perception data is mapped to a corresponding semantic label based on intent matching rules.

3. The method according to claim 2, characterized in that, The semantic tags are set based on the user's intention to operate the smart device, and the semantic tags include at least one of the following: temperature adjustment, lighting control, security alarm control, environmental atmosphere control, and device operation mode control.

4. The method according to claim 1, characterized in that, The semantic oscillation entropy includes: a first semantic oscillation entropy and / or a second semantic oscillation entropy; The semantic oscillation entropy corresponding to the first modality is calculated based on the feature distribution of contradictory semantic switching of semantic labels corresponding to each perceptual data within a preset time interval, including: The semantic oscillation frequency of the first modality is statistically analyzed at preset time intervals to obtain a semantic oscillation frequency distribution; the first semantic oscillation entropy is calculated based on the semantic oscillation frequency distribution, and the first semantic oscillation entropy is used to represent the degree of temporal semantic oscillation fluctuation of the semantic tags corresponding to each perceptual data of the first modality; the semantic oscillation frequency represents the number of times the semantic tag switches between contradictory semantics within a preset time period; and / or, The temporal condition conflict probability between the first mode and the second mode is statistically analyzed at each preset time interval to obtain the condition conflict probability distribution; and the second semantic oscillation entropy is calculated based on the condition conflict probability distribution, wherein the second mode corresponds to any mode other than the first mode among multiple modes; the second semantic oscillation entropy is used to represent the degree of linkage oscillation influence of the first mode on the second mode.

5. The method according to claim 4, characterized in that, The method further includes: In response to satisfying the preset unfreezing strategy, control commands are resumed to be sent to the smart device corresponding to the first mode; The preset defrosting strategy includes at least one of the following: Upon receiving a user-initiated recovery request, a confirmation instruction is sent to the smart device corresponding to the first mode to confirm the control command. The semantic oscillation entropy of the first modality is less than the preset thawing threshold; The blocking duration for stopping the transmission of control commands to the smart device corresponding to the first modality reaches the unfreezing duration, wherein the unfreezing duration is the minimum value between the predicted unfreezing time calculated based on the semantic oscillation frequency of the first modality and the user-preset unfreezing time.

6. The method according to claim 1, characterized in that, The method further includes: Each mode is treated as a particle, and a multimodal particle mutual exclusion dynamics model is constructed based on the particles corresponding to each mode and the semantic tags corresponding to each mode; wherein, the multimodal particle mutual exclusion dynamics model is a dynamics model based on a multibody dynamics system, which transforms multimodal semantic conflicts into mutual exclusion forces between particles and autonomously converges to the lowest energy state; Based on the multimodal particle mutual exclusion dynamics model, the mode with the highest historical reliability is selected as the target decision mode, and corresponding control commands are sent to the corresponding smart devices based on the target decision mode.

7. The method according to claim 6, characterized in that, The process of treating each modality as a particle and constructing a multimodal particle mutual exclusion dynamics model based on the particles corresponding to each modality and the semantic tags corresponding to each modality includes: Each mode is treated as a particle, wherein the particle's attributes include: a position vector, a mass, and a velocity vector. The particle's position vector is obtained by conflict polarization coding of the semantic tags of the corresponding mode. The particle's mass represents the historical reliability of the corresponding mode. The particle's velocity vector is initially set to a zero vector. A pair of particles with contradictory semantics has a vector-type mutual repulsion force. The direction of the vector-type mutual repulsion force is determined along the direction of the difference between the position vectors of the two particles. The magnitude of the vector-type mutual repulsion force is positively correlated with the mass of the two particles and negatively correlated with the square of the position distance between the two particles. The contradictory semantics refer to the position distance between the particle pair being greater than or equal to a preset position threshold. When there is no semantic contradiction between the two particles, the vector-type mutual repulsion force between them is zero. Based on the properties of each particle and the vector-type mutual repulsion force, the total system energy of the multimodal particle mutual repulsion dynamics model is obtained. The total system energy is used to represent the overall energy state of the multimodal particle mutual repulsion dynamics model.

8. The method according to claim 7, characterized in that, The total system energy of the multimodal particle mutual repulsion dynamics model, obtained based on the properties of each particle and the vector-type mutual repulsion force, includes: The resultant force vector of each particle is obtained by summing all the vector-type repulsive forces acting on it. The acceleration vector of each particle is calculated based on the resultant force vector and mass of each particle, and the direction of the acceleration vector is consistent with the direction of the resultant force vector. Based on the acceleration vector, calculate the velocity vector and semantic position vector of each particle; Calculate the total kinetic energy of the particles based on their velocity vectors and masses; calculate the total mutual repulsive potential energy of the particles based on the vector-type mutual repulsive forces and positional distances between each pair of particles. The total kinetic energy of the particles is summed with the total mutual repulsion potential energy of the particles to obtain the total system energy of the multimodal particle mutual repulsion dynamics model.

9. The method according to claim 7, characterized in that, Prior to constructing the multimodal particle mutual exclusion dynamics model, the method further includes: The semantic tags of each modality are subjected to conflict polarization encoding to obtain the semantic position vector of each modality in the preset semantic space; the position vector of the particle is represented by the semantic position vector of the corresponding modality.

10. The method according to claim 9, characterized in that, The step of performing conflict polarization encoding on the semantic tags of each modality to obtain the semantic position vector corresponding to each modality in the preset semantic space includes: The semantic tags of the third modality are mapped to the preset semantic space according to a preset matching rule to generate the initial semantic position vector of the third modality; wherein, the third modality corresponds to any one of multiple modalities; Based on the current semantic position vector of the third mode and the contradictory semantic relationship between the fourth mode and the third mode, a total potential energy function of the third mode is constructed. The total potential energy function is used to represent the semantic repulsion strength between modes with contradictory semantics and the retention constraint strength between the current semantic position vector and the initial semantic position vector; wherein, the fourth mode corresponds to any mode other than the third mode among multiple modes; With the goal of minimizing the total potential energy function, the total potential energy function is iteratively calculated until the change in the total potential energy function is less than a preset convergence threshold or the maximum number of iterations is reached, at which point the iteration converges. The semantic position vector obtained after iterative convergence is used as the semantic position vector corresponding to the third modality.

11. The method according to claim 10, characterized in that, The construction of the total potential energy function of the third mode based on the current semantic position vector of the third mode and the contradictory semantic relationship between the fourth mode and the third mode includes: The contradiction weights of the mode pairs are determined based on the contradictory semantic relationships between the mode pairs formed by the third mode and the fourth mode. The Euclidean distances between the current semantic position vectors of the mode pairs are then weighted and summed based on the contradiction weights to obtain the contradiction polarization potential function of the third mode. The contradiction polarization potential function is used to represent the semantic repulsion strength between modes with contradictory semantics. Based on the Euclidean distance between the current semantic position vector of the third modality and the initial semantic position vector of the third modality, a semantic preservation potential function of the third modality is constructed; the semantic preservation potential function is used to represent the constraint strength that limits the deviation of the current semantic position vector from the initial semantic position vector; The total potential function of the third mode is obtained by weighted summation of the contradictory polarization potential function and the semantic preservation potential function.

12. The method according to claim 7, characterized in that, Based on the multimodal particle mutual exclusion dynamics model, the mode with the highest historical reliability is selected as the target decision mode, and corresponding control commands are sent to the corresponding intelligent device based on the target decision mode, including: With the goal of minimizing the total system energy of the multimodal particle mutual repulsion dynamics model, the total system energy of the multimodal particle mutual repulsion dynamics model is iteratively calculated based on the vector mutual repulsion force between particles until the total system energy is less than a preset total energy threshold or the maximum number of iterations is reached, at which point the iteration converges and the multimodal particle mutual repulsion dynamics model reaches an energy stable state. In the energy stable state, the mode corresponding to the particle with the highest mass whose velocity vector is less than a preset velocity threshold is selected as the target decision mode, and the control command is generated based on the position vector of the particle corresponding to the target decision mode and sent to the corresponding smart device.

13. The method according to claim 12, characterized in that, The method further includes: The mass of the particle is updated based on user feedback information, which is the user's confidence confirmation of the execution result of the intelligent device corresponding to the target decision mode performing the corresponding operation according to the control command.

14. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the intelligent device control method as described in any one of claims 1 to 13.

15. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the intelligent device control method as described in any one of claims 1 to 13.

16. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, implement the steps of the intelligent device control method as described in any one of claims 1 to 13.

Images