Instruction sending method and apparatus, storage medium, and electronic device

Abstract

The application discloses a kind of instruction sending method and device, storage medium and electronic device, it is related to wisdom family technical field, the method comprises: in the case where receiving target object is sent by the first instruction of first device using first protocol to second device, determine the second protocol used by second device, and determine the target data structure under first instruction in second protocol;First protocol and second protocol are extracted, to obtain the first protocol feature corresponding to first protocol and the second protocol feature corresponding to second protocol;Characteristic matching is carried out to first protocol feature and second protocol feature by target matching algorithm, and protocol conversion matrix between first protocol and second protocol is constructed according to feature matching result;First instruction is converted into second instruction meeting second protocol based on protocol conversion matrix and target data structure, and second instruction is sent to second device.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a method and apparatus for transmitting instructions, a storage medium, and an electronic device. Background Technology

[0002] With the development of IoT technology, a large number of devices of different brands and types are connected to the network, each using different communication protocols and UI designs. Traditional smart home systems are often designed for a specific protocol, such as only supporting ZigBee devices. Such systems cannot directly communicate with devices using other protocols (such as BLE), limiting device compatibility and scalability, and making it difficult to meet users' diverse device access needs.

[0003] Therefore, in related technologies, there is a problem that smart home systems only support communication between devices using a single protocol and cannot directly communicate with devices using other protocols.

[0004] There is no effective solution yet to address the issue that smart home systems in related technologies only support communication between devices using a single protocol and cannot directly communicate with devices using other protocols. Summary of the Invention

[0005] This application provides a method and apparatus for sending instructions, a storage medium, and an electronic device to at least solve the problem in the related art that smart home systems only support communication between devices using a single protocol and cannot directly communicate with devices using other protocols.

[0006] According to one embodiment of this application, a method for sending an instruction is provided, comprising: upon receiving a first instruction sent by a target object to a second device via a first device using a first protocol, determining a second protocol used by the second device, and determining a target data structure of the first instruction under the second protocol; performing feature extraction on the first protocol and the second protocol to obtain a first protocol feature corresponding to the first protocol and a second protocol feature corresponding to the second protocol; performing feature matching on the first protocol feature and the second protocol feature using a target matching algorithm, and constructing a protocol conversion matrix between the first protocol and the second protocol based on the feature matching result; converting the first instruction into a second instruction conforming to the second protocol based on the protocol conversion matrix and the target data structure, and sending the second instruction to the second device.

[0007] In one exemplary embodiment, determining the second protocol adopted by the second device includes: if it is determined that the second device has opened the protocol adopted by the second device to the first device, obtaining the device identifier of the second device and determining the second protocol based on the device identifier; if it is determined that the second device has not opened the protocol adopted by the second device to the first device, obtaining historical communication data packets corresponding to the second device and the operation trajectory of the target object on the second control corresponding to the second device; extracting features from the historical communication data packets and inputting the features corresponding to the historical communication data packets and the operation trajectory into a machine learning algorithm to determine the protocol logic corresponding to the protocol adopted by the second device; generating a protocol template corresponding to the protocol adopted by the second device based on the protocol logic, wherein the protocol template is a description file of the second protocol adopted by the second device.

[0008] In an exemplary embodiment, determining the target data structure of the first instruction under the second protocol includes: when it is determined that the target object sends the first instruction to the second device through a first control in the first device, dividing the control image corresponding to the first control into multiple image blocks, and mapping each image block to a target vector; adding position encoding to the target vector to generate an augmented vector containing the spatial position information of each image block in the control image; performing feature extraction on the multiple augmented vectors through a target encoder to obtain a global feature vector corresponding to the first control, and inputting the global feature vector into a classification model to obtain the control type corresponding to the first control; mapping the first control to a target semantic label according to the control type; and determining the target data structure in a semantic protocol mapping table, wherein the semantic protocol mapping table is used to indicate the instruction format of the semantic label corresponding to each control under each protocol.

[0009] In an exemplary embodiment, after sending the second instruction to the second device, the method further includes: if the sending status corresponding to the second instruction is determined to be a sending failure, obtaining historical communication data packets corresponding to the second device, and extracting protocol version change features corresponding to the second protocol from the historical communication data packets using a multidimensional feature extraction algorithm; matching the protocol version change features with preset protocol version change features in a protocol version change library to determine the protocol change status corresponding to the second protocol; if the protocol change status indicates that the protocol version of the second protocol has been changed to a first protocol version, obtaining historical protocol version usage logs corresponding to the second device, and determining a device compatible version list corresponding to the second device based on the historical protocol version usage logs; determining a target protocol version of the second protocol in the device compatible version list, and sending a protocol version rollback instruction to the second device to roll back the second protocol used by the second device from the first protocol version to the target protocol version; and sending the second instruction to the second device using the second protocol of the target protocol version.

[0010] In an exemplary embodiment, after sending the second instruction to the second device, the method further includes: acquiring a current signal of the second device during the execution of the second instruction, and extracting a current fingerprint feature corresponding to the current signal; acquiring state feedback information of the second device after executing the second instruction; encoding the current fingerprint feature and the state feedback information into first feature vectors respectively, and concatenating multiple first feature vectors into a fused feature vector; inputting the fused feature vector into a deep neural network model to determine the execution state of the second device executing the second instruction, wherein the execution state includes one of the following: correct execution, incorrect execution.

[0011] In an exemplary embodiment, after constructing a protocol conversion matrix between the first protocol and the second protocol based on feature matching results, the method further includes: determining whether a preset error conversion library contains an error conversion record corresponding to the protocol conversion matrix; if the preset error conversion library contains the error conversion record, sending a broadcast request to each node to obtain a repair rule corresponding to the protocol conversion matrix, wherein multiple nodes are used to construct the protocol conversion matrix; and repairing the protocol conversion matrix according to the repair rule.

[0012] In an exemplary embodiment, before determining the second protocol adopted by the second device, the method further includes: upon receiving a voice instruction sent by the target object, extracting keywords and semantic structure corresponding to the voice instruction, and converting the voice instruction into target control logic based on the keywords and semantic structure; matching the target control logic with the device interface corresponding to each device to generate an instruction sequence containing a target execution device, wherein the instruction sequence includes: a first instruction, and the target execution device contained in the first instruction is: the second device.

[0013] According to another embodiment of the present application, an instruction sending apparatus is also provided, comprising: a generation module, configured to, upon capturing a call event of a first thread through a hook mechanism, generate a memory allocation record entry corresponding to each first memory block based on the thread identifier of the first thread and attribute information of each first memory block, and write multiple memory allocation record entries into a memory tracking table in the first thread, wherein the first thread is used to allocate multiple first memory blocks in a target system; a determination module, configured to, during the process of inspecting the first thread based on a preset inspection cycle, determine a health index corresponding to the first thread based on the memory tracking table, and determine a target score corresponding to the first thread based on the health index, wherein the health index includes at least one of the following: an aging index corresponding to the multiple first memory blocks, and a cross-thread inconsistency index corresponding to the multiple first memory blocks; and a judgment module, configured to send instructions to the first thread based on the health index and the target score to determine whether the first thread exhibits abnormal behavior.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the sending method of the above-described instructions when it is run.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described method of sending instructions through the computer program.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program, wherein the computer program is executed by a processor using the method of sending the instructions described above.

[0017] In this embodiment, upon receiving a first instruction sent by a target object to a second device via a first device using a first protocol, the second protocol used by the second device and the target data structure of the first instruction under the second protocol are determined. Feature extraction is performed on the first and second protocols to obtain the first protocol features corresponding to the first protocol and the second protocol features corresponding to the second protocol. Feature matching is performed on the first and second protocol features using a target matching algorithm, and a protocol conversion matrix between the first and second protocols is constructed based on the feature matching results. Based on the protocol conversion matrix and the target data structure, the first instruction is converted into a second instruction conforming to the second protocol, and the second instruction is sent to the second device. In other words, when it is determined that a first instruction corresponding to the first protocol needs to be sent to the second device, this application can determine the second protocol used by the second device and the target data structure of the first instruction under the second protocol. Further, feature extraction is performed on the first and second protocols respectively, and a protocol conversion matrix between the first and second protocols is constructed based on the first and second protocol features. Based on the protocol conversion matrix and the target data structure, the first instruction can be converted into a second instruction conforming to the second protocol. Based on this application, the problem that smart home systems in related technologies only support communication between devices using a single protocol and cannot directly communicate with devices using other protocols can be solved. Furthermore, communication between devices using different protocols can be achieved through a technical solution that converts instructions of the first protocol into instructions of the second protocol. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the hardware environment for a method of sending instructions according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of a method for sending instructions according to an embodiment of this application;

[0022] Figure 3 This is an architecture diagram of a cross-platform protocol parsing and adaptation system according to an optional embodiment of this application;

[0023] Figure 4 This is a schematic diagram of a multi-protocol dynamic translation process according to an optional embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the UI space semantic generalization mapping process according to an optional embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the reverse engineering process of a proprietary protocol according to an optional embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the protocol version intelligent downgrade process according to an optional embodiment of this application;

[0027] Figure 8 This is a schematic diagram of the multimodal protocol verification process according to an optional embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the protocol fragment self-healing process according to an optional embodiment of this application;

[0029] Figure 10 This is a schematic diagram of the dynamic extension process of the semantic protocol according to an optional embodiment of this application;

[0030] Figure 11 This is a structural block diagram of an instruction sending device according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] According to one aspect of the embodiments of this application, a method for sending instructions is provided. This method for sending instructions is widely applicable to whole-house intelligent digital control application scenarios such as smart homes, smart home ecosystems, and intelligence house ecosystems. Optionally, Figure 1 This is a schematic diagram of the hardware environment for a method of sending instructions according to an embodiment of this application. In this embodiment, the above-described method of sending instructions can be applied to a computer device, and the computer device can be applied to, for example, Figure 1 The hardware environment shown consists of home appliance 102 and server 104. Figure 1 As shown, server 104 is connected to home appliance 102 via a network and can be used to provide services (such as application services) to nodes or clients installed on nodes. A database can be set up on the server or independently to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently to provide data processing services for server 104.

[0034] The aforementioned networks may include, but are not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network (WAN), metropolitan area network (MAN), local area network (LAN). The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. Home appliances 102 may not be limited to smart air conditioners, smart range hoods, smart refrigerators, smart ovens, smart stoves, smart washing machines, smart water heaters, smart washing equipment, smart dishwashers, smart projectors, smart TVs, smart clothes racks, smart curtains, smart audio-visual equipment, smart sockets, smart speakers, smart speakers, smart fresh air systems, smart kitchen and bathroom equipment, smart bathroom fixtures, smart robot vacuum cleaners, smart window cleaning robots, smart mopping robots, smart air purifiers, smart steam ovens, smart microwave ovens, smart water heaters, smart air purifiers, smart water dispensers, smart door locks, etc.

[0035] This embodiment provides a method for sending instructions, applied to the aforementioned computer device. Figure 2 This is a flowchart of a method for sending instructions according to an embodiment of this application, the process including the following steps:

[0036] Step S202: Upon receiving a first instruction sent by the target object to the second device through the first device using the first protocol, determine the second protocol used by the second device, and determine the target data structure of the first instruction under the second protocol;

[0037] That is, to determine the target instruction format of the first instruction under the second protocol.

[0038] Step S204: Perform feature extraction on the first protocol and the second protocol to obtain the first protocol feature corresponding to the first protocol and the second protocol feature corresponding to the second protocol;

[0039] Step S206: Perform feature matching on the first protocol features and the second protocol features using a target matching algorithm, and construct a protocol conversion matrix between the first protocol and the second protocol based on the feature matching results;

[0040] Among them, the target matching algorithms mentioned above include, but are not limited to, multidimensional feature matching algorithms.

[0041] Step S208: Based on the protocol conversion matrix and the target data structure, the first instruction is converted into a second instruction conforming to the second protocol, and the second instruction is sent to the second device.

[0042] Through the above steps, upon receiving a first instruction sent by a first device using a first protocol to a second device from a target object, the second protocol used by the second device and the target data structure of the first instruction under the second protocol are determined. Feature extraction is performed on the first and second protocols to obtain the first protocol features corresponding to the first protocol and the second protocol features corresponding to the second protocol. Feature matching is performed on the first and second protocol features using a target matching algorithm, and a protocol conversion matrix between the first and second protocols is constructed based on the feature matching results. Based on the protocol conversion matrix and the target data structure, the first instruction is converted into a second instruction conforming to the second protocol, and the second instruction is sent to the second device. In other words, this application, upon determining that a first instruction corresponding to the first protocol needs to be sent to the second device, can determine the second protocol used by the second device and the target data structure of the first instruction under the second protocol; furthermore, feature extraction is performed on the first and second protocols respectively, and a protocol conversion matrix between the first and second protocols is constructed based on the first and second protocol features; based on the protocol conversion matrix and the target data structure, the first instruction can be converted into a second instruction conforming to the second protocol. Based on this application, the problem that smart home systems in related technologies only support communication between devices using a single protocol and cannot directly communicate with devices using other protocols can be solved. Furthermore, communication between devices using different protocols can be achieved through a technical solution that converts instructions of the first protocol into instructions of the second protocol.

[0043] Optionally, the determination of the second protocol adopted by the second device in step S202 includes: if it is determined that the second device has opened the protocol adopted by the second device to the first device, obtaining the device identifier of the second device and determining the second protocol based on the device identifier; if it is determined that the second device has not opened the protocol adopted by the second device to the first device, obtaining the historical communication data packets corresponding to the second device and the operation trajectory of the target object on the second control corresponding to the second device; extracting features from the historical communication data packets and inputting the features corresponding to the historical communication data packets and the operation trajectory into a machine learning algorithm to determine the protocol logic corresponding to the protocol adopted by the second device; generating a protocol template corresponding to the protocol adopted by the second device based on the protocol logic, wherein the protocol template is a description file of the second protocol adopted by the second device.

[0044] It is understandable that smart home systems employ a wide variety of device communication protocols, including open standards such as Matter, ZigBee, and Bluetooth Low Energy (BLE), as well as numerous proprietary protocols. This application proposes a dual-path protocol identification mechanism that employs different strategies to complete protocol identification and adaptation in two scenarios: whether the device uses an open protocol. Specifically:

[0045] 1) Open protocol, direct identification based on device identifier:

[0046] When a second device actively broadcasts its supported communication protocol information to the first device during network access, the first device can directly obtain its protocol type by receiving the unique identifier information reported by the second device. For example, a smart bulb supporting the Matter protocol sends device description information via the standard discovery protocol during network configuration, which includes the manufacturer number, product model, and protocol identifier. After receiving this description information, the first device queries the local protocol registry to match the standard protocol specification corresponding to the model, such as "Matter Lighting Cluster with Thread transport," and automatically loads the protocol's instruction structure, data format, opcode definition, and other metadata to generate the device's protocol description file.

[0047] 2) The protocol is not open; reverse inference of the protocol:

[0048] When the second device fails to declare its protocol information in any standard way, or its protocol is proprietary and undisclosed, an intelligent reverse analysis process is initiated. By observing the device's historical communication behavior and user operation patterns, its communication logic is automatically reconstructed.

[0049] Capture historical communication data packets: During communication between the device and the original control terminal, the wireless communication traffic is recorded through the local network sniffing module. For example, when monitoring a non-standard smart curtain motor of a certain brand, the following typical communication sequence was captured: Time 14:22:03, device receives data packet: 0x07 0x01 0x03 0x00 0x00; Time 14:22:05, device receives data packet: 0x08 0x00 0x00 0x00 0x00; Time 14:22:08, device receives data packet: 0x09 0x02 0x03 0x00 0x00; The data packet length is fixed at 5 bytes, and the sending interval is stable at around 800 milliseconds.

[0050] Align with user operation history: Synchronously acquire user operation records on the control app for this device. For example: 14:22:03, the user clicks the "Open" button; 14:22:05, the user clicks the "Stop" button; 14:22:08, the user clicks the "Close" button.

[0051] By precisely aligning with timestamps, a one-to-one correspondence is established between "operation behavior" and "communication data packet": "Open" → 0x07 0x01; "Stop" → 0x08; "Close" → 0x09 0x02;

[0052] Feature extraction and machine learning modeling: Multidimensional feature extraction of communication data packets, including: instruction byte length distribution; opcode value range and semantic relevance; fixed pattern of data fields (e.g., the second byte is for direction control); instruction sending time interval, frequency, and retry mechanism.

[0053] Simultaneously, user operation sequences (such as "on → stop → off") are used as behavioral context input. These features are integrated into a structured input vector and fed into a fusion machine learning model (such as a combination of temporal neural networks and decision trees). The fusion machine learning model automatically learns and outputs the protocol logic rules: "This device uses a 5-byte fixed-format instruction. The first byte is the opcode, the second byte controls the direction, and three operations form a complete on / off cycle. There must be an interval of 750 to 850 milliseconds between instructions; otherwise, the device ignores the instruction."

[0054] Generate Protocol Template: Based on the model output, generate a structured protocol description file for the device, which serves as the sole basis for subsequent control. This template includes: protocol name (e.g., "Phantom-Curtain-Private"); transport layer type (e.g., ZigBee 3.0); command set (on, stop, off and their corresponding byte sequences); timing constraints (minimum / maximum interval); and expected response (e.g., "The device should return an acknowledgment frame within 200 milliseconds").

[0055] Optionally, the determination of the target data structure of the first instruction under the second protocol in step S202 above includes: when it is determined that the target object sends the first instruction to the second device through the first control in the first device, dividing the control image corresponding to the first control into multiple image blocks, and mapping each image block to a target vector; adding position encoding to the target vector to generate an augmented vector containing the spatial position information of each image block in the control image; performing feature extraction on the multiple augmented vectors through a target encoder to obtain a global feature vector corresponding to the first control, and inputting the global feature vector into a classification model to obtain the control type corresponding to the first control; mapping the first control to a target semantic label according to the control type; and determining the target data structure in a semantic protocol mapping table, wherein the semantic protocol mapping table is used to indicate the instruction format of the semantic label corresponding to each control under each protocol.

[0056] It is understood that the embodiments of this application can automatically identify the function type of the control image by analyzing its visual features, map it to a unified semantic tag, and finally associate it with the standard instruction format under the target protocol. Specifically:

[0057] Image Segmentation and Vector Mapping for Controls: When a user clicks on a control used to control a second device (such as a smart light bulb) on the interface of a first device (such as a smart gateway), the image region of that control is captured. This image may come from any brand of app and could be a circular button, slider, icon, or text label. The control image is divided into multiple regular image patches, each representing a local visual region in the image, such as the border of a button, the icon area of ​​a switch, or the scale of a slider. Each image patch is independently converted into a high-dimensional numerical vector, which encodes visual information such as the color distribution, texture features, and edge structure of the region, forming the original set of image patch vectors.

[0058] For example, a circular "light-on" button from brand A has its image divided into 64 16×16 pixel blocks. Each block generates a 256-dimensional vector describing its visual content.

[0059] Position encoding enhances spatial awareness: To preserve the spatial positional relationship of image blocks within the original control image, position encoding is added to each image block vector. The position encoding is a fixed vector corresponding to the image block's position, reflecting its row and column coordinates in the image, such as "top left corner," "center-right," or "bottom edge." After adding position encoding, the original image block vector is upgraded to an enhanced vector, containing both visual content information and spatial structure information. This process enables the system to distinguish between a "switch located at the top" and a "switch located at the bottom," recognizing their functional semantic differences within the interface even if they appear similar.

[0060] For example, in the same brand's app, there are two identical circular buttons: one located at the top of the screen to control the main light, and the other at the bottom to control the night light.

[0061] Global Feature Extraction and Control Type Recognition: All augmentation vectors are input into the target encoder. This encoder, through a self-attention mechanism, analyzes the global semantic relationships between image patches, extracts the overall structural features of the control, and outputs a unified global feature vector. The global feature vector encodes the "functional semantics" of the entire control, such as "this is a triggerable switch," "this is an adjustable slider," or "this is a command button with a text label." The global feature vector is then fed into a pre-trained classification model (such as a multi-layer neural network). Based on thousands of control samples in the training data, the model determines the functional type of the control, such as: Switch; Slider; Button; Selector; or Label. Processing a slider control from brand B, after extraction by the encoder, the classification model outputs "slider" with an accuracy exceeding 98%. This result is consistent with the user's actual operation behavior; sliding the control adjusts the brightness.

[0062] Semantic Tag Mapping and Protocol Command Format Generation: Based on the identified control type, the control is mapped to a unified semantic tag. For example: all "switch" type controls are mapped to the semantic tag "Power"; all "slider" type controls are mapped to the semantic tag "Brightness"; all "temperature adjustment" type controls are mapped to the semantic tag "Temperature". These semantic tags are a unified expression across brands and apps, independent of the original interface style. Further, the semantic protocol mapping table is queried. This table is a pre-built structured database that records the standard command format for each semantic tag under different communication protocols. When it is recognized that the user clicked the "Power" semantic tag and the target device uses the ZigBee protocol, the corresponding command format "0x01" is automatically extracted from the mapping table, and the final control command is generated and sent to the target device.

[0063] Optionally, after sending the second instruction to the second device in step S208 above, the method further includes: if the sending status corresponding to the second instruction is determined to be a sending failure, obtaining the historical communication data packets corresponding to the second device, and extracting the protocol version change feature corresponding to the second protocol from the historical communication data packets using a multidimensional feature extraction algorithm; matching the protocol version change feature with the preset protocol version change feature in the protocol version change library to determine the protocol change status corresponding to the second protocol; if the protocol change status indicates that the protocol version of the second protocol has been changed to the first protocol version, obtaining the historical protocol version usage log corresponding to the second device, and determining the device compatible version list corresponding to the second device based on the historical protocol version usage log; determining the target protocol version of the second protocol in the device compatible version list, and sending a protocol version rollback instruction to the second device to roll back the second protocol used by the second device from the first protocol version to the target protocol version; and sending the second instruction to the second device using the second protocol of the target protocol version.

[0064] Understandably, when a device communication failure is detected, its historical communication data is automatically analyzed to identify protocol version change traces. Combined with the device's historical usage records, the system intelligently determines its compatible version and triggers a protocol version rollback command, restoring the device to normal operation without manual intervention. Specifically:

[0065] Communication failure triggers automatic diagnostics: When a first device sends a control command (such as "increase the temperature") to a second device but does not receive the expected response or receives an error feedback, the command transmission is deemed to have failed. In this case, instead of immediately reporting an error or prompting the user, the protocol version diagnostic process is automatically initiated. For example: A smart thermostat purchased three years ago could originally be controlled normally through a gateway. After a system upgrade, the user issued a command to "set to 22℃," but the thermostat did not respond. The system detected that the command was not acknowledged, determined it to be a communication failure, and immediately initiated version diagnostics.

[0066] Multidimensional feature extraction of historical communication data: Collect all historical communication data packets of the device within a period of time before the failure, including records of successful and failed command interactions. Using a multidimensional feature extraction algorithm, extract structural change features related to the protocol version from this data, including: changes in command header format (e.g., the protocol header length changed from 4 bytes to 6 bytes); expansion or obsolescence of the command opcode set (e.g., previously only supporting 0x01–0x03, now adding 0x0A–0x0F); addition or removal of data fields (e.g., the "wind speed mode" field was absent in the original version, but appears in the new version); changes in timestamp format (e.g., from Uniform Time Division System (Unix) timestamps to millisecond-level timestamps); and adjustments to the response confirmation mechanism (e.g., from acknowledgment (ACK) confirmation to encrypted signature confirmation). These features constitute the digital fingerprint of the protocol version evolution. For example, analysis of historical data packets from the thermostat revealed that its instruction structure changed from "0x01 + temperature value (1 byte)" to "0x07 + temperature value (2 bytes) + fan speed mode (1 byte)," and a new protocol version number field (0x0003) was added, which was absent in older versions. This indicates that the protocol has been upgraded from v1.1 to v2.0.

[0067] Intelligent matching of protocol change status: The extracted protocol version change features are compared with a locally pre-built protocol version change database. This database is an authoritative knowledge base maintained by the system, recording the specific changes of mainstream protocols between versions. Through feature matching, it is determined that the current protocol version has been upgraded from an older version to a specific new version (such as v2.0), and the device is marked as being in a "protocol version incompatible" state.

[0068] Device Compatibility Version Tracing: The historical protocol version usage logs of the device are retrieved. These logs record all protocol versions supported and successfully communicated by the device in its past operations. Log data originates from the handshake process records of each successful communication or from the manufacturer's pre-configured firmware version binding information. Analyzing these logs generates a device compatibility version list, listing all verified and functionally compatible protocol versions. This list is arranged in reverse chronological order, prioritizing the most recently stable version. For example, the thermostat's historical logs show: 2023-05-18: Successfully used protocol v1.1; 2023-08-03: Successfully used protocol v1.2; 2024-01-10: Initial connection attempt to v2.0 failed; 2024-02-15: Still communicating stably with v1.2. Based on this, its compatible version list is determined to be: v1.2, v1.1, with v1.2 being the most recently stable version.

[0069] Automatic rollback to a compatible version and retry: Select the optimal target version from the list of compatible versions (which can be the most recent stable version) and generate a protocol version rollback command. The protocol rollback command is a dedicated control frame conforming to the original protocol specification, used to notify the device to roll back its protocol stack to the specified version. The protocol rollback command is sent to the device through a reliable communication channel. After receiving it, the device automatically performs an internal firmware protocol stack switch, restarts the communication module, and enters the target version's working state. The original control command is resent. Because the device has switched to a compatible version, the command is parsed and executed normally. For example: Sending a rollback command to a temperature controller: "SetProtocolVersion = 1.2". The temperature controller responds and resets its protocol stack. After 5 seconds, the system sends the "Set temperature 22℃" command again. The temperature controller successfully receives and executes it, the temperature begins to rise, and the system resumes normal control.

[0070] Optionally, after sending the second instruction to the second device in step S208 above, the method further includes: acquiring the current signal of the second device during the execution of the second instruction, and extracting the current fingerprint feature corresponding to the current signal; acquiring the state feedback information of the second device after executing the second instruction; encoding the current fingerprint feature and the state feedback information into first feature vectors respectively, and concatenating multiple first feature vectors into a fused feature vector; inputting the fused feature vector into a deep neural network model to determine the execution state of the second device executing the second instruction, wherein the execution state includes one of the following: correct execution, incorrect execution.

[0071] It is understood that this application embodiment constructs a high-dimensional feature representation by synchronously collecting electrical characteristics and functional response information of the device during operation, thereby achieving accurate judgment of the authenticity of instruction execution. Specifically:

[0072] Current signal acquisition and fingerprint feature extraction: When the device executes control commands (such as "start heating"), the embedded current sensor acquires the current waveform signal in real time during its operation. The current waveform signal records the electrical dynamic behavior of the device throughout the entire process, from standby to operation, power changes, startup impact, and stable operation. Time-frequency domain analysis is performed on the current signal to extract device-specific current fingerprint features, including but not limited to: the peak and steady-state values ​​of the current amplitude change curve; the duration and intensity of the surge current at startup; frequency fluctuations during the operating cycle (such as changes in motor speed); the slope and stability of energy consumption; and the distribution characteristics of current harmonic components.

[0073] For example, when a smart electric heater executes the "turn on heating" command, the current rapidly increases from 0.1A to 3.5A and remains stable for 15 seconds, during which slight periodic fluctuations occur (corresponding to the temperature control cycle of the heating element). This current trajectory is encoded into a set of 256-dimensional electrical feature vectors, characterizing its actual heating behavior.

[0074] Status feedback information acquisition: Receive functional status feedback information from the device. The functional status feedback information can be reported by the device's internal sensors or firmware, including: whether the device reports "on" or "off"; whether the current temperature has reached the target value; whether the operating mode has been switched to the specified level; and whether fault codes exist (such as overheating or communication timeout).

[0075] Multimodal feature encoding and fusion: The two types of heterogeneous information mentioned above are respectively structured and encoded: the current fingerprint feature is input into the feature extraction module to generate the first feature vector A, representing the actual physical action of the device; the state feedback information is parsed into structured parameters (such as Boolean values, enumeration types, and numerical ranges) to generate the first feature vector B, representing the logical result claimed by the device. These two vectors are concatenated by channel to form a fused feature vector with dimensions A+B.

[0076] Deep Neural Network Decision Making and Execution State Judgment: The fused feature vector is input into a pre-trained deep neural network model. This model has been trained on millions of sets of real device operation data, learning multimodal pattern distributions under various states, including normal execution, partial execution, complete failure, and spurious responses. The output of the pre-trained deep neural network model is an execution state classification result, including the following three possibilities: Correct execution: Current behavior and state feedback are highly consistent, and physical actions match logical statements; Incorrect execution: Current behavior is abnormal or missing, but the device still reports success (e.g., idling, spurious response); No execution: No change in current, no response in state feedback, or timeout. The output of the pre-trained deep neural network model is a probability distribution, and the final state is determined based on the highest probability value.

[0077] Optionally, after constructing the protocol conversion matrix between the first protocol and the second protocol based on the feature matching result in step S206, the method further includes: determining whether a preset error conversion library contains an error conversion record corresponding to the protocol conversion matrix; if the preset error conversion library contains the error conversion record, sending a broadcast request to each node to obtain the repair rule corresponding to the protocol conversion matrix, wherein multiple nodes are used to construct the protocol conversion matrix; and repairing the protocol conversion matrix according to the repair rule.

[0078] Understandably, when a protocol conversion error is detected, a broadcast request can be used to automatically retrieve verified repair rules from multiple distributed nodes in the network, dynamically correcting the erroneous conversion logic and achieving real-time self-healing of protocol conversion capabilities. Specifically:

[0079] Automatic matching of error conversion records: During the execution of protocol command conversion, if the output result does not match the expected behavior (such as device non-response, abnormal status, or command distortion), the input command, source protocol format, target protocol output, context environment, and other information of this conversion are recorded to generate an error conversion record. This record is compared to a locally preset error conversion library: a standardized error mode database compiled from historical failure cases. This database contains known and confirmed conversion failure modes, such as: ZigBee command [0x02, 0x10] → incorrectly mapped to BLE command [0x07] (should be [0x08]); Matter command "LevelControl.MoveToLevel (level control, move to specified brightness level)" is incorrectly parsed as "OnOff.Toggle (on / off control, state switching)"; "fan speed = medium" of a certain model of thermostat is mistranslated as "fan speed = high" after conversion; if the current error conversion record completely matches a record in the library, the subsequent self-healing process is triggered.

[0080] Broadcast query for repair rules in a distributed network: Broadcast a request to multiple distributed nodes in the network that are involved in the protocol conversion collaboration, asking: "Do you know the repair plan for this faulty conversion?" These nodes are not central servers, but any intelligent devices or gateways in the network that have the ability to cache protocol knowledge.

[0081] Local dynamic repair of the protocol conversion matrix: Valid repair rules are confirmed and dynamically injected into the local protocol conversion matrix, overriding the original erroneous mapping relationships and completing real-time self-correction of the conversion logic. After repair, the original instruction is immediately retried to verify whether normal operation has been restored.

[0082] Optionally, before determining the second protocol adopted by the second device in step S202 above, the method further includes: upon receiving a voice command sent by the target object, extracting the keywords and semantic structure corresponding to the voice command, and converting the voice command into target control logic based on the keywords and semantic structure; matching the target control logic with the device interface corresponding to each device to generate an instruction sequence containing a target execution device, wherein the instruction sequence contains: the first instruction, and the target execution device contained in the first instruction is: the second device.

[0083] It is understood that the embodiments of this application can also parse the user's voice commands, extract the core semantic elements, convert them into executable control logic, and intelligently match the functional interfaces of each device to automatically generate an ordered sequence of commands containing multiple target devices. Specifically:

[0084] Keyword and semantic structure extraction from voice commands: After receiving a voice command from a user, the system first performs speech recognition to obtain the original text statement. Then, using natural language processing (NLP) technology, the statement undergoes lexical analysis and syntactic parsing to identify the core keywords and logical relationships. Keywords: These are entity words in the statement that carry specific device actions or parameters, such as "turn off the lights," "lower the temperature," "turn on the humidifier," "26℃," "fan speed level one," etc. Semantic structure: This refers to the temporal, conditional, parallel, or causal relationships between these keywords, such as the logical framework constructed by conjunctions like "and," "at the same time," "when," "equal to," and "pattern."

[0085] Voice commands are converted into target control logic: The extracted keywords and semantic structure are combined into a structured target control logic. The structure of the target control logic can be: [Scene Name] = [Action 1] + [Action 2] + ... + [Action N]; each action includes: device type + operation type + parameter value.

[0086] Intelligent matching of control logic and device interfaces: The system iterates through all connected smart devices in the current network and binds each action in the target control logic to the most matching device interface based on the device's functional interface definition. Each device has clearly defined functional capabilities registered in the system. For example: Smart light A: supports "on / off control" and "brightness adjustment"; Air conditioner B: supports "temperature setting," "fan speed adjustment," and "mode switching"; Humidifier C: supports "on / off control" and "humidity setting." Matching is completed automatically based on the semantic consistency between the action (e.g., "off," "temperature adjustment") and the device's capabilities, requiring no manual configuration.

[0087] Generate an instruction sequence containing the target execution device: After matching, all actions are converted into low-level protocol instructions that the device can directly execute, and then combined into an ordered instruction sequence according to semantic order, where each instruction explicitly points to an execution device. The instruction sequence includes: instruction number, target device address, protocol type (such as Matter, ZigBee, BLE), and the converted specific byte instructions.

[0088] To better understand the process of sending the above instructions, the flow of the above instructions sending method will be described below in conjunction with optional embodiments, but this is not intended to limit the technical solutions of the embodiments of this application.

[0089] Several technical challenges exist in the field of smart home and related cross-platform device interaction. First, different devices use different communication protocols, such as Matter, ZigBee, and BLE. Protocol conversion and real-time parsing are difficult, resulting in significant latency in bidirectional command conversion, low compatibility, and hindering efficient communication between devices. Second, the UI controls of different brands vary greatly, making the adaptation to map them to unified semantic tags a massive undertaking. Third, proprietary protocols are subject to manufacturer technical blockades, making them difficult to crack and affecting device interoperability. Additionally, protocol version updates may render older devices unusable, the rate of erroneous operations during protocol execution is high, protocol parsing vulnerabilities affect system robustness, and users face difficulties in customizing protocol semantic rules for personalized control.

[0090] Related technical solutions include: Manual UI control adaptation: Previously, for UI controls from different brands, the function of each control was analyzed manually and mapped to a unified semantic tag. This method is labor-intensive, inefficient, and prone to errors, making it unsuitable for adapting to a large number of different brand devices; Unable to crack proprietary protocols: Faced with manufacturer proprietary protocols, some systems lack effective cracking methods and can only rely on interfaces provided by the manufacturer. If manufacturers do not provide or restrict interfaces, interoperability between devices cannot be achieved, locking users into the ecosystem of specific brand devices; Fixed protocol version solutions: When protocol versions are updated, older devices often become unusable due to incompatibility with the new protocols, and traditional systems lack an effective mechanism to solve this problem, resulting in wasted device resources; Simple protocol verification solutions: Some systems only verify protocol execution through device status feedback, without considering multimodal information such as current fingerprints, leading to a high rate of misoperation and inaccurate judgment of protocol execution status; Centralized protocol management solutions: Some systems use centralized protocol management, and when the central node malfunctions or protocol parsing vulnerabilities occur, the entire system may be paralyzed, resulting in poor system robustness; No custom protocol semantics solutions: Systems in related technologies do not support user-defined protocol semantic rules, and users can only operate devices according to preset control logic, unable to create personalized control logic according to their own needs.

[0091] However, the technical solutions in these areas suffer from the following problems: Manual UI control adaptation is labor-intensive, inefficient, and prone to errors, hindering unified control and management of devices from different brands and increasing development costs and time. The inability to crack proprietary protocols locks users into specific brand ecosystems, limiting device choice and hindering the opening and development of the smart home market. Fixed protocol versions render older devices unusable due to protocol incompatibility, wasting resources and impacting long-term user experience. Simple protocol verification has a high error rate, failing to accurately determine protocol execution, potentially leading to device malfunctions, affecting user experience and device lifespan. Centralized protocol management systems have poor robustness; central node failures or protocol parsing vulnerabilities can paralyze the entire system, reducing reliability. Customizable solutions without protocol semantics cannot meet users' personalized control needs, limiting the flexibility and user autonomy of smart home systems.

[0092] To address the aforementioned issues, optional embodiments of this application employ a multi-protocol dynamic translation middleware architecture to construct a protocol conversion matrix engine. This engine parses multiple communication protocols in real time, achieving low-latency, high-compatibility bidirectional instruction conversion and improving communication efficiency and compatibility between devices. Furthermore, leveraging a UI control semantic generalization mapping algorithm and based on ViT's cross-App control standardization technology, UI elements from different brands are mapped to unified semantic tags, significantly reducing adaptation workload and enabling unified control of devices from different brands.

[0093] Figure 3 This is an architecture diagram of a cross-platform protocol parsing and adaptation system according to an optional embodiment of this application, such as... Figure 3 As shown:

[0094] Cross-platform protocol parsing and adaptation systems include:

[0095] 1) Multi-protocol dynamic translation middleware architecture: connects devices with different protocols (e.g.: Figure 3 The device 1 (using protocol 1) and device 2 (using protocol 2) use a protocol conversion matrix engine to parse multiple communication protocols in real time, enabling bidirectional conversion of instructions between devices, reducing latency and improving compatibility.

[0096] 2) Protocol Conversion Matrix Engine: Constructs conversion matrices based on the characteristics of different protocols to achieve efficient conversion between protocols.

[0097] 3) Bidirectional command conversion and communication module: responsible for bidirectional command conversion and communication between devices with different protocols, ensuring accurate information transmission between devices.

[0098] Specifically, the protocol conversion matrix engine extracts features from 23 supported communication protocols (e.g., Matter (Unified Connectivity Standard), ZigBee (Cellular Protocol), BLE (Bare-Low Energy Bluetooth), etc.), analyzing their data format, instruction structure, communication flow, and other characteristics to build a protocol feature library. For any two protocols (denoted as Protocol A and Protocol B), by comparing the features in the feature library, the mapping relationship between Protocol A and Protocol B is found, and a protocol conversion matrix is ​​constructed. When converting instructions from protocol A to protocol B, the instructions from protocol A will be... According to the transformation matrix Perform the conversion to obtain the instructions for Protocol B. During real-time parsing, a caching mechanism is used to store frequently used conversion results, accelerating the conversion speed. For example, when frequently performing ZigBee to BLE instruction conversions, the most recent conversion results are cached. When the same or similar instructions are encountered again, the conversion results are directly retrieved from the cache, achieving a bidirectional instruction conversion latency of <50ms. Simultaneously, through testing and feedback on a large number of devices with different protocols, the protocol conversion matrix is ​​continuously optimized, improving the compatibility rate to 98%.

[0099] For example: Suppose device 1 uses the ZigBee protocol to send the command "turn on the light", with the command format [0x01, 0x02, 0x03]. Device 2 uses the BLE protocol. The protocol conversion matrix engine constructs a ZigBee to BLE conversion matrix through feature analysis: According to the conversion matrix, the ZigBee command [0x01, 0x02, 0x03] is converted into the BLE protocol command [0x05, 0x06, 0x07], which device 2 can then receive and execute. In practical applications, if device 1 sends the "turn on the light" command to device 2 multiple times in a short period of time, the middleware architecture can quickly retrieve the converted BLE command from the cache, reducing conversion time and meeting low latency requirements.

[0100] In summary, the protocol conversion matrix engine can achieve real-time parsing of multiple protocols and efficient bidirectional instruction conversion through techniques such as feature extraction, matrix construction, and cache optimization, thereby improving the efficiency and compatibility of communication between devices.

[0101] 4) User Interface (UI) Control Semantic Generalization Mapping Module: Based on the Vision Transformer (ViT) cross-application (App) control standardization module, this module analyzes UI elements from different brands, extracts features, and generates unified semantic tags, reducing adaptation workload. The unified semantic tag generation module maps UI elements from different brands to unified semantic tags, facilitating unified control. Specifically:

[0102] The ViT-based cross-app control standardization module takes images of UI elements from different brands as input. It preprocesses these images, adjusting image size and normalizing them. The preprocessed images are then divided into multiple patches, each linearly mapped to a vector, with positional encoding added to preserve spatial information. These vectors are input into a Transformer encoder, where multi-head self-attention and a multilayer perceptron are used for feature extraction to obtain feature representations of the UI elements. The unified semantic label generation module then classifies these UI element feature representations using a pre-trained classification model (e.g., Support Vector Machine (SVM) or neural network) into unified semantic labels, such as "switch" or "slider." For example, after the above processing, the classification model maps switch UI element images from different brands to the semantic label "switch," achieving standardization of UI elements from different brands and reducing adaptation workload by 80%.

[0103] For example: Images of switch UI elements from brand A and brand B are collected. After preprocessing, they are divided into patches and mapped to vectors. Position encoding is added before inputting them into a Transformer encoder. The Transformer encoder extracts the feature representation of the switch image from brand A. Feature representation of brand B switch image The unified semantic tag generation module utilizes a pre-trained classification model to... and After categorization, both switches were labeled with the semantic tag "switch". In subsequent unified control, these two switches from different brands can be treated as controls with the same semantic meaning and operated accordingly, significantly reducing the workload of adapting UI elements from different brands separately.

[0104] In summary, the algorithm of the optional embodiments of this application is based on Vi to extract features and standardize the classification of UI elements of different brands, and automatically maps them to unified semantic tags, reducing manual intervention and improving adaptation efficiency.

[0105] 5) Private Protocol Reverse Engineering Module: The traffic capture and UI operation trajectory analysis module acquires device communication traffic and user UI operation trajectory data. The protocol logic reverse engineering and control template generation module reverse-engineers the protocol logic based on this data and automatically generates control templates. Specifically:

[0106] The traffic capture and UI operation trajectory analysis module captures device communication traffic data using network packet capture tools, while simultaneously recording the user's operation trajectory on the device UI, including information such as operation sequence and operation time. The protocol logic reverse engineering and control template generation module analyzes the traffic data, extracting packet characteristics such as packet size, sending frequency, and data content. Combined with UI operation trajectory information, data mining and machine learning algorithms, such as decision trees and cluster analysis, are used to reverse engineer the protocol logic. For example, analysis reveals that when a user performs the "open device" operation on the UI, a set of data packets of a specific size and content is sent within a specific time interval, thus inferring the corresponding protocol command. Based on the reverse-engineered protocol logic, a control template is automatically generated, containing the protocol commands and parameter settings corresponding to various operations.

[0107] For example, when testing a proprietary protocol device, a traffic capture tool captured a series of communication traffic events and recorded user actions such as opening and closing the device on the UI. Analysis of the traffic data revealed that each time the device was opened, a 1024-byte data packet with a specific header was sent. Combined with the operation log, this data packet was determined to correspond to the "open device" operation. Through multiple operations and data analysis, the complete protocol logic was deduced, and a control template containing operations such as opening, closing, and parameter adjustment was generated, enabling other devices to communicate with the proprietary protocol device according to the template.

[0108] In summary, the toolchain in the optional embodiments of this application combines traffic packet capture and UI operation trajectory analysis, and uses data mining and machine learning algorithms to reverse-engineer protocol logic and generate control templates, providing a new approach to achieve device interconnection and interoperability.

[0109] 6) Intelligent Protocol Version Downgrade Module: The protocol change feature detection module monitors protocol version change characteristics. The version rollback decision module determines whether a version rollback is needed based on the detection results and identifies which compatible version to roll back to, ensuring the continued availability of older equipment. Specifically:

[0110] The protocol change feature detection module continuously monitors protocol data and extracts features related to protocol version changes, such as changes in protocol header format, instruction set updates, and additions or deletions of data fields. These protocol version change-related features are compared with a predefined protocol change feature library to determine whether the protocol has changed and the type of change. Based on the detection results, the version rollback decision module, if it detects a protocol change and the current device is incompatible with the new protocol, analyzes the device's historical protocol version usage records and compatibility information to determine the most recent compatible version. For example, if the device has historically used versions 1.0, 1.1, and 1.2, and the current version is 1.3 but is incompatible, analysis shows that version 1.2 has the best compatibility with the current device, so a rollback to version 1.2 is decided. Then, a version rollback command is sent to the device to ensure the continued control availability of older devices.

[0111] For example, a device originally used protocol version 1.1. The protocol change feature detection module detected a change in the protocol header format. After comparing with the feature library, it was determined that the protocol had been upgraded to version 1.3, and the device was incompatible. The version rollback decision module queried the device's historical version records and found that version 1.2 was compatible. Therefore, it sent a command to the device to roll back the protocol version to version 1.2, and the device resumed normal operation, ensuring the continued availability of the old device.

[0112] In summary, the optional embodiments of this application ensure the continued usability of older devices by dynamically detecting protocol change features and automatically reverting to a compatible version, thereby reducing device obsolescence and resource waste and improving the long-term user experience of the smart home system.

[0113] 7) Multimodal Protocol Verification Module: The current fingerprint detection module acquires the device's current fingerprint information, the device status feedback module acquires the device status information, and the multimodal fusion verification module merges the two to verify protocol execution and reduce the error rate. Specifically:

[0114] The current fingerprint detection module acquires the current signal from the device during protocol command execution via a current sensor. It preprocesses the current signal, performing operations such as filtering and amplification, to extract current fingerprint features, such as current amplitude changes and current frequency. The device status feedback module acquires the device's status information after command execution, such as whether the device has been successfully powered on and its current operating parameters. The multimodal fusion verification module fuses the current fingerprint features and device status information, employing machine learning algorithms (such as neural networks) for training and verification. First, the current fingerprint features and device status information are encoded into feature vectors. and Then they are concatenated into a fused feature vector: .Will The data is input into a trained neural network model, which outputs a verification result of the protocol execution, determining whether the protocol execution was correct. Through this multimodal fusion verification, the error rate was reduced from 12% to 1.2%.

[0115] For example, when the device executes the protocol command to "adjust the temperature to 25℃", the current fingerprint detection module acquires the current signal during the temperature adjustment process and extracts current fingerprint features such as the current amplitude gradually stabilizing at a specific value. The device status feedback module receives the current temperature display as 25℃, indicating that the device has successfully executed the command. The multimodal fusion verification module fuses the current fingerprint features and device status information into a feature vector, which is then input into a neural network model for verification. The neural network model outputs a verification result indicating that the protocol execution is correct. If the current fingerprint features and device status information do not match, such as an abnormal current amplitude but the device displays successful temperature adjustment, the model outputs that the protocol execution is incorrect, thus avoiding erroneous operations and reducing the error rate.

[0116] In summary, the optional embodiments of this application integrate current fingerprint and device status feedback modal information for protocol execution verification, utilize machine learning algorithms to improve verification accuracy, reduce the error rate, and improve the reliability and stability of device operation.

[0117] 8) Protocol Fragment Self-Healing Network Architecture: Distributed protocol knowledge-sharing nodes store and share protocol knowledge. A local protocol parsing vulnerability detection and remediation module utilizes this knowledge to detect and automatically remediate local protocol parsing vulnerabilities, improving system robustness. Specifically:

[0118] The distributed protocol knowledge-sharing nodes employ distributed storage technology, distributing protocol-related knowledge (such as protocol parsing rules, protocol conversion methods, and solutions to common problems) across various nodes. A local protocol parsing vulnerability detection and remediation module periodically checks the local protocol parsing performance, comparing the parsing results with expected results to determine the presence of parsing vulnerabilities. If a vulnerability is found, the local protocol parsing vulnerability detection and remediation module sends a request to the distributed protocol knowledge-sharing nodes to obtain relevant remediation knowledge. For example, node A, while parsing a protocol data packet, finds that the parsing result does not conform to the protocol specification, confirming a parsing vulnerability. It sends a request to other nodes; node B, which stores a remediation method for this vulnerability, returns the remediation method to node A. Node A automatically remediates the protocol parsing vulnerability based on the remediation method, improving system robustness by three times.

[0119] For example, in a smart home system, a node is responsible for parsing ZigBee protocol data packets. During one parsing process, an error is found in a field of the data packet, indicating a parsing vulnerability. This node broadcasts a request for repair information to a distributed protocol knowledge-sharing node. Other nodes, upon receiving the request, check their stored knowledge. One node, containing a repair rule for this ZigBee protocol field parsing error, sends the rule to the vulnerable node. The vulnerable node adjusts its parsing algorithm according to the repair rule, successfully fixing the vulnerability, ensuring normal protocol parsing, and improving the robustness of the entire system.

[0120] In summary, the optional embodiments of this application employ distributed protocol knowledge-sharing nodes and an automatic vulnerability detection and repair mechanism to improve the system's ability to respond to local protocol parsing vulnerabilities, enhance system robustness, and ensure stable system operation.

[0121] 9) Semantic Protocol Dynamic Extension Module: The user-defined semantic rule input module receives user-inputted semantic rules, and the natural language processing and logic generation module converts the natural language into executable control logic to meet the user's personalized control needs. Specifically:

[0122] The user-defined semantic rule input module receives custom semantic rules input by the user in natural language, such as "sleep mode = 26℃ + fan speed level 1". The natural language processing and logic generation module performs lexical and syntactic analysis on the natural language, extracting keywords and semantic structures. For example, for the above rule, keywords such as "sleep mode", "26℃", and "fan speed level 1" and the logical relationships between them are extracted. Then, this semantic information is converted into executable control logic. By matching with the device's functional interfaces, specific control command sequences are generated. For example, the interface for controlling temperature is found and the temperature is set to 26℃; the interface for controlling fan speed is found and the fan speed is set to level 1. These commands are combined into the control logic corresponding to "sleep mode", allowing users to create new control logic through natural language.

[0123] For example, a user inputs "Wake-up mode = open curtains + adjust light brightness to 80% + play music". The natural language processing and logic generation module analyzes the input, extracting information such as "wake-up mode", "open curtains", "adjust light brightness to 80%", and "play music". Then, it matches the opening interface of the curtain control device, the brightness adjustment interface of the light control device, and the playback interface of the music playback device to generate the corresponding control command sequence, realizing the customized control logic of "wake-up mode" and meeting the user's personalized control needs.

[0124] In summary, the optional embodiments of this application provide user-defined semantic rule input and natural language processing mechanisms to convert natural language into executable control logic, meet users' personalized control needs, and enhance the flexibility and user autonomy of the smart home system.

[0125] 10) Unified Control Interface Module: Integrates unified semantic tags, control templates, version rollback instructions, protocol verification results, vulnerability repair information, and custom control logic to provide a unified control interface for devices, enabling unified and efficient control of devices.

[0126] The optional embodiments of this application include the following processes:

[0127] 1) Multi-protocol dynamic translation process;

[0128] Figure 4 This is a schematic diagram of a multi-protocol dynamic translation process according to an optional embodiment of this application, such as... Figure 4 As shown:

[0129] Device 1 and Device 2 each send their respective protocol-specific instructions (i.e., Protocol 1 instructions and Protocol 2 instructions) to the multi-protocol dynamic translation middleware architecture. The protocol conversion matrix engine parses the protocol and converts the instructions, returning the converted instructions to the multi-protocol dynamic translation middleware architecture. The middleware architecture then sends the converted instructions to the corresponding devices (converting Protocol 2 instructions to Device 1 and Protocol 1 instructions to Device 2), thus enabling instruction conversion and communication between devices using different protocols.

[0130] 2) UI space semantic generalization mapping process;

[0131] Figure 5 This is a schematic diagram of the UI space semantic generalization mapping process according to an optional embodiment of this application, such as... Figure 5 As shown:

[0132] Device 1 and Device 2 provide their respective UI element images to the UI control semantic generalization mapping module. The ViT-based cross-App control standardization module processes the UI element images and extracts feature representations, which are then sent to the unified semantic tag generation module. The unified semantic tag generation module generates unified semantic tags and returns them to the UI control semantic generalization mapping module, which then sends them to Device 1 and Device 2, thereby achieving semantic standardization of UI elements from different brands.

[0133] 3) Reverse engineering process of proprietary protocols;

[0134] Figure 6 This is a schematic diagram of the reverse engineering process of the proprietary protocol according to an optional embodiment of this application, such as... Figure 6 As shown:

[0135] Device 1 and Device 2 generate communication traffic and UI operations. The traffic capture and UI operation trajectory analysis module of the proprietary protocol reverse engineering module collects the communication traffic and UI operations and sends them to the protocol logic reverse engineering and control template generation module. The protocol logic reverse engineering and control template generation module reverse-engineers the protocol logic and generates a control template, which is returned to the proprietary protocol reverse engineering module. The proprietary protocol reverse engineering module then sends the template to Device 1 and Device 2, thus overcoming the limitations of the proprietary protocol.

[0136] 4) Intelligent protocol version downgrade process;

[0137] Figure 7 This is a schematic diagram of the protocol version intelligent downgrade process according to an optional embodiment of this application, such as... Figure 7 As shown:

[0138] Device 1 and Device 2 report their protocol operation status to the protocol version intelligent degradation module. The protocol change feature detection module detects protocol changes and sends the change information to the version rollback decision module. If the decision requires a version rollback, the protocol version intelligent degradation module sends a version rollback command to Device 1 and Device 2 to ensure the continued availability of older devices.

[0139] 5) Multimodal protocol verification process;

[0140] Figure 8 This is a schematic diagram of the multimodal protocol verification process according to an optional embodiment of this application, such as... Figure 8 As shown:

[0141] After executing the protocol instructions, Device 1 and Device 2 send current and device status information back to the multimodal protocol verification module. The current fingerprint detection module acquires the current fingerprint, and the device status feedback module acquires the device status. Both send the relevant information to the multimodal fusion verification module, which returns the verification result to the multimodal protocol verification module. The multimodal protocol verification module then feeds the result back to Device 1 and Device 2, reducing the error rate.

[0142] 6) Protocol fragment self-healing process;

[0143] Figure 9 This is a schematic diagram of the protocol fragment self-healing process according to an optional embodiment of this application, such as... Figure 9 As shown:

[0144] Device 1 and Device 2 report the protocol parsing status to the protocol fragment self-healing network module. The local protocol parsing vulnerability detection and repair module detects the existence of parsing vulnerabilities. If a vulnerability is found, it requests repair knowledge from the distributed protocol knowledge sharing node, performs vulnerability repair after obtaining the repair knowledge, and feeds back the repair results to Device 1 and Device 2 to improve system robustness.

[0145] 7) Semantic protocol dynamic extension process;

[0146] Figure 10 This is a schematic diagram of the dynamic extension process of the semantic protocol according to an optional embodiment of this application, such as... Figure 10 As shown:

[0147] The user-defined semantic rule input module of the semantic protocol dynamic extension module acquires user-defined rules and sends them to the natural language processing and logic generation module. This module generates control logic and returns it to the semantic protocol dynamic extension module, which then sends the control logic to the applicable devices 1 and 2 to meet the user's personalized control needs.

[0148] In one exemplary embodiment, the scenario is set as follows: In a smart home environment, there are smart lights using the ZigBee protocol and smart sockets using the BLE protocol, as well as a smart air conditioner using a proprietary protocol and some older devices. The user wants to achieve interconnectivity and unified control between the devices and create personalized control logic according to their own needs.

[0149] The following steps can be performed:

[0150] 1) Multi-protocol dynamic translation: The smart light (ZigBee protocol) sends a "turn on light" command to the multi-protocol dynamic translation middleware architecture. The protocol conversion matrix engine parses the ZigBee protocol command and, through a pre-built ZigBee to BLE conversion matrix, converts the command into a BLE protocol command before sending it to the smart socket (BLE protocol). Simultaneously, the smart socket's commands can also be converted into ZigBee protocol commands in a similar manner to communicate with the smart light, achieving low-latency, highly compatible cross-protocol communication.

[0151] 2) UI Control Semantic Generalization Mapping: Smart lights and smart sockets each provide their UI element images to the UI control semantic generalization mapping module. The ViT-based cross-App control standardization module processes the images, extracts features, and sends them to the unified semantic tag generation module. This module maps the on / off UI elements of both the smart lights and smart sockets to the unified semantic tag "on / off," facilitating user operation through a unified control interface and reducing adaptation workload.

[0152] 3) Private Protocol Reverse Engineering: During operation, the private protocol reverse engineering module's traffic capture and UI operation trajectory analysis module collects the smart air conditioner's communication traffic and user operation trajectory on the air conditioner's UI. The protocol logic reverse engineering and control template generation module uses this data to reverse engineer the air conditioner's private protocol logic and generate control templates. For example, it reverse-engineers the protocol commands corresponding to operations such as "adjust temperature" and "adjust fan speed," allowing other devices to communicate with the smart air conditioner based on the control templates.

[0153] 4) Intelligent Protocol Version Downgrade: Suppose that an incompatibility issue arises after the smart light's protocol version is upgraded. The protocol change feature detection module of the intelligent protocol version downgrade module detects the protocol change, and the version rollback decision module decides to roll back to the previous compatible version based on the device's historical version records and compatibility information. It then sends a version rollback command to the smart light, enabling the smart light to resume normal operation and ensuring the continued availability of older devices.

[0154] 5) Multimodal Protocol Verification: When the smart socket executes the "Power On" protocol command, the current fingerprint detection module of the multimodal protocol verification module acquires the current fingerprint characteristics during the power-on process, and the device status feedback module acquires the status information of successful power-on. The multimodal fusion verification module merges the two and verifies the protocol execution through a neural network model. If the current fingerprint characteristics match the device status information, it indicates that the protocol execution is correct; otherwise, it reports an error in protocol execution, reducing the error rate.

[0155] 6) Protocol Fragment Self-Healing: When parsing ZigBee protocol data packets of a smart light, the local protocol parsing vulnerability detection and repair module of the protocol fragment self-healing network module detects a parsing error in a certain field. It requests repair knowledge from the distributed protocol knowledge sharing node. After a node provides a repair method, the module automatically repairs the parsing vulnerability, ensuring normal protocol parsing and improving system robustness.

[0156] 7) Dynamic Semantic Protocol Extension: Users input custom semantic rules such as "Night Mode = Turn off smart lights + Turn off smart sockets + Adjust smart air conditioner temperature to 26℃" through the dynamic semantic protocol extension module. The natural language processing and logic generation module analyzes the rules, extracts keywords and logical relationships, and generates executable control logic. This control logic is then sent to the smart lights, smart sockets, and smart air conditioner to fulfill the user's personalized control needs.

[0157] In summary, the optional embodiments of this application are constructed through 1) a protocol conversion matrix engine in a multi-protocol dynamic translation middleware architecture: protecting the construction method of the protocol conversion matrix engine, including feature extraction methods for multiple communication protocols, conversion matrix construction algorithms, and caching optimization mechanisms, to achieve real-time parsing of multiple protocols and efficient bidirectional instruction conversion, reduce latency and improve compatibility; the real-time parsing and bidirectional conversion technology can protect the specific technology for implementing real-time parsing of multiple protocols and bidirectional instruction conversion, ensuring accurate and fast communication between devices with different protocols, breaking down protocol barriers, and improving the efficiency and compatibility of communication between devices. 2) UI element feature extraction based on ViT in the semantic generalization mapping algorithm for UI controls: This protects the specific method for feature extraction of UI element images from different brands based on ViT, including image preprocessing, patch partitioning, linear mapping, position encoding, and the application of Transformer encoders; Unified semantic label generation and mapping: This protects the method of classifying UI element feature representations into unified semantic labels using a pre-trained classification model, and the mechanism for mapping UI elements from different brands to unified semantic labels, reducing the workload of adaptation and achieving unified control of UI on different brand devices; 3) Traffic packet capture and UI operation trajectory analysis in the reverse engineering toolchain of private protocols: This protects the method of collecting and processing device communication traffic and UI operation trajectory data in the traffic packet capture and UI operation trajectory analysis module, including data acquisition methods and feature extraction techniques, to provide accurate data for reverse engineering protocol logic; Protocol logic reverse engineering and control template generation: This protects the specific method of reverse engineering protocol logic using data mining and machine learning algorithms and automatically generating control templates to achieve interconnection between devices; 4) Intelligent protocol version downgrading Protocol change feature detection in the protection system: The protocol change feature detection module monitors protocol data, extracts protocol version change features, and compares them with a predefined feature library to accurately determine whether the protocol has changed and the type of change; Version rollback decision mechanism: The version rollback decision module determines the rollback version and sends a rollback command based on the protocol change detection results, device historical version records, and compatibility information to ensure the continuous control availability of aging devices; 5) Multimodal information acquisition and fusion in the multimodal protocol verification engine: The current fingerprint detection module acquires current fingerprint features, the device status feedback module acquires device status information, and the multimodal fusion verification module fuses the two into a feature vector, providing comprehensive information for protocol execution verification; Machine learning-based verification algorithm: The protection uses machine learning algorithms such as neural networks to train and verify the fused feature vector to determine whether the protocol execution is correct, reducing the error rate and improving the accuracy and stability of device operation; 6) Distributed protocol knowledge sharing nodes in the protocol fragment self-healing network architecture: Provides knowledge support for protocol parsing vulnerability repair;Local Protocol Parsing Vulnerability Detection and Remediation: Protects the mechanism of the local protocol parsing vulnerability detection and remediation module to detect protocol parsing vulnerabilities, request remediation knowledge from shared nodes, and automatically remediate vulnerabilities, improving system robustness and ensuring stable system operation; 7) Natural Language Processing and Semantic Extraction in the Semantic Protocol Dynamic Extension Framework: Protects the method of the natural language processing and logic generation module to perform lexical and syntactic analysis on user-input natural language, extract keywords and semantic structures, and convert natural language into processable semantic information; Control Logic Generation and Mapping: Protects the method of converting extracted semantic information into executable control logic, matching it with device function interfaces, and generating control command sequences, supporting users to create personalized control logic through natural language to meet users' personalized control needs;

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0159] Figure 11 This is a structural block diagram of an instruction transmitting device according to an embodiment of this application; as shown below. Figure 11 As shown, it includes:

[0160] The determining module 112 is configured to, upon receiving a first instruction sent by a first device to a second device through a first device employing a first protocol, determine the second protocol employed by the second device and determine the target data structure of the first instruction under the second protocol;

[0161] Extraction module 114 is used to extract features from the first protocol and the second protocol to obtain the first protocol features corresponding to the first protocol and the second protocol features corresponding to the second protocol;

[0162] The matching module 116 is used to perform feature matching on the first protocol feature and the second protocol feature using a target matching algorithm, and to construct a protocol conversion matrix between the first protocol and the second protocol based on the feature matching results;

[0163] The conversion module 118 is used to convert the first instruction into a second instruction conforming to the second protocol based on the protocol conversion matrix and the target data structure, and send the second instruction to the second device.

[0164] According to the above apparatus, upon receiving a first instruction sent by a target object to a second device via a first device using a first protocol, the apparatus determines the second protocol used by the second device and the target data structure of the first instruction under the second protocol; it extracts features from the first and second protocols to obtain the first protocol features corresponding to the first protocol and the second protocol features corresponding to the second protocol; it performs feature matching on the first and second protocol features using a target matching algorithm, and constructs a protocol conversion matrix between the first and second protocols based on the feature matching results; based on the protocol conversion matrix and the target data structure, it converts the first instruction into a second instruction conforming to the second protocol, and sends the second instruction to the second device. In other words, this application, upon determining that a first instruction corresponding to the first protocol needs to be sent to the second device, can determine the second protocol used by the second device and the target data structure of the first instruction under the second protocol; furthermore, it extracts features from the first and second protocols respectively, and constructs a protocol conversion matrix between the first and second protocols based on the first and second protocol features; based on the protocol conversion matrix and the target data structure, it can convert the first instruction into a second instruction conforming to the second protocol. Based on this application, the problem that smart home systems in related technologies only support communication between devices using a single protocol and cannot directly communicate with devices using other protocols can be solved. Furthermore, communication between devices using different protocols can be achieved through a technical solution that converts instructions of the first protocol into instructions of the second protocol.

[0165] In an exemplary embodiment, the determining module 112 is further configured to: obtain a device identifier of the second device and determine the second protocol based on the device identifier if it is determined that the second device has opened the protocol adopted by the second device to the first device; obtain historical communication data packets corresponding to the second device and the operation trajectory of the target object on the second control corresponding to the second device if it is determined that the second device has not opened the protocol adopted by the second device to the first device; extract features from the historical communication data packets and input the features corresponding to the historical communication data packets and the operation trajectory into a machine learning algorithm to determine the protocol logic corresponding to the protocol adopted by the second device; and generate a protocol template corresponding to the protocol adopted by the second device based on the protocol logic, wherein the protocol template is a description file of the second protocol adopted by the second device.

[0166] In an exemplary embodiment, the determining module 112 is further configured to, when determining that the target object sends the first instruction to the second device through the first control in the first device, divide the control image corresponding to the first control into multiple image blocks and map each image block to a target vector; add position encoding to the target vector to generate an augmented vector containing the spatial position information of each image block in the control image; extract features from the multiple augmented vectors through a target encoder to obtain a global feature vector corresponding to the first control, and input the global feature vector into a classification model to obtain the control type corresponding to the first control; map the first control to a target semantic label according to the control type; and determine the target data structure in a semantic protocol mapping table, wherein the semantic protocol mapping table is used to indicate the instruction format of the semantic label corresponding to each control under each protocol.

[0167] In an exemplary embodiment, the conversion module 118 is further configured to, when determining that the sending status corresponding to the second instruction is a sending failure, obtain historical communication data packets corresponding to the second device, and extract protocol version change features corresponding to the second protocol from the historical communication data packets using a multidimensional feature extraction algorithm; match the protocol version change features with preset protocol version change features in the protocol version change library to determine the protocol change status corresponding to the second protocol; when the protocol change status indicates that the protocol version of the second protocol has been changed to the first protocol version, obtain historical protocol version usage logs corresponding to the second device, and determine a device compatible version list corresponding to the second device based on the historical protocol version usage logs; determine the target protocol version of the second protocol in the device compatible version list, and send a protocol version rollback instruction to the second device to roll back the second protocol used by the second device from the first protocol version to the target protocol version; and send the second instruction to the second device using the second protocol of the target protocol version.

[0168] In an exemplary embodiment, the conversion module 118 is further configured to acquire the current signal of the second device during the execution of the second instruction, and extract the current fingerprint feature corresponding to the current signal; and acquire the state feedback information of the second device after executing the second instruction; encode the current fingerprint feature and the state feedback information into first feature vectors respectively, and concatenate multiple first feature vectors into a fused feature vector; input the fused feature vector into a deep neural network model to determine the execution state of the second device executing the second instruction, wherein the execution state includes one of the following: correct execution, incorrect execution.

[0169] In an exemplary embodiment, the matching module 116 is further configured to determine whether a preset error conversion library contains an error conversion record corresponding to the protocol conversion matrix; if it is determined that the preset error conversion library contains the error conversion record, a broadcast request is sent to each node to obtain the repair rules corresponding to the protocol conversion matrix, wherein multiple nodes are used to construct the protocol conversion matrix; and the protocol conversion matrix is ​​repaired according to the repair rules.

[0170] In an exemplary embodiment, the determining module 112 is further configured to, upon receiving a voice command sent by the target object, extract keywords and semantic structure corresponding to the voice command, and convert the voice command into target control logic based on the keywords and semantic structure; match the target control logic with the device interface corresponding to each device to generate an instruction sequence containing a target execution device, wherein the instruction sequence contains: a first instruction, and the target execution device contained in the first instruction is: the second device.

[0171] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0172] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0173] S1, upon receiving a first instruction sent by the target object to the second device through the first device using the first protocol, determine the second protocol used by the second device, and determine the target data structure of the first instruction under the second protocol;

[0174] S2, perform feature extraction on the first protocol and the second protocol to obtain the first protocol feature corresponding to the first protocol and the second protocol feature corresponding to the second protocol;

[0175] S3, perform feature matching on the first protocol features and the second protocol features using a target matching algorithm, and construct a protocol conversion matrix between the first protocol and the second protocol based on the feature matching results;

[0176] S4, based on the protocol conversion matrix and the target data structure, the first instruction is converted into a second instruction conforming to the second protocol, and the second instruction is sent to the second device.

[0177] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0178] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0179] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0180] S1, upon receiving a first instruction sent by the target object to the second device through the first device using the first protocol, determine the second protocol used by the second device, and determine the target data structure of the first instruction under the second protocol;

[0181] S2, perform feature extraction on the first protocol and the second protocol to obtain the first protocol feature corresponding to the first protocol and the second protocol feature corresponding to the second protocol;

[0182] S3, perform feature matching on the first protocol features and the second protocol features using a target matching algorithm, and construct a protocol conversion matrix between the first protocol and the second protocol based on the feature matching results;

[0183] S4, based on the protocol conversion matrix and the target data structure, the first instruction is converted into a second instruction conforming to the second protocol, and the second instruction is sent to the second device.

[0184] Embodiments of this application also provide a computer program product, including a computer program that is executed by a processor through the steps of any of the above method embodiments.

[0185] Optionally, in this embodiment, the above-mentioned computer program product can be executed by a processor using the following steps:

[0186] S1, upon receiving a first instruction sent by the target object to the second device through the first device using the first protocol, determine the second protocol used by the second device, and determine the target data structure of the first instruction under the second protocol;

[0187] S2, perform feature extraction on the first protocol and the second protocol to obtain the first protocol feature corresponding to the first protocol and the second protocol feature corresponding to the second protocol;

[0188] S3, perform feature matching on the first protocol features and the second protocol features using a target matching algorithm, and construct a protocol conversion matrix between the first protocol and the second protocol based on the feature matching results;

[0189] S4, based on the protocol conversion matrix and the target data structure, the first instruction is converted into a second instruction conforming to the second protocol, and the second instruction is sent to the second device.

[0190] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0191] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0192] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0193] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for sending instructions, characterized in that, include: Upon receiving a first instruction sent by a target object to a second device via a first device using a first protocol, the second protocol used by the second device is determined, and the target data structure of the first instruction under the second protocol is determined. Feature extraction is performed on the first protocol and the second protocol to obtain the first protocol feature corresponding to the first protocol and the second protocol feature corresponding to the second protocol; The first protocol features and the second protocol features are matched using a target matching algorithm, and a protocol conversion matrix between the first protocol and the second protocol is constructed based on the feature matching results. Based on the protocol conversion matrix and the target data structure, the first instruction is converted into a second instruction that conforms to the second protocol, and the second instruction is sent to the second device.

2. The method for sending instructions according to claim 1, characterized in that, Determining the second protocol used by the second device includes: If it is determined that the second device opens the protocol adopted by the second device to the first device, the device identifier of the second device is obtained, and the second protocol is determined based on the device identifier; If it is determined that the second device has not disclosed the protocol used by the second device to the first device, the historical communication data packets corresponding to the second device and the operation trajectory of the target object on the second control corresponding to the second device are obtained; Feature extraction is performed on the historical communication data packets, and the features corresponding to the historical communication data packets and the operation trajectory are input into a machine learning algorithm to determine the protocol logic corresponding to the protocol used by the second device; A protocol template corresponding to the protocol used by the second device is generated according to the protocol logic, wherein the protocol template is a description file of the second protocol used by the second device.

3. The method for sending instructions according to claim 1, characterized in that, Determining the target data structure of the first instruction under the second protocol includes: When it is determined that the target object sends the first instruction to the second device through the first control in the first device, the control image corresponding to the first control is divided into multiple image blocks, and each image block is mapped to a target vector. A positional encoding is added to the target vector to generate an augmented vector containing the spatial location information of each image patch in the control image; The target encoder extracts features from multiple augmentation vectors to obtain the global feature vector corresponding to the first control, and inputs the global feature vector into the classification model to obtain the control type corresponding to the first control. Map the first control to the target semantic tag according to the control type; The target semantic tag determines the target data structure in the semantic protocol mapping table, wherein the semantic protocol mapping table is used to indicate the instruction format of the semantic tag corresponding to each control under each protocol.

4. The method for sending instructions according to claim 1, characterized in that, After sending the second instruction to the second device, the method further includes: If the sending status corresponding to the second instruction is determined to be a sending failure, the historical communication data packets corresponding to the second device are obtained, and the protocol version change features corresponding to the second protocol are extracted from the historical communication data packets using a multidimensional feature extraction algorithm. The protocol version change feature is matched with the preset protocol version change feature in the protocol version change library to determine the protocol change status corresponding to the second protocol; When the protocol change status indicates that the protocol version of the second protocol has been changed to the first protocol version, the historical protocol version usage log corresponding to the second device is obtained, and the device compatible version list corresponding to the second device is determined based on the historical protocol version usage log. The target protocol version of the second protocol is determined from the list of compatible device versions, and a protocol version rollback instruction is sent to the second device to roll back the second protocol used by the second device from the first protocol version to the target protocol version. The second instruction is sent to a second device that uses the second protocol version of the target protocol.

5. The method for sending instructions according to claim 1, characterized in that, After sending the second instruction to the second device, the method further includes: Acquire the current signal of the second device during the execution of the second instruction, and extract the current fingerprint feature corresponding to the current signal; and, Obtain the status feedback information of the second device after executing the second instruction; The current fingerprint feature and the state feedback information are respectively encoded into first feature vectors, and multiple first feature vectors are concatenated into a fused feature vector; The fused feature vector is input into a deep neural network model to determine the execution state of the second device executing the second instruction, wherein the execution state includes one of the following: correct execution or incorrect execution.

6. The method for sending instructions according to claim 1, characterized in that, After constructing the protocol conversion matrix between the first protocol and the second protocol based on the feature matching results, the method further includes: Determine whether the preset error conversion library contains the error conversion record corresponding to the protocol conversion matrix; If it is determined that the preset error conversion library contains the error conversion record, a broadcast request is sent to each node to obtain the repair rules corresponding to the protocol conversion matrix, wherein multiple nodes are used to construct the protocol conversion matrix; The protocol conversion matrix is ​​repaired according to the repair rules.

7. The method for sending instructions according to claim 1, characterized in that, Before determining the second protocol adopted by the second device, the method further includes: Upon receiving a voice command from the target object, extract the keywords and semantic structure corresponding to the voice command, and convert the voice command into target control logic based on the keywords and semantic structure. The target control logic is matched with the device interface corresponding to each device to generate an instruction sequence containing the target execution device, wherein the instruction sequence includes: the first instruction, and the target execution device contained in the first instruction is: the second device.

8. A means for transmitting instructions, characterized in that, include: The determination module is configured to, upon receiving a first instruction sent by a target object to a second device via a first device employing a first protocol, determine the second protocol employed by the second device and determine the target data structure of the first instruction under the second protocol; The extraction module is used to extract features from the first protocol and the second protocol to obtain the first protocol features corresponding to the first protocol and the second protocol features corresponding to the second protocol. The matching module is used to perform feature matching on the first protocol features and the second protocol features using a target matching algorithm, and to construct a protocol conversion matrix between the first protocol and the second protocol based on the feature matching results; The conversion module is used to convert the first instruction into a second instruction conforming to the second protocol based on the protocol conversion matrix and the target data structure, and send the second instruction to the second device.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images