Semantic information generation method and device, storage medium and electronic device

Abstract

The invention discloses a semantic information generation method and device, a storage medium and an electronic device, and relates to the technical field of artificial intelligence and Internet of Things crossing, and the semantic information generation method comprises the steps: encoding a plurality of pieces of first metadata in a first scene and a first natural language instruction corresponding to the first scene sent by a target object, wherein the first metadata at least comprises one of a device list in the first scene, a capability attribute of each device in the device list and a scene label of the first scene; according to the encoded target vector, determining an attention weight corresponding to an attention mechanism in the edge device, and according to the attention weight, performing fusion processing on the plurality of first metadata and the first natural language instruction to generate target semantic information corresponding to the first natural language instruction, the target vector comprises a first metadata coding vector corresponding to the multiple pieces of first metadata and a first instruction coding vector corresponding to the first natural language instruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and Internet of Things, in particular to a semantic information generation method and device, a storage medium and an electronic device. BACKGROUND

[0002] In the current field of artificial intelligence and Internet of Things, edge device models have become a key technology for processing local data and providing instant responses in scenarios such as smart homes and industrial automation. However, the edge device models in related technologies have significant limitations and challenges when facing dynamically changing environmental metadata. Traditional edge device models use static training, which means that once the model is trained, it cannot adapt to changes in the environment. This characteristic is particularly inadequate when facing constantly updated device lists, scene configurations, or user needs. Edge device models lack the ability to dynamically absorb external environmental metadata, resulting in poor performance when handling new device additions or complex and variable intentions, and failing to accurately match user instructions and device operations.

[0003] That is, traditional edge device model designs are often based on predefined static environmental configurations and cannot autonomously integrate and respond to real-time changing metadata information, such as device state updates and scene mode adjustments, during the inference process. This means that when the device configuration in the user environment changes or new devices are introduced, the model's intention analysis capability significantly decreases, making it unable to accurately match user instructions and device operations, resulting in delayed control responses and misoperations.

[0004] To address the problem that edge devices in related technologies cannot autonomously integrate dynamically changing metadata information during the inference process, no effective solutions have been proposed. SUMMARY

[0005] The embodiments of the present application provide a semantic information generation method and device, a storage medium and an electronic device to at least solve the problem that edge devices in related technologies cannot autonomously integrate dynamically changing metadata information during the inference process.

[0006] According to an embodiment of the present application, a method for generating semantic information is provided, comprising: encoding a plurality of first metadata in a first scene and a first natural language instruction corresponding to the first scene sent by a target object, wherein the first metadata at least includes one of the following: a device list in the first scene, a capability attribute of each device in the device list, and a scene label of the first scene; determining an attention weight corresponding to an attention mechanism in an edge device according to a target vector after encoding, and performing fusion processing on the plurality of first metadata and the first natural language instruction according to the attention weight to generate target semantic information corresponding to the first natural language instruction, wherein the target vector includes a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction.

[0007] In an example embodiment, encoding the plurality of first metadata in the first scene and the first natural language instruction sent by the target object comprises: converting each metadata into a natural language description, and splicing the plurality of natural language description metadata and the first natural language instruction to generate a first prompt string; performing word segmentation processing on the first prompt string by a word analyzer to generate a plurality of first word units, wherein the plurality of first word units include a plurality of second word units corresponding to the plurality of first metadata and a plurality of third word units corresponding to the first natural language instruction; and uniformly encoding the plurality of first word units into a vector embedding space to generate the first instruction encoding vector and the first metadata encoding vector.

[0008] In an example embodiment, splicing the plurality of natural language description metadata and the first natural language instruction to generate a first prompt string comprises: parsing the first natural language instruction to extract position metadata corresponding to the first natural language instruction, wherein the position metadata is used to indicate position information contained in the first natural language instruction; encoding the position metadata and splicing the encoded position metadata with the first natural language instruction to obtain a second prompt string; screening device information corresponding to the second prompt string from the first metadata, and splicing the device information with the second prompt string to obtain the first prompt string, wherein the device information has a corresponding relationship with the position metadata.

[0009] In an example embodiment, the method further comprises: encoding the first natural language instruction to generate a first instruction vector, and encoding each first metadata to generate a first metadata vector; determining a first dimension of the first instruction vector, a second dimension of the first metadata vector, and a third dimension corresponding to a common mapping space, respectively; determining an instruction mapping matrix corresponding to the first instruction vector according to the first dimension and the third dimension, and determining a metadata mapping matrix corresponding to the first metadata vector according to the second dimension and the third dimension; mapping the instruction vector to the common mapping space according to the instruction mapping matrix to generate a first instruction encoded vector, and mapping the metadata vector to the common mapping space according to the metadata mapping matrix to generate a first metadata encoded vector, wherein the vector dimension of the first instruction encoded vector and the first metadata encoded vector is the third dimension.

[0010] In an example embodiment, encoding the first natural language instruction to generate a first instruction vector comprises: processing the first natural language instruction by a token analyzer to generate a plurality of fourth tokens; encoding each fourth token by an encoder to generate a second vector corresponding to the each fourth token; and integrating the plurality of second vectors to generate the first instruction vector.

[0011] In an example embodiment, encoding each first metadata to generate a first metadata vector comprises: converting each first metadata into a text form, and processing each text form of the first metadata by a token analyzer to generate a plurality of fifth tokens corresponding to the each first metadata; encoding the plurality of fifth tokens corresponding to the each first metadata by an encoding mechanism corresponding to the each first metadata to generate a third vector corresponding to the each first metadata, wherein the encoding mechanism corresponding to the first metadata containing different metadata information is different; and splicing the plurality of third vectors to generate a metadata vector corresponding to the plurality of first metadata.

[0012] In an example embodiment, the method further includes: determining, according to the encoded target vector, an attention weight corresponding to an attention mechanism in the edge device, and performing fusion processing on the plurality of first metadata and the first natural language instruction according to the attention weight to generate target semantic information corresponding to the first natural language instruction, including: performing linear transformation on the first metadata encoding vector according to a key vector matrix and a value vector matrix in the edge device respectively to generate a key vector and a value vector corresponding to each first metadata, and performing linear transformation on the first instruction encoding vector according to a query vector matrix in the edge device to generate a query vector, wherein the key vector is used to indicate a semantic label corresponding to each first metadata, and the value vector is used to indicate metadata information contained in the each first metadata; performing dot product calculation on the query vector and each key vector respectively according to the attention mechanism to determine a similarity score between the query vector and the each key vector, and determining the attention weight according to a plurality of similarity scores; performing weighted summation on a plurality of value vectors according to the attention weight to generate a context vector, and generating the target semantic information according to the context vector, wherein the context vector is used to indicate an association degree between the first natural language instruction and each first metadata.

[0013] In an example embodiment, after performing fusion processing on the plurality of first metadata and the first natural language instruction according to the attention weight to generate target semantic information corresponding to the first natural language instruction, the method further includes: storing a plurality of key vectors and a plurality of value vectors into a target cache space in the edge device; in a case where a second natural language instruction corresponding to the first scene is received, encoding the second natural language instruction to generate a second instruction encoding vector; adjusting the attention weight according to the second instruction encoding vector and the first metadata encoding vector, and performing fusion processing on the plurality of first metadata and the second natural language instruction according to the adjusted attention weight to generate semantic information corresponding to the second natural language instruction.

[0014] In an example embodiment, after the plurality of first metadata and the first natural language instruction are fused according to the attention weight to generate the target semantic information corresponding to the first natural language instruction, the method further comprises: in a case where the first scene has changed to a second scene or a second metadata has been added in the first scene, obtaining a first parameter of a sub-network in the edge device, and adjusting the first parameter based on an incremental adaptation network to generate a second parameter of the sub-network, wherein the first parameter is used for adaptation with the first metadata, and the second parameter is used for adaptation with a third metadata or the second metadata, and the third metadata is other metadata in the second scene except the first metadata; and performing semantic recognition on a third natural language instruction based on the edge device containing the sub-network adjusted to the second parameter, wherein the third natural language instruction is a natural language instruction corresponding to the second scene or the first scene after the second metadata is added.

[0015] In an example embodiment, the apparatus further comprises an incremental adaptation module configured to: in a case where the first scene has changed to a second scene or a second metadata has been added in the first scene, obtain a first parameter of a sub-network in the edge device, and adjust the first parameter based on an incremental adaptation network to generate a second parameter of the sub-network, wherein the first parameter is used for adaptation with the first metadata, and the second parameter is used for adaptation with a third metadata or the second metadata, and the third metadata is other metadata in the second scene except the first metadata.

[0016] According to another embodiment of the embodiment of the present application, a semantic information generation apparatus is further provided, comprising: an encoding module configured to encode a plurality of first metadata in a first scene and a first natural language instruction corresponding to the first scene and sent by a target object, wherein the first metadata at least include one of the following: a device list in the first scene, a capability attribute of each device in the device list, and a scene label of the first scene; a determination module configured to determine an attention weight corresponding to an attention mechanism in an edge device according to an encoded target vector, and fuse the plurality of first metadata and the first natural language instruction according to the attention weight to generate target semantic information corresponding to the first natural language instruction, wherein the target vector includes a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction.

[0017] According to a further aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is configured to execute the above-mentioned method for generating semantic information when running.

[0018] According to a further aspect of the embodiments of the present application, an electronic device is provided, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor. The processor executes the above-mentioned method for generating semantic information through the computer program.

[0019] According to a further aspect of the embodiments of the present application, a computer program product is provided, which comprises a computer program. The computer program is executed by a processor to execute the above-mentioned method.

[0020] In the embodiments of the present application, the plurality of first metadata (including: a device list in the first scene, a capability attribute of each device, a scene tag, etc.) in the first scene and the first natural language instruction corresponding to the target object sent by the first scene are encoded, and the attention weight corresponding to the attention mechanism in the edge device is determined according to the encoded target vector (including: a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction). The plurality of first metadata and the first natural language instruction are fused to generate the target semantic information corresponding to the first natural language instruction according to the attention weight. That is, the metadata and the natural language instruction are encoded in the present application, and then the attention weight is dynamically adjusted, and the degree of combination of the metadata and the natural language instruction is dynamically adjusted using the attention mechanism. Through the above technical solution, the edge device can autonomously integrate dynamically changing metadata information, and flexibly focus on the metadata most relevant to the current natural language instruction. Even if the metadata changes, the above technical solution can quickly respond, solving the problem in the related art that the edge device cannot autonomously integrate dynamically changing metadata information in the reasoning process, and thus the edge device can autonomously integrate dynamically changing metadata information in the reasoning process. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0023] Figure 1 is a hardware environment schematic diagram of a semantic information generation method according to an embodiment of the present application;

[0024] Figure 2 is a flowchart of a semantic information generation method according to an embodiment of the present application;

[0025] Figure 3 is an architecture diagram of an edge device-based intelligent device control system according to an optional embodiment of the present application;

[0026] Figure 4 is a flowchart of an edge device-based dynamic metadata fusion method according to an optional embodiment of the present application;

[0027] Figure 5 is a Prompt template schematic diagram constructed according to an optional embodiment of the present application;

[0028] Figure 6 is a metadata encoding source schematic diagram according to an optional embodiment of the present application;

[0029] Figure 7 is a structural block diagram of a semantic information generation apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0031] It should be noted that, in the description of the present application, the terms “include”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0032] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0033] In combination with the specific application environment architecture or specific hardware architecture on which the test case generation method is dependent, the specific application environment architecture or specific hardware architecture is described here.

[0034] The methods and embodiments provided in this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a test case generation method according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor (MPU) or programmable logic device (PLD)) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the method for determining the interaction state in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0037] A semantic information generation method is provided in the embodiment, which is applied to the computer terminal, Figure 2 is a flowchart of the semantic information generation method according to the embodiment of the application, which comprises the following steps:

[0038] In step S202, the plurality of first metadata in the first scene and the first natural language instruction corresponding to the first scene sent by the target object are encoded, wherein the first metadata at least include one of the following: a device list in the first scene, a capability attribute of each device in the device list, and a scene label of the first scene.

[0039] The first scene can be a living room in a user's home or a specific work conference space.

[0040] In step S204, the attention weight corresponding to the attention mechanism in the edge device is determined according to the encoded target vector, and the plurality of first metadata and the first natural language instruction are fused according to the attention weight to generate the target semantic information corresponding to the first natural language instruction, wherein the target vector includes a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction.

[0041] Through the above steps, the plurality of first metadata (including a device list in the first scene, a capability attribute of each device, a scene label, etc.) in the first scene and the first natural language instruction corresponding to the first scene sent by the target object are encoded, and the attention weight corresponding to the attention mechanism in the edge device is determined according to the encoded target vector (including a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction). The plurality of first metadata and the first natural language instruction are fused according to the attention weight to generate the target semantic information corresponding to the first natural language instruction. That is, the metadata and the natural language instruction are encoded in the application, and then the attention weight is dynamically adjusted, and the attention mechanism is used to dynamically adjust the degree of combination of the metadata and the natural language instruction. Through the above technical solution, the edge device can autonomously integrate dynamically changing metadata information and flexibly focus on the metadata most relevant to the current natural language instruction. Even if the metadata changes, the above technical solution can quickly respond, solving the problem in the related art that the edge device cannot autonomously integrate dynamically changing metadata information in the reasoning process, and thus the edge device can autonomously integrate dynamically changing metadata information in the reasoning process.

[0042] Optionally, the encoding of the plurality of first metadata in the first scene and the first natural language instruction corresponding to the first scene sent by the target object in step S202 can include the following two schemes:

[0043] (1) Metadata dynamic fusion mechanism: convert each metadata into a natural language description, and splice multiple natural language description metadata and the first natural language instruction to generate a first prompt string; perform word segmentation processing on the first prompt string through a word unit analyzer to generate multiple first word units, wherein the multiple first word units include multiple second word units corresponding to the multiple first metadata and multiple third word units corresponding to the first natural language instruction; uniformly encode the multiple first word units into a vector embedding space to generate the first instruction encoding vector and the first metadata encoding vector.

[0044] It can be understood that the steps of the metadata dynamic fusion mechanism can include:

[0045] 1) Metadata conversion to natural language description and splicing:

[0046] Convert structured metadata into a natural language description to enable the metadata to be understood by a model (e.g., an edge device) in the form of text. For example, information contained in a device list such as device name, location, type, and status is converted into a description such as “The bedroom lamp is in the bedroom, it is an adjustable brightness lamp, and it is currently on; the air conditioner is in the living room, it is a cooling device, and it is currently off.”

[0047] Splice the above natural language description metadata and the user's first natural language instruction to form a complete input sequence, i.e., a first prompt string. For example, if the user's instruction is “Please turn up the brightness of the bedroom lamp”, then the first prompt string will be “The bedroom lamp is in the bedroom, it is an adjustable brightness lamp, and it is currently on; the air conditioner is in the living room, it is a cooling device, and it is currently off. Please turn up the brightness of the bedroom lamp.”

[0048] Among them, the specific steps of splicing multiple natural language description metadata and the first natural language instruction to generate a first prompt string can include: parsing the first natural language instruction using a rule engine or a location prior identification module to extract location metadata corresponding to the first natural language instruction, wherein the location metadata is used to indicate location information contained in the first natural language instruction; encoding the location metadata and splicing the encoded location metadata with the first natural language instruction to obtain a second prompt string; filtering device information corresponding to the second prompt string from the first metadata, and splicing the device information with the second prompt string to obtain the first prompt string, wherein the device information and the location metadata have a corresponding relationship.

[0049] It can be understood that, as user instructions often imply spatial location information, for example: "turn off the light in the bedroom", but the technical solutions in the related art still need to splice the device information of all rooms, making the inference process inefficient. Therefore, the present application defines a technical solution that identifies the room location involved in advance before model inference, thereby splicing only local metadata, to achieve the purpose of reducing the burden and speeding up. Specifically:

[0050] Prior location recognition module: after the user voice is transcribed into text by automatic speech recognition (ASR), the spatial location keywords are extracted by the following methods: 1) regular expression + keyword table matching (for example: "bedroom", "living room", "master bedroom", etc.); 2) Trie tree (prefix tree) matching to realize fast text scanning; 3) support custom location alias library (for example: "Xiaoming's room" corresponds to "bedroom").

[0051] Local metadata construction mechanism: after the location is identified, the device information of the corresponding room is filtered from the local cache metadata, and only this subset is constructed as the metadata part of the Prompt (prompt string).

[0052] For example: Prompt: You are a smart home assistant, and the following is the bedroom device configuration:

[0053] Device1 (suggestion 1): name = bedroom light, type = light, state = on;

[0054] Device2 (suggestion 2): name = curtain, type = curtain, state = open;

[0055] User instruction: turn off the light in the bedroom.

[0056] Compared with splicing the device information of the whole house, this method greatly reduces the number of tokens.

[0057] Prompt cache coordination mechanism:

[0058] The static KV cache structure remains unchanged, but only the Prompt related to the current location is constructed for each round of inference;

[0059] Combined with the historical location cache, the user's frequently used room data can be reused, further reducing the response delay;

[0060] If the user does not mention the location, the default room (for example: the last operating room or global metadata) can be used.

[0061] Edge Large Language Model (Edge LLM) focuses on attention and enhances semantic significance:

[0062] Since the input only contains relevant room devices, Attention will naturally focus on the current task context, significantly improving intent recognition accuracy.

[0063] Through the above technical solution, the prior position recognition mechanism based on the rule engine accurately trims the Prompt content before reasoning; guides the large model to reason within the task-related context, reducing redundant Token input; the low-cost and highly compatible design concept is suitable for all edge LLM deployment solutions that support Prompt splicing. The above technical solution can reduce the number of input Tokens by an average of 40%-80%, adapt to lower resource edge devices; shorten the LLM reasoning delay by 30%-60%; improve the intent recognition accuracy by about 5%-10%; the rule engine has low overhead and is easy to implement, and can be directly deployed on edge devices; this mechanism can complement and enhance the original patent solution (LoRA fine-tuning, structure-guided Prompt, etc.).

[0064] 2) Tokenization by Wordpiece Analyzer:

[0065] The first prompt string is processed by the wordpiece analyzer to tokenize the text into a series of first wordpieces. Wordpieces can be words, phrases, or symbols, depending on the setting rules of the wordpiece analyzer. For example, "bedroom light", "in", "bedroom", "inside" and other words will be recognized as independent wordpieces.

[0066] The above first wordpieces include second wordpieces and third wordpieces. The second wordpieces refer to wordpieces derived from metadata, such as "bedroom light", "on state", etc.; the third wordpieces are derived from the user's first natural language instructions, such as "please", "brighten" and other words.

[0067] 3) Unified encoding to vector embedding space:

[0068] All first wordpieces (including second wordpieces and third wordpieces) are uniformly encoded into a vector embedding space through the word vector embedding layer of the model. In the vector embedding space, each wordpiece is converted into a fixed-length vector, which not only contains the semantic information of the wordpiece itself, but also can express the relationship with other wordpieces.

[0069] Finally, the first instruction encoding vector (representing the semantic information of the user's instruction) and the first metadata encoding vector (representing the semantic information of the environmental metadata) can be obtained.

[0070] For example, assume that the user issues the instruction “Please lower the temperature of the air conditioner in the living room to 25℃” in a smart home environment. First, the list of devices in the room (metadata) is converted into a natural language description: “The air conditioner in the living room is on, and the current temperature setting is 27℃.” Then, this description is concatenated with the user instruction to form the first prompt string: “The air conditioner in the living room is on, and the current temperature setting is 27℃. Please lower the temperature of the air conditioner in the living room to 25℃.” Next, the text string is broken down into a sequence of tokens (e.g., “living room,” “air conditioner,” “on,” “25℃,” etc.) by a tokenizer. Finally, each token is converted into a corresponding vector representation by an embedding layer, forming the first instruction encoding vector and the first metadata encoding vector.

[0071] (2) The way of the dual-modal encoding layer:

[0072] 1) encoding the first natural language instruction to generate a first instruction vector, specifically:

[0073] The first natural language instruction is processed by a tokenizer to generate a plurality of fourth tokens. Each fourth token is encoded by an encoder to generate a second vector corresponding to the fourth token. The plurality of second vectors are integrated to generate the first instruction vector.

[0074] It can be understood that the tokenizer is used to tokenize the user's first natural language instruction to generate a series of fourth tokens. For example, the user instruction is “Please dim the light in the bedroom a little bit,” and the token sequence obtained after tokenization may be “please,” “dim,” “the light in the bedroom,” “a little bit.”

[0075] Each fourth token is converted into a corresponding second vector, i.e., a vector embedding representation, by an encoder (e.g., a Transformer encoder). The embedding vector of each token has a dimension of d2, such as d2 = 768, which means that each token is represented as a 768-dimensional real vector.

[0076] The obtained plurality of second vectors are integrated, such as by a pooling operation or sequence encoding, to generate a first instruction vector of a fixed length. This vector captures the semantic features of the entire instruction.

[0077] 2) encoding each first metadata to generate a first metadata vector, specifically:

[0078] Each first metadata is converted into text form, and each text form of the first metadata is processed by a wordpiece analyzer to generate a plurality of fifth wordpieces corresponding to each first metadata; the plurality of fifth wordpieces corresponding to each first metadata is encoded by an encoding mechanism corresponding to each first metadata to generate a third vector corresponding to each first metadata, wherein the encoding mechanism corresponding to the first metadata containing different metadata information is different; and the plurality of third vectors are spliced to generate a metadata vector corresponding to the plurality of first metadata.

[0079] It can be understood that for metadata, it is first converted into a text form of natural language description, and then processed by a wordpiece analyzer to obtain a series of fifth wordpieces. For example, the metadata "device name: bedroom lamp, device status: on, brightness: 50%" can be converted into the wordpiece sequence "device name", ":", "bedroom lamp", ",", "device status", ":", "on", ",", "brightness", ":", "50%", "%".

[0080] Each fifth wordpiece generates a third vector through a corresponding encoding mechanism. For different types of metadata, the encoding mechanism can be different. For example, the device name "bedroom lamp" can generate a vector through a special word embedding, while the device status "on" can generate another vector representation through a pre-defined one-hot encoding and dimension reduction network.

[0081] The plurality of processed third vectors are spliced to form a metadata vector, which integrates information from different metadata fields.

[0082] 3) respectively determining a first dimension of the first instruction vector, a second dimension of the first metadata vector, and a third dimension of a common mapping space;

[0083] It can be understood that a first dimension (d1) of the first instruction vector, a second dimension (d2) of the first metadata vector, and a third dimension (d3) of a common mapping space for fusion are determined respectively. For example, d1 = 768, d2 = 512 (total dimension after splicing of the plurality of metadata vectors), and d3 = 640.

[0084] 4) determining an instruction mapping matrix corresponding to the first instruction vector according to the first dimension and the third dimension, and determining a metadata mapping matrix corresponding to the first metadata vector according to the second dimension and the third dimension;

[0085] That is, based on the first dimension and the third dimension, an instruction mapping matrix is generated for mapping the first instruction vector from d1 to d3 dimensions. Similarly, based on the second dimension and the third dimension, a metadata mapping matrix is generated for mapping the first metadata vector from d2 to d3 dimensions. These mapping matrices can be derived through learning and can be included in the edge device.

[0086] 5) Map the instruction vector to the common mapping space according to the instruction mapping matrix to generate the first instruction encoding vector, and map the metadata vector to the common mapping space according to the metadata mapping matrix to generate the first metadata encoding vector, wherein the vector dimensions of the first instruction encoding vector and the first metadata encoding vector are both the third dimension.

[0087] That is, by using the above instruction mapping matrix and metadata mapping matrix, the first instruction vector and the first metadata vector are projected into the common mapping space respectively, and the first instruction encoding vector and the first metadata encoding vector of uniform dimension are generated, both of which have a dimension of d3.

[0088] For example: Suppose the user says "Please dim the lights in the bedroom a little" in the smart home environment. This instruction is tokenized by the tokenizer and encoded into a vector sequence representing the semantics, and then integrated into a 768-dimensional first instruction vector through the pooling operation. At the same time, the metadata "the lights in the bedroom are in an open state with a brightness of 50%" is converted into a token sequence and encoded, and finally a 512-dimensional metadata vector containing device state and brightness information is obtained through vector splicing.

[0089] In order to enable the two vectors to be integrated in a unified space, the dimension of the common mapping space is determined to be 640, and the instruction mapping matrix and the metadata mapping matrix are generated. Through these two mapping matrices, the first instruction vector and the first metadata vector are mapped to the 640-dimensional common space respectively, generating the first instruction encoding vector and the first metadata encoding vector. In this way, the instruction and metadata can be processed under a unified vector dimension, thereby achieving more efficient and accurate intent analysis and device control.

[0090] Optionally, the step S204 of determining the attention weight corresponding to the attention mechanism in the edge device according to the encoded target vector, and fusing the plurality of first metadata and the first natural language instruction according to the attention weight to generate the target semantic information corresponding to the first natural language instruction, comprises: respectively linearly transforming the first metadata encoding vector according to the key vector matrix and the value vector matrix in the edge device to generate the key vector and the value vector corresponding to each first metadata, and linearly transforming the first instruction encoding vector according to the query vector matrix in the edge device to generate the query vector, wherein the key vector is used to indicate the semantic label corresponding to each first metadata, and the value vector is used to indicate the metadata information contained in the each first metadata; respectively performing dot product calculation on the query vector and each key vector according to the attention mechanism to determine the similarity score between the query vector and the each key vector, and determining the attention weight according to a plurality of similarity scores; and performing weighted summation on a plurality of value vectors according to the attention weight to generate a context vector, and generating the target semantic information according to the context vector, wherein the context vector is used to indicate the correlation degree between the first natural language instruction and each first metadata.

[0091] It can be understood that the self-attention mechanism of the Transformer can be used in the edge device to dynamically fuse metadata, so as to enhance the model's ability to analyze user instructions. Specifically:

[0092] 1) Linear transformation generates key-value pair and query vector:

[0093] The first metadata encoding vector is linearly transformed according to the key vector matrix and the value vector matrix stored in the edge device to generate the key vector (Key Vector) and the value vector (Value Vector) corresponding to each first metadata. The key vector is mainly used to indicate the semantic label of each metadata, such as labels like "bedroom light" and "brightness"; the value vector contains specific metadata information, such as "bedroom light" state "on" and brightness "50%".

[0094] At the same time, the first instruction encoding vector is linearly transformed according to the query vector matrix in the edge device to generate the query vector (Query Vector). The query vector represents the semantic demand of the user's first natural language instruction, such as "dim" and "bedroom light".

[0095] 2) Attention mechanism calculates similarity score and weight:

[0096] The edge device (or edge model) computes the dot product between the query vector and each key vector through an attention mechanism to determine their similarity scores. The similarity scores reflect the semantic relevance between the query vector and the metadata key vectors. Intuitively, a high score means high relevance, and a low score means lower relevance.

[0097] Based on the obtained multiple similarity scores, the model computes the attention weights for each key vector through a softmax function. The attention weights represent the importance of different metadata in the attention mechanism calculation.

[0098] The specific formula can be: Where Q is the Q vector (query vector), V is the V vector (value vector), K is the K vector (key vector), and attention() is the attention weight.

[0099] 3) Context vector generation and target semantic information:

[0100] According to the attention weights, the multiple value vectors are weighted and summed to generate a context vector. The context vector integrates the most relevant metadata information for the user instruction, and its size and direction depend on the matching degree of the metadata and the user instruction.

[0101] The edge model generates target semantic information based on the context vector, i.e., understands and analyzes the control intent after associating the metadata with the user instruction.

[0102] Taking the smart home control scenario as an example, suppose the user issues the instruction "Please dim the bedroom light a bit." At this time, the metadata includes "bedroom light" (on, brightness 50%), "living room light" (off), "air conditioner" (on, temperature 25°C), etc.

[0103] The user instruction encoding vector is transformed into a query vector through a query vector matrix, while each device's metadata encoding vector is transformed into a key vector and a value vector through a key vector matrix and a value vector matrix. For example, the key vector of the "bedroom light" device may reflect the "bedroom" and "light" semantic labels, and the value vector encodes the specific state information of "on" and "brightness 50%".

[0104] Dot product similarity calculation: the query vector and the key vector of the "bedroom light" are dot product calculated, and since they are highly relevant in semantics, a higher similarity score is obtained. In contrast, the dot product calculation with the key vectors of "living room light" or "air conditioner" results in much lower scores, as the user instruction does not involve these devices.

[0105] The similarity score determined by the softmax function is converted into attention weights, with the highest attention weight for the "bedroom light", representing the closest association between the user instruction and the metadata.

[0106] According to the attention weights, the value vectors of all devices are weighted and summed. Since the attention weight of the "bedroom light" is the highest, its value vector contributes the most to the context vector, and the result will mainly reflect the state of the "bedroom light" and its association with the user instruction.

[0107] Based on the context vector, the model generates target semantic information, i.e. executable control instructions. In this example, the model will generate the instruction "lower the brightness of the bedroom light to a lower level", thus achieving accurate response and intelligent control of the user's natural language command.

[0108] Optionally, after the above step S204 of fusing the plurality of first metadata and the first natural language instruction according to the attention weight to generate the target semantic information corresponding to the first natural language instruction, the method further comprises: storing the plurality of key vectors and the plurality of value vectors into a target cache space in the edge device; in the case of receiving a second natural language instruction corresponding to the first scene, encoding the second natural language instruction to generate a second instruction encoding vector; adjusting the attention weight according to the second instruction encoding vector and the first metadata encoding vector, and fusing the plurality of first metadata and the second natural language instruction according to the adjusted attention weight to generate semantic information corresponding to the second natural language instruction.

[0109] It can be understood that the key vectors and value vectors of dynamic metadata can be stored and utilized on the edge device, and the attention weight can be encoded and adjusted when receiving a new instruction to fuse the metadata and the instruction to generate more accurate semantic information. Specifically:

[0110] 1) Store the key vectors and value vectors:

[0111] Store each key vector and value vector generated by the dynamic fusion mechanism into the target cache space of the edge device. These vectors store the semantic labels and specific content of the metadata, which are designed to be quickly extracted and utilized in subsequent user instruction processing.

[0112] 2) Encode the new natural language instruction:

[0113] When the edge device receives a new natural language instruction, such as "please turn up the volume of the living room TV", it will be converted into a second instruction encoding vector by the word analyzer and encoder. The second instruction encoding vector captures the semantic features of the new instruction.

[0114] 3) Adjusting attention weights:

[0115] Using the key vectors and value vectors stored in the cache space, and the newly generated second instruction encoding vector, the new attention weights are calculated through the attention mechanism. The attention weights reflect the degree of association between the new instruction and each metadata item.

[0116] 4) Fusion processing and generating semantic information:

[0117] According to the adjusted attention weights, the edge model dynamically fuses and processes the stored metadata. In this stage, the metadata with high degree of association with the new instruction is considered more, while the metadata with low degree of association has less impact. Finally, the result of fusion processing will generate the semantic information corresponding to the second natural language instruction, that is, the deep understanding of the user's intention.

[0118] For example: in the smart home control scenario, assume that the system has converted the metadata of all room devices into corresponding key vectors and value vectors, and stored them in the cache space of the edge device. The metadata here may include the names, states, positions, etc. of devices such as "bedroom light", "living room light", "kitchen faucet", etc. have been encoded and converted into corresponding vector representations.

[0119] When the user subsequently issues a new instruction: "Turn up the volume of the TV in the living room", the instruction can be processed through the following steps:

[0120] Step 1: Encode the new instruction, convert the new instruction through the word analyzer and encoder into the second instruction encoding vector. Assume that the encoded vector is: living room, TV, volume, turn up.

[0121] Step 2: Adjust the attention weights, use the key vectors (such as semantic tags representing "living room" and "TV") and value vectors (such as specific states representing "TV volume") stored in the cache space, and the second instruction encoding vector of the new instruction to perform self-attention mechanism calculation. Through dot product and softmax function, the device with high relevance to "living room TV" (such as "TV in the living room") has a higher attention weight, while the device with low relevance to other devices (such as "bedroom light") has a lower attention weight.

[0122] Step 3: Fusion processing and generating semantic information, according to the adjusted attention weights, all stored metadata are weighted and summed, focusing on the metadata related to "TV in the living room". The fused context vector will fully carry the current position, state, etc. of the TV in the living room. Based on this context vector, the model generates the semantic information of the new instruction, clearly understanding that the user wants to "turn up the volume of the TV in the living room".

[0123] Optionally, after the step S204 of fusing the plurality of first metadata and the first natural language instruction according to the attention weight to generate the target semantic information corresponding to the first natural language instruction, the method further comprises: in the case that the first scene has changed into a second scene or a second metadata has been added in the first scene, obtaining a first parameter of a sub-network in the edge device, and adjusting the first parameter based on an incremental adaptation network to generate a second parameter of the sub-network, wherein the first parameter is used for adaptation with the first metadata, and the second parameter is used for adaptation with a third metadata or the second metadata, and the third metadata is other metadata in the second scene except the first metadata; and performing semantic recognition on a third natural language instruction based on the edge device containing the sub-network adjusted to the second parameter, wherein the third natural language instruction is a natural language instruction corresponding to the second scene or the first scene after the second metadata is added.

[0124] That is, the application can also embed an independently trainable lightweight sub-network in the edge device, and only by adjusting the parameters of the sub-network (i.e., adjusting the first parameter) can the new metadata be adapted, keeping the main model parameters fixed; the application can also design a standardized interface to support the instant loading and coding of new metadata, automatically absorb new information through the adaptation network, and realize the dynamic optimization of model behavior.

[0125] The parameters of the subnetwork are adjusted to implement semantic recognition of a third natural language instruction based on an edge device of the subnetwork after the adjustment of the second parameter, including: inserting a low-rank adaptation mechanism into a target projection layer in the edge device to decompose a target weight matrix corresponding to the target projection layer into an original vector matrix and a first low-rank vector matrix according to the low-rank adaptation mechanism, wherein the target projection layer at least includes one of: a first projection layer corresponding to a key vector matrix in the edge device, a second projection layer corresponding to a value vector matrix in the edge device, and a third projection layer corresponding to a query vector matrix in the edge device, and the target weight matrix at least includes one of: the key vector matrix, the value vector matrix, and the query vector matrix; in a case where it is detected that the first scene has changed to a second scene or a second metadata has been added in the first scene, freezing the original vector matrix, and updating the first low-rank vector matrix according to a plurality of fourth metadata in the second scene or the second metadata to determine a second low-rank vector matrix corresponding to the second scene, wherein the fourth metadata includes the third metadata; in a case where a third natural language instruction corresponding to the second scene or the first scene after the second metadata is added is received, calling the second low-rank vector matrix, and performing semantic recognition of the third natural language instruction according to the edge device after the second low-rank vector matrix is called.

[0126] It can be understood that the low-rank adaptation (LoRA) mechanism can be used to dynamically update the model parameters on the edge device for the changed scene or the newly added metadata without affecting the overall model structure and stability. Specifically:

[0127] 1) Insert a low-rank adaptation mechanism: in the target projection layer (such as the first projection layer corresponding to the key vector matrix, the second projection layer corresponding to the value vector matrix, and the third projection layer corresponding to the query vector matrix) of the edge device, the target weight matrix is decomposed into two parts: the original vector matrix and the first low-rank vector matrix by the low-rank adaptation (LoRA) mechanism. The original vector matrix remains fixed and represents the basic structure of the model; while the first low-rank vector matrix has fewer parameters and is used to adapt to the change of the scene or the addition of new metadata.

[0128] 2) Update the low-rank vector matrix: when it is detected that the first scene has changed to a second scene (for example, from "day mode" to "night mode"), or a second metadata has been added in the first scene (such as a new device "coffee machine"), the model will freeze the original vector matrix to avoid interference with the parameters of the basic model. Then, according to a plurality of third metadata (such as device status in night mode) in the second scene or the second metadata, the first low-rank vector matrix is updated to generate a second low-rank vector matrix that adapts to the new scene or the new device.

[0129] 3) Semantic recognition and control instruction generation: When the edge device receives the third natural language instruction, the model will call the updated second low-rank vector matrix. This means that without modifying the underlying model, the model can accurately recognize the semantics and generate control instructions based on the new scene or device information carried in the second low-rank vector matrix.

[0130] For example: Assuming in a smart home environment, the initial scene is "day mode", which has been initialized through the low-rank adaptation mechanism of key, value, and query vector matrix, and can understand and execute various instructions in "day mode", such as "close the curtains", "turn on the living room lights", etc.

[0131] When the scene changes to "night mode" (i.e. the second scene), it needs to adapt to the changes in lighting requirements in night mode, which may include: the state of the curtains is no longer concerned; the demand for night light is added; the light brightness adjustment strategy of the bedroom is changed.

[0132] At this time, dynamic adaptation is needed:

[0133] Insert and update low-rank vector matrix: Insert the low-rank adaptation mechanism in the first projection layer corresponding to the key vector matrix, and decompose the original key vector matrix into the original vector matrix and the first low-rank vector matrix. Then, for the second scene of "night mode", only update the device and scene information related to night mode in the first low-rank vector matrix, such as night light on, bedroom light brightness adjustment, etc., to generate a second low-rank vector matrix that adapts to night mode. During this process, the original vector matrix remains unchanged, ensuring the stability of the model's underlying structure.

[0134] Process instructions: In the night mode, assume the user issues the instruction "please turn on the night light and adjust the bedroom light". The model calls the second low-rank vector matrix and uses its night mode information to recognize the semantics of this instruction. By adjusting the attention weight, the model can correctly identify the state adjustment instructions of the night light and the bedroom light, while ignoring information unrelated to the current scene (such as the curtain state).

[0135] Generate control instructions: The edge model generates specific control instructions, including "turn on the night light" and "dim the bedroom light to the appropriate sleep brightness". These instructions directly reflect the user's needs in the night scene, without relying on traditional rule engines or retraining in the cloud, improving response speed and privacy security.

[0136] Optionally, according to the low-rank adaptation mechanism, determine each third low-rank vector matrix corresponding to each third scene, and in the case of receiving a fourth natural language instruction corresponding to any third scene, call any low-rank vector matrix corresponding to the any third scene; according to the edge device calling the any low-rank vector matrix, perform semantic recognition on the fourth natural language instruction.

[0137] It can be understood that the low-rank adaptation mechanism (LoRA) can be used to dynamically adapt different scenes on the edge device and perform semantic recognition on natural language instructions within the scene. Specifically:

[0138] Determine the low-rank vector matrix of the scene according to LoRA: To adapt to each third scene, a third low-rank vector matrix corresponding to each scene is determined using the low-rank adaptation mechanism. The third low-rank vector matrix is generated by the LoRA mechanism when the edge model is initialized or when the scene changes, and carries scene-specific adaptation information, which is used to fine-tune the model's behavior in a specific scene.

[0139] Call the low-rank vector matrix of the scene: When receiving a fourth natural language instruction corresponding to any third scene, the edge device will call the low-rank vector matrix corresponding to the scene. For example, if the instruction "Please adjust the living room lights to the viewing mode" is received, the low-rank vector matrix under the "viewing mode" scene can be called to more accurately understand the specific context and requirements in the instruction.

[0140] Semantic recognition and control: After calling the low-rank vector matrix of a specific scene, the edge device can perform more accurate semantic recognition based on this matrix. This means that the edge model can more intelligently parse instructions and take into account the special nature of the scene and the dynamic changes of the metadata. For example, in the "viewing mode" scene, the edge model will understand that "adjust to viewing mode" means adjusting the light to soft and reducing the brightness to create a suitable environment for viewing.

[0141] For example: In a smart home system, suppose there are multiple specific scenes such as "viewing mode", "reading mode", "away mode", etc., each corresponding to different device states and user preferences. At this time, the LoRA mechanism can generate a corresponding low-rank vector matrix for each scene during system deployment or when the user changes the scene configuration. For example, for "viewing mode", the system will generate a matrix that will help the model understand viewing-related device operations such as reducing light brightness, closing curtains, etc.

[0142] When the user issues the instruction "dim the lights in the living room and draw the curtains" in the "movie mode", the edge device will immediately call the low-rank vector matrix corresponding to the "movie mode". This matrix will guide the model to more accurately identify the specific intent in the user's instruction, such as dimming the lights in the living room and drawing the curtains, which are typical operations in the movie mode.

[0143] After calling the low-rank vector matrix, the model will perform semantic recognition based on its content in combination with real-time user input (such as the aforementioned instruction). The model not only understands the user's specific instruction, but also automatically infers that "dim" means adjusting the living room lights to a brightness suitable for movie watching and "draw the curtains" ensures privacy and optimal visual effect during movie watching according to the pre-set scene rules in the "movie mode". Subsequently, the edge device will generate and execute the corresponding control instructions directly based on the recognized semantic information, without the intervention of the cloud or the matching of artificial rules, thereby improving the response speed and privacy security of the system.

[0144] In order to better understand the process of the above-mentioned semantic information generation method, the implementation method flow of the above-mentioned semantic information generation will be described in combination with optional embodiments below, but not used to limit the technical solutions of the embodiments of the present application.

[0145] The following explains the proprietary terms that may be involved in the optional embodiments of the present application:

[0146] 1) Rule engine is a technology for automatically executing decision-making processes in large-scale data sets and complex business environments; rule engine uses rule expressions and conditional statements to filter and process input data, and generates corresponding output results according to pre-defined rules;

[0147] 2) Token refers to the data unit input into the system. In the field of natural language processing, it refers to the basic units such as words and phrases after the text is segmented, which are further processed and analyzed as input for various computing tasks;

[0148] 3) Edge device refers to the device located at the forefront of the Internet of Things architecture, which is directly connected to the cloud or data center and generates data exchange. Edge devices can be smart devices, sensors, smartphones, embedded systems, etc., and can perform computing, storage and control operations. Edge devices can process data and make decisions in real time, perform various tasks and operations, thereby providing users with better experience and service.

[0149] The edge device metadata fusion method in the related art has the following problems:

[0150] 1) Static model limitations: Traditional edge devices cannot dynamically absorb external environment metadata (e.g., device configuration, scene information), resulting in insufficient parsing capabilities for new devices or complex intentions;

[0151] 2) Rule engine dependency: Existing systems rely on manual rules to match user instructions with device operations, which is inflexible and difficult to adapt to high-frequency changes in dynamic environments;

[0152] 3) Input redundancy and computational power contradiction: Related technologies significantly increase computational load by explicitly concatenating metadata to input tokens, making it difficult for edge devices to support real-time responses.

[0153] In summary, there is an urgent need for an edge localization model architecture that can achieve efficient intention parsing through dynamic metadata fusion without relying on artificial rules or cloud collaboration, while supporting seamless incremental adaptation to new environment configurations to meet real-time, privacy, and personalized needs.

[0154] Based on the above technical problems, the optional embodiments of the present application propose a dynamic metadata fusion method based on edge devices and an intelligent device control system, which is used to realize dynamic metadata fusion and adaptive reasoning on resource-constrained edge devices, and is suitable for localized intention parsing and control in smart home, industrial Internet of Things and other scenarios.

[0155] The optional embodiments of the present application can solve the following problems: 1) Dynamic metadata fusion: how to enable edge devices to integrate dynamically changing metadata information during reasoning, avoiding the waste of computational power caused by explicit input concatenation; 2) Rule-free adaptation: how to replace artificial rules with endogenous mechanisms to achieve precise matching of complex intentions and device configurations; 3) Lightweight incremental learning: how to support continuous dynamic loading of subsequent metadata through single initial training without global model retraining.

[0156] Figure 3 is the architecture diagram of the intelligent device control system based on edge devices according to the optional embodiments of the present application, as Figure 3 shown:

[0157] Figure 3 includes a user interaction interface, edge hardware, a communication module, intelligent devices, and an IoT cloud, specifically:

[0158] (1) User interaction interface: includes a microphone and a speaker, users issue instructions through voice, and the system feedbacks control results or asks for more information through the speaker.

[0159] (2) Edge hardware (i.e., edge device):

[0160] 1) Edge Large Language Model (ELL): responsible for processing first natural language instructions, analyzing intent, and generating control instructions. ELL contains functions such as Natural Language Understanding (NLU), knowledge reasoning, sentiment analysis, Natural Language Generation (NLG), etc.

[0161] 2) Meta-awareness and Meta-loading: responsible for collecting and updating the metadata information of home devices, such as device name, type, location, and status, etc., and integrating these information into the decision-making process of ELL in real time.

[0162] 3) Home device metadata: stores all metadata related to home devices, providing environmental information for the model.

[0163] (3) Communication module: through various wireless communication technologies (such as Zigbee, Bluetooth Low Energy Mesh Network (BLEMesh), Open Thread Networking (OpenThread), Wireless Fidelity Technology (Wi-Fi)), data transmission with smart devices is realized, and control instructions and state information are returned.

[0164] (4) Smart devices (i.e. smart device 1 and smart device 2 in the figure): containing all connected smart devices in the home, such as lights, air conditioners, televisions, security systems, etc., smart devices execute corresponding operations according to the control instructions received from the edge hardware.

[0165] (5) IoT cloud: although the edge hardware is mainly responsible for local processing, the IoT cloud still plays an important role in the system, which can be used for data backup, remote monitoring, system upgrade, etc.

[0166] Figure 4 is a flowchart of a dynamic metadata fusion method based on edge devices according to an optional embodiment of the present application, which is applied to the system of Figure 3 , as shown in Figure 4 :

[0167] Step S401, incremental metadata loading;

[0168] Incremental metadata refers to metadata that is dynamically added or updated during system operation, reflecting real-time changes in the home environment or device configuration. It can include: 1) new device information: when new smart devices are added to the user's home, such as newly purchased smart light bulbs, smart sockets, etc., the system needs to obtain and load detailed information about these new devices, including device name, type, location, function, etc.; 2) attribute updates: as devices are used, their attributes may change, such as device status (on / off), location changes, or device function upgrades, which need to be updated in a timely manner; 3) scene configuration changes: users may adjust home scene settings according to their needs, such as adding or removing devices in "movie mode", or adjusting device behavior in the mode, such as reducing light brightness or adjusting temperature settings; 4) user preferences and habits: the system will learn and record user device operation preferences at specific times or in specific situations, such as preferring dimmed lighting in the evening or a quiet environment when reading.

[0169] Step S402, load the home metadata (i.e. the first metadata);

[0170] According to the incremental metadata and the initialized home metadata information, load the home metadata.

[0171] Step S403, encode the home metadata through the metadata encoding module;

[0172] Step S404, obtain the natural language instruction input by the user (i.e. the first natural language instruction);

[0173] Step S405, parse the natural language instruction through the semantic analysis module;

[0174] Step S406, dynamically fuse the encoding result (metadata) corresponding to the metadata encoding module of step S403 and the parsing result (semantics) corresponding to the semantic analysis module of step S405 through the dynamic fusion layer;

[0175] Among them, for step S406, the optional embodiment of the present application limits two kinds of metadata dynamic fusion mechanisms:

[0176] The first metadata dynamic fusion mechanism: to fuse the semantic information of the environment metadata on the edge device and avoid input redundancy, a Prompt splicing + KV cache mechanism combined way is adopted to realize dynamic fusion, specifically:

[0177] Step S406-1-1, system initialization and construction of prompt string (Prompt);

[0178] Step S406-1-1-1, initialize the Prompt template;

[0179] At the beginning of system deployment or when the device encounters new environment metadata, an Prompt template is automatically built, which contains static environment information and device detailed description.

[0180] Step S406-1-1-2, metadata texturization;

[0181] The home metadata can contain structured information in the user environment, such as: room device list (Device_List), capability attributes of each device (such as: type, state, location), scene labels (such as: "good night mode", "movie mode"), etc. The original metadata is in structured JavaScript Object Notation (JSON, a lightweight data exchange format) format. The structured metadata is converted into natural language text format to form a text describing the home environment and device state, and then the structured metadata in natural language text format is spliced to the front of the user input.

[0182] Step S406-1-1-3, user instruction positioning;

[0183] In the construction of Prompt, the boundary between system description information and user natural language instruction is clearly divided to ensure that the model can clearly distinguish between input instruction and environment metadata.

[0184] Figure 5 is a Prompt template diagram constructed according to an optional embodiment of the present application, as shown in Figure 5 The constructed Prompt template can include the following pure text mode:

[0185] System prompt template: you are a smart home assistant, the following is the room configuration: Device1 (suggestion 1): name (name) = bedroom light, location (device location) = bedroom, type (device type) = light; Device2 (suggestion 2): name (name) = air conditioner, location (device location) = living room, type (device type) = AC (air conditioning device); User instruction: turn off the light in the bedroom.

[0186] Figure 5 The Prompt template of is only an example, and the Prompt template can be modified according to actual needs in specific applications.

[0187] Figure 5The Prompt template in the Prompt constitutes a complete Prompt, which is sent to the edge large language model after tokenization.

[0188] The tokenizer is a computer technology for processing text, which decomposes text (i.e., the text obtained after the home metadata is converted into a natural language description and spliced to the front part of the user input) into individual tokens, i.e., basic units such as words, numbers, symbols, etc. in the text.

[0189] All content in the Prompt is encoded into a token sequence using the same tokenizer as the main model and embedded into a unified embedding space (i.e., through embedding operation, the original data (e.g., words in the text) is mapped to this space, so that similar entities are closer in space), without additional training of the encoder.

[0190] Step S406-1-2, pre-filling and processing of the Key-Value Cache (KV Cache);

[0191] Step S406-1-2-1, pre-filling the KV cache;

[0192] When the edge large language model is first deployed or new metadata is loaded, the static metadata part (i.e., the part not containing user input) in the Prompt template is sent to the encoder of the edge large language model for processing. The encoder converts each token in the static metadata into a corresponding embedding vector, and calculates the key vector (Key) and value vector (Value) for each position through the attention mechanism, and stores them in the cache (i.e., KV cache).

[0193] The attention mechanism is a technology that simulates human cognitive attention by enhancing the weight of important parts of the input data of the neural network and reducing the weight of other parts, focusing the attention of the network on a small part of the most important data. The context determines which parts are more important, which can be trained through gradient descent.

[0194] Step S406-1-2-2, cache hierarchy;

[0195] The KV cache is designed in two layers, the static layer and the dynamic layer. The static layer retains the Key and Value of the device inventory, attributes, and scene configuration, while the dynamic layer is used to store the real-time calculation results of the user instruction token, facilitating immediate fusion and reasoning. That is:

[0196] Static layer: store the system prompt part (can be reused across users / sessions);

[0197] Dynamic layer: only save current user input (Token level short-lived).

[0198] The hierarchical caching can be quickly implemented by, for example, using the CacheEngine such as vLLM (Large Language Model Deployment Accelerator) or TensorRT-LLM (a toolkit designed for accelerating large language model inference) prefill_cache.

[0199] Step S406-1-3, user input and dynamic fusion;

[0200] Step S406-1-3-1, user input processing;

[0201] When the user issues an instruction, for example: “turn on the air conditioner in the bedroom”, first transcribe this instruction into text (for voice instructions, it can be completed by ASR). Further, the natural language instruction is encoded into a token sequence by the same tokenizer;

[0202] Step S406-1-3-2, dynamic Prompt splicing;

[0203] The user instruction token is added to the end of the Prompt template, and the static metadata together constitute the complete context.

[0204] Step S406-1-3-3, Query (query vector) calculation;

[0205] In the self-attention mechanism, each spliced template is converted into Query vector, Key vector and Value vector through three different linear transformation layers. However, in order to avoid re-computing the Key and Value vectors of the static metadata part (such as device list, device state, etc.) at each inference, for each token of the user input part, the model only needs to calculate its Query vector, and the Key and Value vectors are obtained from the results of KV cache.

[0206] Step S406-1-3-4, dynamic fusion;

[0207] In the self-attention mechanism, the Query vector and the Key vector in the cache calculate the attention weight (i.e. the similarity between the Query vector and each Key vector, the higher the similarity, the greater the attention weight, indicating that the current user input is more relevant to the corresponding metadata), and according to the similarity dynamic adjustment, the implicit fusion with the metadata is realized.

[0208] Among them, the Attention Score calculated by the model tends to focus on semantically related device fields, for example: "bedroom light" is highly related to "turn off the light"; if the user does not involve a certain type of device (such as air conditioner), the attention weight of the related token is naturally lower, achieving the effect of "implicit fusion and dynamic weight control"; the Prompt structure adopts segmentation, line break and comma separation to help the model better distinguish the token range of different metadata items.

[0209] The Query vector will perform a dot product operation with all Key vectors in the KV cache to calculate an attention weight score, and then weight average all Value vectors in the KV cache according to the attention weight score to generate the Context vector of the current token (attention context vector, containing the essence of the most relevant metadata to the current user instruction, guiding the model's decision, for example: generating operation instructions to control smart devices).

[0210] The second metadata dynamic fusion mechanism: after encoding the user's natural language instruction and the device-side metadata into semantic vectors and metadata vectors respectively, and mapping them to a unified low-dimensional space, they are used as the input of the subsequent attention mechanism. Specifically:

[0211] Step S406-2-1, encode the user's natural language instruction (User Instruction).

[0212] Input: natural language instruction, for example, "turn on the air conditioner in the living room".

[0213] Encoding method: use a lightweight Transformer encoder, for example: DistilBERT, SmallBERT or MobileBERT, to encode the natural language instruction.

[0214] Among them, the Transformer encoder is the part of the Transformer architecture responsible for encoding the input data, which is a neural network architecture based on self-attention mechanism.

[0215] Output features: by default, the length of the encoded instruction sequence is limited to 64 tokens, and the dimension of the output vector is d1, which is set to 128 dimensions (the above data are examples, which can be adjusted according to actual needs in specific applications).

[0216] Output: a vector (Vec_instruction) containing instruction semantic information.

[0217] That is, the instruction encoding submodule uses a Transformer-based Encoder (such as Distil BERT) to encode the text, and the output is the average pooling vector of the entire instruction sequence or the token vector of the [CLS] (classification) position. The output dimension is: (the instruction vector belongs to a 128-dimensional real number space).

[0218] Step S406-2-2, metadata encoding;

[0219] The metadata encoding submodule adopts different processing methods for each type of metadata field, and the results are spliced and linearly transformed and reduced in dimension.

[0220] Figure 6 is a metadata encoding source diagram according to an optional embodiment of the present application, as Figure 6 shown, the metadata encoding source: structured input, including device type (for example: air conditioner, encoding method can be: lookup table + trainable embedding), device state (for example: on, offline, encoding method can be: one-hot + FC dimension reduction (one-hot encoding + fully connected layer dimension reduction)), room position (for example: living room, which can be encoded as: trainable embedding), and context scene label fields (for example: energy saving mode, evening, encoding method can be trainable embedding).

[0221] Encoding process:

[0222] Device type encoding: for example, "air conditioner"→ (the device type embedding vector belongs to a 32-dimensional real number space);

[0223] Device state encoding: for example, "on"→ one-hot encoding→ FC dimension reduction to 16 dimensions→ (the state embedding vector belongs to a 16-dimensional real number space);

[0224] Room position encoding: for example, "living room"→ (the room embedding vector belongs to a 32-dimensional real number space);

[0225] Scene label encoding: for example, "energy saving mode"→ (the scene embedding vector belongs to a 32-dimensional real number space).

[0226] Splicing and dimension reduction: splice multiple encoded vectors to obtain (the original metadata vector belongs to a 112-dimensional real number space); and then reduce it to (metadata vector belongs to 64-dimensional real number space).

[0227] Step S406-2-3, feature alignment (mapping to the same space);

[0228] Objective: since the two vectors have different dimensions and , it is necessary to unify the dimensions of Vec_instruction and Vec_meta so as to be fused subsequently.

[0229] Method: use a linear transformation matrix to map the two vectors to a common fusion space, such as

[0230] Output:

[0231] Instruction encoding vector: V u (96-dimensional instruction encoding vector) = W u (instruction weight matrix) * Vec_instruction (128-dimensional instruction vector).

[0232] That is, the user instruction vector is mapped from the original 128-dimensional space to the 96-dimensional space through a linear transformation of a weight matrix.

[0233] Metadata encoding vector: V m (96-dimensional metadata encoding vector) = W m (metadata weight matrix) * Vec_meta (64-dimensional metadata vector);

[0234] wherein, (96 rows and 128 columns) and (96 rows and 64 columns) are trainable parameter matrices.

[0235] Step S406-2-4, context-aware attention distribution:

[0236] Input splicing: splice the metadata encoding vector and the instruction encoding vector into a Prompt, and input the Prompt into the model, wherein the Prompt contains system description, device configuration metadata and user instruction;

[0237] Unified participation Attention (attention mechanism): all tokens, whether from user instruction or metadata, participate in weight calculation equally in the self-attention mechanism, so that the model can judge which metadata tokens are more relevant to the user instruction semantics according to the current context.

[0238] Implicit association modeling: in the Attention layer of the model, the model automatically assigns higher weights to the metadata tokens that are more relevant to the current context through learning, realizing the implicit fusion between instructions and metadata.

[0239] Prompt-driven training: by designing various prompts, the model can learn how to use metadata to improve the accuracy of instruction understanding and how to respond more accurately in different scenarios.

[0240] Through the above steps, the semantic aggregation vector after closely combining the meaning of the natural language instruction with the details of the environmental metadata can be determined (which helps the model to generate more accurate device control instructions).

[0241] Step S407, metadata awareness, inject metadata vectors at the model input layer or intermediate layer, and dynamically adjust the sensitivity of the model to metadata by combining lightweight parameter adaptation technology;

[0242] In order to support the dynamic adaptation capability of edge devices when the environment changes and new devices are added, the optional embodiment of the present application proposes a lightweight fine-tuning scheme based on Low-Rank Adaptation (Low-Rank Adaptation, referred to as LoRA), specifically:

[0243] LoRA can be understood as a patch for large models, used to modify styles / objects. The principle of LORA is to freeze the pre-trained model weight parameters, and then inject trainable layers in each Transtormer block (use attention mechanism to improve model training speed). Since it does not need to recalculate the gradient of the model weight parameters, it can reduce the parameter amount and computational amount, improve the training efficiency and generation quality.

[0244] The lightweight fine-tuning scheme based on low-rank adaptation mainly increases a small multi-layer perceptron (MLP) sublayer (e.g., 256→64→256) after each Transformer Block (target projection layer) through an Adapter module to adjust the output through a residual path. The Adapter is an interface adapter that converts an interface of a class into an interface expected by a client. The specific process includes: 1) inserting a LoRA definition module in the Attention layer of the model positioning; 2) keeping all parameters of the original model frozen, and only training the LoRA actual parameters A / B; 3) inputting fine-tuning samples (forming a weakly supervised pair through user instructions + controlling actual behavior), and performing local or asynchronous fine-tuning training; after fine-tuning, only the LoRA parameters (such as adapter_model.bin) are exported and issued, and the edge is continued to run; 4) using the Transformers+PEFT (Parameter Efficient Fine-Tuning) framework to define the LoRA configuration.

[0245] LoRA implementation principle: in the Transformer Attention layer, the Query / Value weight matrix W (i.e., the value vector matrix, the query vector matrix, i.e., the original vector matrix) is decomposed into: W' = W + ΔW = W + BA, where W' is the updated model weight matrix, ΔW represents the update amount or the increment matrix of the weight matrix W (i.e., the second low-rank vector matrix), B and A are two low-rank matrices (i.e., the first low-rank vector matrix), Rank (matrix rank) = 4-8, which only occupies a very small part (<1%) of the original model parameter amount.

[0246] When new devices or scenarios are added, the model behavior can be updated in the following ways:

[0247] Data collection method: record user voice commands (ASR) and actual control behavior (through home automation log) as weakly supervised training samples; wherein the home automation log refers to a log file that records various operations, events and state changes related to the home automation system.

[0248] Training trigger mechanism: trigger the fine-tuning process periodically or when user behavior changes significantly (e.g., once a night, or update when 3 new devices are added);

[0249] The update execution manner is: incremental training is performed locally (such as supporting an EdgeTPU / NPU (Edge Tensor Processing Unit / Neural Network Processing Unit) environment); or an asynchronous offline fine-tuning strategy is adopted (training during an idle period or a network outage); or if the edge device is limited, a Federated Learning (federated learning) is used to aggregate multi-user data for unified fine-tuning and then the parameter delta is issued.

[0250] The federated learning is a distributed machine learning method. Under the method, multiple users (or clients) train a model on local devices using their own data, each user sends the parameter update information (such as the parameter delta) of the local model to the center server, the center server aggregates the parameter update information to optimize the global model, and then issues the optimized model parameters to each user for updating. Under the premise of protecting user data privacy (data does not need to leave the local device) and reducing data transmission costs (without the need to transmit a large amount of original data), the joint training and optimization of the model are realized, and the method is suitable for scenarios where edge devices are limited.

[0251] The LoRA module supports "hot plug" and does not need to recompile the model structure, and can be enabled as needed. Multiple sets of adaptive parameters can be saved according to the user's home ID (identifier) or scene ID, and version management can be performed in combination with config.json and weights. The above technical solution supports the federated learning mechanism: multiple edge devices collect behavior samples through logs and device operation records; local fine-tuning forms incremental weights; the server side aggregates multiple LoRA weight parameter differences, optimizes them uniformly, and then issues a new version; and the optional embodiment of the present application also supports a custom sample weighting strategy (such as preferentially using data in the past 7 days) during training to avoid the problem of historical drift.

[0252] In summary, the parameter efficient incremental adaptation module of the optional embodiment of the present application includes: a lightweight parameter adaptation network: an independently trainable lightweight subnetwork is embedded in the model, and only the parameters of the subnetwork need to be adjusted to adapt to new metadata, and the main model parameters remain fixed; an incremental metadata injection interface: a standardized interface is designed to support the instant loading and encoding of new metadata, and new information is automatically absorbed through the adaptation network to realize the dynamic optimization of the model behavior.

[0253] Step S408: performing intent analysis through the intent analysis module;

[0254] Step S409: generating a control instruction through the control instruction generation module;

[0255] Step S410: sending the control instruction to the device to be controlled through the local communication module.

[0256] Through the optional embodiments of the present application, rule-free autonomous reasoning logic can be realized: through deep fusion of metadata and semantic information, the model autonomously generates device control instructions, completely abandoning the hard-coded matching process of the traditional rule engine.

[0257] In summary, the optional embodiments of the present application disclose an edge device adaptive reasoning method based on dynamic metadata fusion. Through three core technologies of endogenous metadata fusion, parameter efficient incremental adaptation and rule-free autonomous reasoning, efficient and privacy-safe intelligent control is realized on the edge device. And it breaks through the limitations of traditional rule engines and static models, so that the model can dynamically adapt to complex environmental changes through single training, providing an innovative paradigm for lightweight and adaptive edge AI in Internet of Things scenarios. Through the optional embodiments of the present application, the following effects can be achieved: endogenous metadata fusion: the model realizes implicit fusion and dynamic adaptation of metadata through architecture-level design, without external rules or manual intervention; single training multi-scene generalization: after initial training, the model automatically absorbs incremental metadata through a lightweight adaptation network to adapt to different home environments and device configurations; edge computing friendliness: the fusion and adaptation process are based on low-dimensional vector operations, and the algorithmic demand is reduced by more than 80%, suitable for low-power edge devices.

[0258] Through the description of the above implementation manners, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better implementation manner. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method of each embodiment of the present application.

[0259] Figure 7 is a structural block diagram of a semantic information generation device according to an embodiment of the present application; as shown in Figure 7 , it includes:

[0260] The encoding module 72 is configured to encode a plurality of first metadata in the first scene and a first natural language instruction corresponding to the first scene and sent by a target object, wherein the first metadata at least includes one of the following: a device list in the first scene, a capability attribute of each device in the device list, and a scene label of the first scene.

[0261] The determining module 74 is configured to determine an attention weight corresponding to an attention mechanism in the edge device according to the encoded target vector, and perform fusion processing on the plurality of first metadata and the first natural language instruction according to the attention weight to generate target semantic information corresponding to the first natural language instruction, wherein the target vector includes a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction.

[0262] Through the above device, the plurality of first metadata (including a device list in the first scene, a capability attribute of each device, a scene label, etc.) in the first scene and the first natural language instruction corresponding to the first scene sent by the target object are encoded, and an attention weight corresponding to an attention mechanism in the edge device is determined according to an encoded target vector (including a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction). The plurality of first metadata and the first natural language instruction are fused to generate target semantic information corresponding to the first natural language instruction. That is, the metadata and the natural language instruction are encoded in the application, and then the attention weight is dynamically adjusted, and the attention mechanism is used to dynamically adjust the degree of combination of the metadata and the natural language instruction. Through the above technical solution, the edge device can autonomously integrate dynamically changing metadata information and flexibly focus on the metadata most relevant to the current natural language instruction. Even if the metadata changes, the above technical solution can quickly respond, solving the problem that the edge device in the related art cannot autonomously integrate dynamically changing metadata information in the reasoning process, and thus the edge device can autonomously integrate dynamically changing metadata information in the reasoning process.

[0263] In one example embodiment, the encoding module 72 is further configured to convert each metadata into a natural language description, and splice the plurality of natural language description metadata and the first natural language instruction to generate a first prompt string; perform word segmentation processing on the first prompt string through a word unit analyzer to generate a plurality of first word units, wherein the plurality of first word units include a plurality of second word units corresponding to the plurality of first metadata and a plurality of third word units corresponding to the first natural language instruction; and uniformly encode the plurality of first word units into a vector embedding space to generate the first instruction encoding vector and the first metadata encoding vector.

[0264] In an example embodiment, the encoding module 72 is further configured to parse the first natural language instruction to extract position metadata corresponding to the first natural language instruction, wherein the position metadata is used to indicate position information contained in the first natural language instruction; encode the position metadata, and splice the encoded position metadata with the first natural language instruction to obtain a second prompt string; filter device information corresponding to the second prompt string from the first metadata, and splice the device information with the second prompt string to obtain the first prompt string, wherein the device information and the position metadata have a corresponding relationship.

[0265] In an example embodiment, the encoding module 72 is further configured to encode the first natural language instruction to generate a first instruction vector, and encode each first metadata to generate a first metadata vector; determine a first dimension of the first instruction vector, a second dimension of the first metadata vector, and a third dimension of a common mapping space, respectively; determine an instruction mapping matrix corresponding to the first instruction vector according to the first dimension and the third dimension, and determine a metadata mapping matrix corresponding to the first metadata vector according to the second dimension and the third dimension; map the instruction vector to the common mapping space according to the instruction mapping matrix to generate a first instruction encoding vector, and map the metadata vector to the common mapping space according to the metadata mapping matrix to generate a first metadata encoding vector, wherein the vector dimensions of the first instruction encoding vector and the first metadata encoding vector are both the third dimension.

[0266] In an example embodiment, the encoding module 72 is further configured to perform word segmentation processing on the first natural language instruction by a word unit analyzer to generate a plurality of fourth word units; encode each fourth word unit by an encoder to generate a second vector corresponding to the fourth word unit; and integrate a plurality of second vectors to generate the first instruction vector.

[0267] In an example embodiment, the encoding module 72 is further configured to convert each first metadata into a text form, and perform word segmentation processing on each text form of the first metadata by a word unit analyzer to generate a plurality of fifth word units corresponding to the first metadata; encode the plurality of fifth word units corresponding to the first metadata by an encoding mechanism corresponding to the first metadata to generate a third vector corresponding to the first metadata, wherein the encoding mechanism corresponding to the first metadata containing different metadata information is different; and splice a plurality of third vectors to generate a metadata vector corresponding to the plurality of first metadata.

[0268] In an example embodiment, the determining module 74 is further configured to perform linear transformation on the first metadata encoding vectors according to a key vector matrix and a value vector matrix in the edge device respectively, to generate a key vector and a value vector corresponding to each first metadata, and perform linear transformation on the first instruction encoding vector according to a query vector matrix in the edge device, to generate a query vector, wherein the key vector is used to indicate a semantic label corresponding to each first metadata, and the value vector is used to indicate metadata information contained in the each first metadata; perform dot product calculation on the query vector and each key vector respectively according to the attention mechanism, to determine a similarity score between the query vector and the each key vector, and determine the attention weight according to a plurality of similarity scores; perform weighted summation on a plurality of value vectors according to the attention weight, to generate a context vector, and generate the target semantic information according to the context vector, wherein the context vector is used to indicate an association degree between the first natural language instruction and each first metadata.

[0269] In an example embodiment, the determining module 74 is further configured to store the plurality of key vectors and the plurality of value vectors into a target cache space in the edge device; in a case where a second natural language instruction corresponding to the first scene is received, encode the second natural language instruction to generate a second instruction encoding vector; adjust the attention weight according to the second instruction encoding vector and the first metadata encoding vector, and perform fusion processing on the plurality of first metadata and the second natural language instruction according to the adjusted attention weight, to generate semantic information corresponding to the second natural language instruction.

[0270] In an example embodiment, the apparatus further comprises an incremental adaptation module configured to: in a case where it is detected that the first scene has been changed to a second scene, or a second metadata has been added in the first scene, acquire a first parameter of a sub-network in the edge device, and adjust the first parameter based on an incremental adaptation network to generate a second parameter of the sub-network, wherein the first parameter is used for adaptation with the first metadata, and the second parameter is used for adaptation with a third metadata or the second metadata, the third metadata being other metadata in the second scene except the first metadata; and the determining module 74 is configured to perform semantic recognition on a third natural language instruction based on the edge device containing the sub-network adjusted to the second parameter, wherein the third natural language instruction is a natural language instruction corresponding to the second scene or the first scene after the second metadata is added.

[0271] Embodiments of the present application also provide a storage medium including a stored program, wherein the program performs any of the above methods when executed.

[0272] Optionally, in the embodiment, the storage medium can be configured to store program code for performing the following steps:

[0273] S1, encode a plurality of first metadata in a first scene and a first natural language instruction corresponding to the first scene and sent by a target object in the first scene, wherein the first metadata at least include one of the following: a device list in the first scene, a capability attribute of each device in the device list, and a scene label of the first scene;

[0274] S2, determine an attention weight corresponding to an attention mechanism in an edge device according to the encoded target vector, and fuse the plurality of first metadata and the first natural language instruction according to the attention weight to generate target semantic information corresponding to the first natural language instruction, wherein the target vector includes a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction.

[0275] Embodiments of the present application also provide an electronic device including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the steps in any of the above method embodiments.

[0276] Optionally, the electronic device can further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0277] Optionally, in the embodiment, the processor can be configured to execute the following steps through the computer program:

[0278] S1, encode a plurality of first metadata in a first scene and a first natural language instruction corresponding to the first scene and sent by a target object in the first scene, wherein the first metadata at least include one of the following: a device list in the first scene, a capability attribute of each device in the device list, and a scene label of the first scene;

[0279] S2, determine an attention weight corresponding to an attention mechanism in an edge device according to the encoded target vector, and fuse the plurality of first metadata and the first natural language instruction according to the attention weight to generate target semantic information corresponding to the first natural language instruction, wherein the target vector includes a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction.

[0280] The embodiment of the present application further provides a computer program product comprising a computer program which is executed by a processor to perform the steps in any of the method embodiments described above.

[0281] Optionally, in the embodiment, the computer program product described above can be executed by the processor to perform the following steps:

[0282] S1, encode a plurality of first metadata in a first scene and a first natural language instruction corresponding to the first scene and sent by a target object in the first scene, wherein the first metadata at least include one of the following: a device list in the first scene, a capability attribute of each device in the device list, and a scene label of the first scene;

[0283] S2, determine an attention weight corresponding to an attention mechanism in an edge device according to the encoded target vector, and perform fusion processing on the plurality of first metadata and the first natural language instruction according to the attention weight, to generate target semantic information corresponding to the first natural language instruction, wherein the target vector includes a first metadata encoding vector corresponding to the plurality of first metadata and a first instruction encoding vector corresponding to the first natural language instruction.

[0284] Optionally, in the embodiment, the storage medium described above can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0285] Optionally, specific examples in the embodiment can refer to examples described in the above embodiments and optional implementation manners, and the embodiment will not be described here.

[0286] Obviously, those skilled in the art should understand that each module or each step of the present application described above can be realized by a general computing device, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into each integrated circuit module respectively, or multiple modules or steps among them can be manufactured into a single integrated circuit module to realize. Thus, the present application is not limited to any specific hardware and software combination.

[0287] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for generating semantic information, characterized in that, include: Encode multiple first metadata and the first natural language instruction corresponding to the first scenario sent by the target object within the first scenario, wherein the first metadata includes at least one of the following: a device list within the first scenario, the capability attributes of each device in the device list, and the scenario tag of the first scenario; The attention weights corresponding to the attention mechanism in the edge device are determined based on the encoded target vector, and the plurality of first metadata and the first natural language instruction are fused based on the attention weights to generate target semantic information corresponding to the first natural language instruction. The target vector includes: the first metadata encoding vector corresponding to the plurality of first metadata and the first instruction encoding vector corresponding to the first natural language instruction.

2. The method for generating semantic information according to claim 1, characterized in that, Encoding the first metadata and the first natural language instruction corresponding to the first scenario sent by the target object within the first scenario, including: Each metadata is converted into a natural language description, and the metadata of multiple natural language descriptions is concatenated with the first natural language instruction to generate a first prompt string; The first prompt string is segmented by a lexical analyzer to generate multiple first lexical units, wherein the multiple first lexical units include multiple second lexical units corresponding to the multiple first data elements and multiple third lexical units corresponding to the first natural language instruction; The multiple first tokens are uniformly encoded into a vector embedding space to generate the first instruction encoding vector and the first metadata encoding vector.

3. The method for generating semantic information according to claim 2, characterized in that, The metadata of multiple natural language descriptions and the first natural language instruction are concatenated to generate a first prompt string, including: The first natural language instruction is parsed to extract the location metadata corresponding to the first natural language instruction, wherein the location metadata is used to indicate the location information contained in the first natural language instruction; The location metadata is encoded, and the encoded location metadata is concatenated with the first natural language instruction to obtain a second prompt string; The device information corresponding to the second prompt string is filtered from the first metadata, and the device information is concatenated with the second prompt string to obtain the first prompt string, wherein the device information has a corresponding relationship with the location metadata.

4. The method for generating semantic information according to claim 1, characterized in that, The method further includes encoding multiple first metadata within a first scenario and a first natural language instruction corresponding to the first scenario sent by a target object. The first natural language instruction is encoded to generate a first instruction vector, and each first metadata is encoded to generate a first metadata vector; Determine the first dimension of the first instruction vector, the second dimension of the first metadata vector, and the third dimension corresponding to the common mapping space, respectively. The instruction mapping matrix corresponding to the first instruction vector is determined based on the first dimension and the third dimension, and the metadata mapping matrix corresponding to the first metadata vector is determined based on the second dimension and the third dimension. The instruction vector is mapped to the common mapping space according to the instruction mapping matrix to generate the first instruction encoding vector, and the metadata vector is mapped to the common mapping space according to the metadata mapping matrix to generate the first metadata encoding vector, wherein the vector dimension of the first instruction encoding vector and the first metadata encoding vector are both the third dimension.

5. The method for generating semantic information according to claim 4, characterized in that, Encoding the first natural language instruction to generate a first instruction vector includes: The first natural language instruction is segmented using a lexical analyzer to generate multiple fourth lexical units; Each fourth word is encoded by an encoder to generate a second vector corresponding to each fourth word; Multiple second vectors are combined to generate the first instruction vector.

6. The method for generating semantic information according to claim 4, characterized in that, Each piece of first metadata is encoded to generate a first metadata vector, including: Each first metadata is converted into text form, and each text-form first metadata is segmented by a lexical analyzer to generate multiple fifth lexical units corresponding to each first metadata. The multiple fifth lexical units corresponding to each first metadata are encoded by the encoding mechanism corresponding to each first metadata to generate a third vector corresponding to each first metadata. The encoding mechanisms corresponding to the first metadata containing different metadata information are different. Multiple third vectors are concatenated to generate metadata vectors corresponding to the multiple first metadata.

7. The method for generating semantic information according to claim 1, characterized in that, Based on the encoded target vector, the attention weights corresponding to the attention mechanism in the edge device are determined, and the plurality of first metadata and the first natural language instruction are fused according to the attention weights to generate target semantic information corresponding to the first natural language instruction, including: The first metadata encoding vector is linearly transformed based on the key vector matrix and value vector matrix in the edge device to generate a key vector and a value vector corresponding to each first metadata. The first instruction encoding vector is linearly transformed based on the query vector matrix in the edge device to generate a query vector. The key vector is used to indicate the semantic tag corresponding to each first metadata, and the value vector is used to indicate the metadata information contained in each first metadata. The query vector and each key vector are respectively calculated using the attention mechanism to determine the similarity score between the query vector and each key vector, and the attention weight is determined based on multiple similarity scores. Multiple value vectors are weighted and summed according to the attention weights to generate a context vector, and the target semantic information is generated according to the context vector, wherein the context vector is used to indicate the degree of association between the first natural language instruction and each first metadata.

8. The method for generating semantic information according to claim 7, characterized in that, After fusing the plurality of first metadata and the first natural language instruction according to the attention weights to generate target semantic information corresponding to the first natural language instruction, the method further includes: Multiple key vectors and multiple value vectors are stored in the target cache space of the edge device; Upon receiving a second natural language instruction corresponding to the first scenario, the second natural language instruction is encoded to generate a second instruction encoding vector; The attention weights are adjusted based on the second instruction encoding vector and the first metadata encoding vector, and the multiple first metadata and the second natural language instruction are fused based on the adjusted attention weights to generate semantic information corresponding to the second natural language instruction.

9. The method for generating semantic information according to claim 1, characterized in that, After fusing the plurality of first metadata and the first natural language instruction according to the attention weights to generate target semantic information corresponding to the first natural language instruction, the method further includes: If it is detected that the first scene has been changed to the second scene, or that the second metadata has been added to the first scene, the first parameter of the sub-network in the edge device is obtained, and the first parameter is adjusted based on the incremental adaptation network to generate the second parameter of the sub-network. The first parameter is used to adapt to the first metadata, and the second parameter is used to adapt to the third metadata or the second metadata. The third metadata is other metadata in the second scene besides the first metadata. Semantic recognition of a third natural language instruction is performed on an edge device that includes a sub-network adjusted to the second parameter, wherein the third natural language instruction is a natural language instruction corresponding to the second scenario or the first scenario after adding the second metadata.

10. A semantic information generation device, characterized in that, include: The encoding module is used to encode multiple first metadata in the first scenario and the first natural language instruction corresponding to the first scenario sent by the target object, wherein the first metadata includes at least one of the following: a device list in the first scenario, the capability attributes of each device in the device list, and the scenario tag of the first scenario; The determination module is used to determine the attention weights corresponding to the attention mechanism in the edge device based on the encoded target vector, and to perform fusion processing on the plurality of first metadata and the first natural language instruction based on the attention weights to generate target semantic information corresponding to the first natural language instruction. The target vector includes: the first metadata encoding vector corresponding to the plurality of first metadata and the first instruction encoding vector corresponding to the first natural language instruction.

11. The semantic information generation apparatus according to claim 10, characterized in that, The device further includes: an incremental adaptation module, used for: If it is detected that the first scene has been changed to the second scene, or that the second metadata has been added to the first scene, the first parameter of the sub-network in the edge device is obtained, and the first parameter is adjusted based on the incremental adaptation network to generate the second parameter of the sub-network. The first parameter is used to adapt to the first metadata, and the second parameter is used to adapt to the third metadata or the second metadata. The third metadata is other metadata in the second scene besides the first metadata.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method according to any one of claims 1 to 9.

13. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 9 through the computer program.

14. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor as described in any one of claims 1 to 9.

Images