Multi-modal feature signature generation method and device for intelligent agent and medium

Abstract

The invention provides a multi-modal feature signature generation method and device for an intelligent agent and a medium, and the method comprises the steps: S1, enabling the intelligent agent to collect multi-modal perception data at a t moment, and carrying out the feature extraction of the multi-modal perception data, and obtaining a multi-modal feature; s2, the multi-modal features are mapped to a shared vector space, multi-modal feature vectors are obtained, and the dimensions of all the modal feature vectors are the same; s3, fusing the multi-modal feature vectors to obtain a fusion state vector; s4, quantizing the fusion state vector to obtain a quantized fusion state vector; and S5, calculating the meta-information of the multi-modal sensing data and the quantized fusion state vector through a hash function, and generating a multi-modal feature signature at the t moment. According to the technical scheme, the feature signature can be generated for the state of the intelligent agent at each moment, rapid searching and historical state tracking are supported, and transmission of repeated or redundant data is avoided.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of artificial intelligence and multimodal data processing, specifically to a method for generating multimodal feature signatures for intelligent agents, which can be widely applied to industrial robots, intelligent vehicles, digital humans, and multi-agent collaborative scenarios. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent agents (such as industrial robots, autonomous vehicles, and service digital humans) are increasingly widely used in production and daily life, and the modal diversity and data scale of their perception systems are growing exponentially. Intelligent agents typically need to collect multimodal data such as images, point clouds, audio, pose, and equipment status through multiple sensors such as cameras, lidar, microphones, and IMUs (inertial measurement units) to achieve comprehensive perception of the environment and accurate decision-making.

[0003] However, traditional methods for describing the state of intelligent agents (such as directly using pose vectors, raw image features, discrete semantic labels, etc.) face the following bottlenecks in practical applications: First, there is data redundancy and high transmission pressure: the raw multimodal sensing data is mostly high-dimensional floating-point data, and the data volume is huge. If all the raw data is uploaded to the central server in real time or interacted between agents, it will occupy a lot of network bandwidth, causing transmission delays and even system congestion, which will seriously affect the real-time performance of multi-agent collaboration.

[0004] Secondly, cross-modal representations are not uniform: data from different modalities such as vision, touch, audio, and pose are heterogeneous, making it difficult for traditional methods to establish a unified state representation system. This makes it difficult to compare, fuse, and collaboratively analyze cross-modal data, and fails to accurately reflect the comprehensive state of the agent at the same moment.

[0005] Finally, the matching efficiency is low and the real-time performance is poor: cross-modal or cross-time dimension state comparison requires complex feature playback, modality transformation or deep model inference process, which has high computational complexity and takes a long time, making it difficult to meet the needs of intelligent agents in real-time decision-making, collision avoidance, dynamic collaboration and other scenarios. Therefore, there is an urgent need for a technical solution that can generate lightweight unique identifiers after multimodal data fusion, so that intelligent agents can achieve fast matching, redundancy detection and state tracking at any time through signature. Summary of the Invention

[0006] In view of this, this application provides a method, device and medium for generating multimodal feature signatures for intelligent agents, in order to solve the above-mentioned technical problems in the prior art.

[0007] According to one aspect of this application, a method for generating multimodal feature signatures for intelligent agents is provided, the method comprising the following steps: S1: the agent collects multi-modal perception data at time t, and performs feature extraction on the multi-modal perception data to obtain multi-modal features; S2: mapping the multi-modal features to a shared vector space to obtain multi-modal feature vectors, each of which has the same dimension; S3: fusing the multi-modal feature vectors to obtain a fusion state vector; S4: quantizing the fusion state vector to obtain a quantized fusion state vector; S5: calculating the meta-information of the multi-modal perception data and the quantized fusion state vector through a hash function to generate a multi-modal feature signature at time t.

[0008] According to the preferred embodiment of the present application, the method further comprises the following steps: S6: calculating the Hamming distance between the multi-modal feature signature at time t and another multi-modal feature signature, and if the Hamming distance is less than a preset threshold, determining the state of the agent at time t as a redundant state.

[0009] According to the preferred embodiment of the present application, the multi-modal perception data includes at least two of image modal data, point cloud modal data, pose modal data, audio modal data, and state modal data.

[0010] According to the preferred embodiment of the present application, in step S1, the feature extraction on the multi-modal perception data is implemented using a dedicated encoder, wherein the feature extraction on the image modal data uses a convolutional neural network or a visual Transformer encoder, the feature extraction on the point cloud modal data uses a PointNet or PointNet++ encoder, the feature extraction on the audio modal data uses a convolutional and recurrent neural network encoder, and the feature extraction on the pose modal and state modal data uses an MLP or Transformer encoder.

[0011] According to the preferred embodiment of the present application, step S2 is implemented using a multi-layer perception neural network model, which is trained through contrastive learning so that the distance between multi-modal vectors of the same state is smaller than the distance between multi-modal vectors of different states.

[0012] According to the preferred embodiment of the present application, in step S3, the multi-modal feature vectors are fused with weights, and the weights of the modal feature vectors are determined by the task type and the data quality.

[0013] According to the preferred embodiment of the present application, in step S4, the fusion state vector is quantized by symbol binarization to obtain a quantized fusion state vector.

[0014] According to the preferred embodiment of the present application, in step S5, the hash function adopts a local sensitive hash algorithm, and the meta information of the multi-modal perception data includes one or more of the following information: agent ID, timestamp, task label.

[0015] According to another aspect of the present application, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; the one or more processors are configured to implement the above-described method via the executable instructions.

[0016] According to still another aspect of the present application, a computer-readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, causes the processor to perform the above-described method.

[0017] The technical solutions of the embodiments of the present application have the following beneficial effects.

[0018] Strong cross-modal uniformity: through the cross-modal alignment mapping step, different types and heterogeneous multi-modal features are mapped to a unified shared vector space, solving the problem of non-uniform cross-modal representation in traditional methods and realizing unified representation of the comprehensive state of the agent.

[0019] High efficiency: the generated multi-modal feature signature is, for example, 64-bit or 128-bit binary data, which is lightweight and easy to store and transmit, greatly reducing storage costs and network bandwidth occupancy compared to high-dimensional multi-modal data in the prior art; at the same time, signature comparison based on Hamming distance can be completed in microseconds, meeting the real-time decision-making and collaboration needs of the agent.

[0020] Excellent scalability: supports multi-agent parallel signature generation, and each agent can independently complete data acquisition, feature encoding, signature generation, etc., and only needs to upload the final signature to the central server, without relying on the computing resources of the central node, and can adapt to the expansion needs of large-scale multi-agent collaboration scenarios.

[0021] Strong robustness: dynamic weights and attention mechanisms are used in the fusion stage to adaptively adjust the contribution of each modality, so that the generated state feature vector has strong anti-interference ability to noise, local disturbance and low-quality data, thereby ensuring the stability and reliability of the dynamic feature signature.

[0022] Good traceability: the signature index table records the mapping relationship between the feature signature and the metadata, enabling fast retrieval and trajectory reconstruction of the agent's full-cycle state, and providing strong support for behavior consistency verification, fault tracing, task review, etc. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings are used to provide further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical scheme of the present application, but do not constitute a limitation on the technical scheme of the present application.

[0024] Figure 1 A flow chart of a method of an exemplary embodiment of the present application is shown; Figure 2 A structural block diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] Various exemplary embodiments of the present application will be described in detail below with reference to the accompanying drawings. The description of the exemplary embodiments is merely illustrative and does not serve as any limitation on the present application and its application or use. The present application can be implemented in many different forms, not limited to the embodiments described herein. These embodiments are provided to make the present application thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0026] Unless specifically defined, an element can be one or more unless the number of elements is specifically limited. The term "plurality" means two or more, the term "based on" should be interpreted as "based at least in part on", the terms "and / or" and "at least one of" cover any one and all possible combinations of the listed items. In addition, "first", "second", and similar expressions are merely for descriptive purposes, not to indicate or imply relative importance or imply the number of indicated technical features.

[0027] Figure 1 A flow chart of a multi-modal feature signature generation method for an agent is shown. The agent may, for example, be an industrial robot, an intelligent vehicle, or a digital human, etc. As shown, the method comprises: Figure 1 S1: The agent collects multi-modal perception data at time t, and performs feature extraction on the multi-modal perception data to obtain multi-modal features; S2: Map the multi-modal features to a shared vector space to obtain multi-modal feature vectors, each of which has the same dimension; S3: Fuse the multi-modal feature vectors to obtain a fusion state vector; S4: Quantize the fusion state vector to obtain a quantized fusion state vector; S5: Calculate the meta-information of the multi-modal perception data and the quantized fusion state vector through a hash function to generate a multi-modal feature signature at time t.

[0028] According to a preferred embodiment of the present application, the method can further comprise the following steps:​ S6: Calculate the Hamming distance between the multi-modal feature signature at time t and the multi-modal feature signature at another time, and if the Hamming distance is less than a preset threshold, determine the state of the agent at time t as a redundant state.

[0029] In step S1, the agent collects multi-modal perception data at time t, and extracts features from the multi-modal perception data to obtain multi-modal features. The agent collects multi-modal perception data at time t through its mounted multi-sensor array .

[0030] wherein, is image modal data, obtained by an RGB camera; is point cloud modal data, obtained by a laser radar or binocular vision sensor, point cloud (Point Cloud) is a collection of discrete points in three-dimensional space, each point usually contains spatial coordinates and optional attributes, wherein the optional attributes include color and reflectivity intensity and other information; is pose modal data, obtained by a position and attitude sensor, for example, including the position coordinates, attitude angle, motion speed and acceleration of the agent, and other information; is audio modal data, including environmental sound, interactive voice, etc., obtained by a microphone; is state modal data, including information of the internal state of the agent, used to describe the behavior or execution state of the agent at a certain time, for example, including device running state, task execution progress, energy remaining amount and other structured data of the agent.

[0031] For each modal data, a dedicated encoder is used to extract a modal feature vector , , denotes data of a modal, RGB denotes image modal, Depth denotes point cloud modal, Pose denotes pose modal, Audio denotes audio modal, and State denotes state modal.

[0032] The specific types of the dedicated encoder for each modal data are as follows: Image modal data: CNN (Convolutional Neural Network) or visual Transformer is used as the encoder, and local texture features and high-level semantic features of the image are extracted through multiple layers of convolution and attention mechanism to obtain an image feature vector.

[0033] Point cloud modal data: PointNet / PointNet++ is used to directly process irregular point sets to obtain a point cloud feature vector.

[0034] Audio modality data: Convolutional and recurrent neural networks (CNN+RNN) are used as encoders. First, CNN is used to extract local time-frequency features of audio, and then RNN is used to capture the time sequence dependence of audio, to obtain audio feature vectors.

[0035] Pose and state modality data: MLP (Multi-Layer Perceptron) or Transformer is used as an encoder to perform nonlinear mapping on pose data and state data, mapping scalars / vectors to a feature space to obtain pose feature vectors and state feature vectors.

[0036] In step S2: Map multi-modal features to a shared vector space to obtain multi-modal feature vectors, each with the same dimension.

[0037] Each modality feature vector , Align represents the mapping function. The mapping of each modality feature can be implemented using a multi-layer perception neural network (MLP) model. The multi-layer perception neural network model is trained through contrastive learning, so that the distance between multi-modal vectors of the same state is close, while the distance between multi-modal vectors of different states is far. For contrastive learning training, optional pairs include: visual modality vs. point cloud modality, visual modality vs. pose modality, audio modality vs. state modality.

[0038] In step S3, the multi-modal feature vectors are fused to obtain a fused state vector.

[0039] The fusion of multi-modal feature vectors can be in the form of weighted or attention mechanism fusion to form a fused (unified) state vector , is the modality weight.

[0040] The modality weight is determined by the task type and data quality. Different task scenarios have different dependencies on modalities. For example, in the driving scenario, in the vehicle driving task: the image modality and the point cloud modality have higher weights to identify roads and obstacles. For vehicle reversing or low-speed precision operation tasks, the pose modality and the state modality have higher weights to determine the vehicle position, attitude, and driving state. For emergency alarm tasks, the audio modality has a higher weight to accurately identify alarm signals.

[0041] In addition, the quality of the modality data also affects the weight. When the quality of the modality data is poor, the weight of the corresponding modality should be reduced to avoid affecting the judgment of the agent state due to data distortion. For example, when the image is blocked or overexposed, the weight of the image modality should be reduced. When the point cloud is sparse, the weight of the point cloud modality should be reduced.

[0042] S4: Quantize the fused state vector to obtain a quantized fused state vector.

[0043] quantized fusion state vector Quantize denotes a quantization function.

[0044] Due to the high dimension of the fusion state vector, the storage and transmission overhead is large. In addition, the fusion state vector includes floating-point features, which will cause high cost of similarity calculation (such as L2 distance, cosine), and increase the computational burden of fast matching, deduplication or indexing operation.

[0045] By quantizing the fusion state vector through, for example, sign binary, the floating-point features can be eliminated, the dimension of the fusion state vector can be reduced, and fast matching, deduplication or indexing operation can be facilitated.

[0046] In step S5, the meta information of the multi-modal perception data and the quantized fusion state vector are calculated by a hash function to generate a multi-modal feature signature at time t.

[0047] In combination with the meta information of the multi-modal perception data (agent ID, timestamp, task label, etc.), a unique signature is generated by a hash function where M represents the encoded meta information. The task label is the category or number of the task or subtask being performed by the agent at a certain time, which can be encoded in the form of an integer, a string or one-hot encoding, etc. In this step, the similarity of the input vector space is mapped to the binary space, so that the hash values of similar vectors are similar. The unique signature is the label of Metadata, and Metadata represents all multi-modal perception data collected at time t, including image modal data, point cloud modal data, pose modal data, audio modal data, and state modal data, etc. The hash function can use local sensitive hashing (LSH) or SimHash, and output a 64-bit or 128-bit binary signature.

[0048] Specifically, in this step, the input information is: quantized fusion state vector + encoded meta information (agent ID, timestamp, task label). The agent ID encoding is used to distinguish different agents, for example, an integer encoding can be used, and each agent is assigned a unique integer ID. The timestamp encoding records the collection time, which is used for state tracking and signature uniqueness. For example, absolute time encoding can be used, that is, the number of seconds or milliseconds from a fixed starting point at a certain time. The task label encoding is used to identify the task or subtask currently performed by the agent. For example, an integer, a string, or one-hot encoding can be used.

[0049] The output of this step is a unique signature (Unique Signature), which is a fixed-length binary vector and can be used for matching, deduplication, and historical tracking.

[0050] SimHash (Similarity Hash) is a method of Locality-Sensitive Hashing (LSH) that maps high-dimensional vectors into fixed-length binary signatures while preserving the property that similar vectors correspond to similar signatures. Taking SimHash as an example, assuming that a signature of length L is generated, the generation process of a unique signature is as follows: (1) Generate a random projection vector. L random vectors are generated, and each vector does an inner product with the input vector to reflect the projection in that direction.

[0051] (2) Inner product symbolization. For example, if the inner product is greater than or equal to 0, symbolize it as 1; if the inner product is less than 0, symbolize it as 0.

[0052] (3) Combine the binary signature. The symbolized inner product forms an L-bit binary vector. It can be used as a unique signature or a fast index.

[0053] In step S6, the Hamming distance between the multi-modal feature signature at time t and another multi-modal feature signature is calculated, and if the Hamming distance is less than a preset threshold, the state of the agent at time t is determined as a redundant state.

[0054] The Hamming distance between the multi-modal feature signature at time t and another multi-modal feature signature , Hamming represents a function for calculating the Hamming distance. When d < δ (δ is a preset threshold), it is determined as a redundant state, otherwise a new index is added. All signature indexes are indexed to a hash table to realize fast retrieval, conflict detection and time series reconstruction.

[0055] In addition to the above technical solutions, the present application also provides an electronic device, which includes one or more processors, a GPU, and a memory for storing executable instructions. Wherein, the one or more processors are configured to implement the above method via executable instructions. The present application also provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the above method. In the following part of the specification, illustrative examples of the foregoing electronic device and computer readable storage medium will be described in conjunction with Figure 2 .

[0056] Figure 2 An example configuration of an electronic device 300 that can be used to implement the methods described in the present application is shown. The technical solutions of the present application can also be implemented in whole or at least in part by the electronic device 300 or similar devices / systems. The electronic device 300 can be various different types of devices. Examples of the electronic device 300 include, but are not limited to, a desktop computer, a server computer, a notebook or netbook computer, a mobile device, a wearable device, an entertainment device, a television or other display device, an automobile computer, etc.

[0057] The electronic device 300 can include at least one processor 302, memory 304, communication interface(s) 309, display device 301, other input / output (I / O) devices 310, and one or more mass storage devices 303, which can communicate with each other by way of a system bus 311 or other appropriate communication link.

[0058] The processor 302 can be a single or multiple processing unit(s) comprising single or multiple computing unit(s) or core(s). The processor 302 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor 302 can be configured to fetch and execute computer-readable instructions stored in the memory 304, the mass storage device 303, or any other computer-readable medium, such as program code of an operating system 305 or application programs 306 or other programs 307, etc.

[0059] The memory 304 and the mass storage device 303 are examples of computer-readable storage media for storing instructions which are executed by the processor 302 to practice the various functions described above. For example, the memory 304 can generally include both volatile memory and nonvolatile memory. Additionally, the mass storage device 303 can generally include hard disk drives, solid state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks, storage arrays, network attached storage, storage area networks, etc. Both the memory 304 and the mass storage device 303 can be collectively referred to in this disclosure as memory or computer-readable storage media, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code that can be executed by the processor 302 as a particular machine configured to implement the operations and functions described in the examples of the present disclosure.

[0060] A number of program modules can be stored on the mass storage device 303. These programs include an operating system 305, one or more application programs 306, other programs 307, and program data 308, and they can be loaded into the memory 304 for execution. Examples of such application programs or program modules can include, for example, computer program logic (e.g., computer program code or instructions) for implementing the methods provided by the present disclosure (including any suitable steps of the methods) and / or additional embodiments described herein.

[0061] Although the various examples of the present disclosure are illustrated and described herein as being implemented on a computer system, it is understood that the present disclosure is not limited to computer systems. For example, the present disclosure can be implemented on a mobile device, a server, a computer system, a computer program product, or any other suitable device or system. Figure 2The modular operating system 305, application programs 306, other programs 307, and program data 308, or portions thereof, are shown stored in the memory 304 of the electronic device 300, but can be implemented using any form of computer-readable media accessible by the electronic device 300. Herein, the computer-readable media can be any available media or communication media that can be accessed by a computer. By way of example, and not limitation, computer-readable media can comprise computer-readable storage media and communication media. Computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information in a computer-readable format, and which can be accessed by the electronic device 300. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The foregoing examples of computer-readable storage media and communication media do not encompass transitory media.

[0062] For example, computer-readable storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information in a computer-readable format, and which can be accessed by the electronic device 300.

[0063] One or more communication interfaces 309 are used to exchange data with other devices, such as over a network, a direct connection, etc. This communication interface can be one or more of any type of network interface, wired or wireless (e.g., WLAN) interface, Wi-MAX interface, Ethernet interface, USB interface, cellular network interface, Bluetooth interface, NFC interface, etc. The communication interface 309 can facilitate communication over a variety of network and protocol types, including wired and wireless networks, the Internet, etc. The communication interface 309 can also provide communication with external storage devices (not shown), such as storage arrays, network attached storage, storage area networks, etc.

[0064] In some examples, a display device 301, such as a monitor, can be included for displaying information and images to a user. Other I / O devices 310 can be devices that receive user input and provide output to a user, and can include touch / gesture input devices, cameras, keyboards, remote controls, mice, audio input / output devices, etc.

[0065] The technical solutions described in the present application can be supported by these various configurations of the electronic device 300, and are not limited to the specific examples of the technical solutions described in the present application. The foregoing and the accompanying drawings of the present application are not restrictive, and it is obvious to those skilled in the art that the present application is not limited to the details of the above-mentioned example embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the scope of the present application claimed is defined by the claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are covered within the scope of the present application.

Claims

1. A method for multi-modal feature signature generation for an agent, the method comprising: The method includes the following steps: S1: The agent collects multimodal sensing data at time t and extracts features from the multimodal sensing data to obtain multimodal features; S2: Map the multimodal features to a shared vector space to obtain multimodal feature vectors, where each modal feature vector has the same dimension; S3: Fuse the multimodal feature vectors to obtain the fused state vector; S4: Quantize the fused state vector to obtain the quantized fused state vector; S5: Calculate the metadata of the multimodal sensing data and the quantized fusion state vector using a hash function to generate the multimodal feature signature at time t.

2. The method of claim 1, wherein, The method further includes the following steps: S6: Calculate the Hamming distance between the multimodal feature signature at time t and another multimodal feature signature. If the Hamming distance is less than a preset threshold, the state of the agent at time t is determined to be a redundant state.

3. The method of claim 1, wherein, The multimodal sensing data includes at least two of the following: image modal data, point cloud modal data, pose modal data, audio modal data, and state modal data.

4. The method of claim 3, wherein, In step S1, feature extraction of multimodal perception data is performed using a dedicated encoder. Specifically, feature extraction of image modal data is performed using a convolutional neural network or a visual Transformer encoder, feature extraction of point cloud modal data is performed using a PointNet or PointNet++ encoder, feature extraction of audio modal data is performed using a convolutional and recurrent neural network encoder, and feature extraction of pose and state modal data is performed using an MLP or a Transformer encoder.

5. The method of claim 1, wherein, Step S2 is implemented using a multilayer perceptron neural network model. The multilayer perceptron neural network model is trained through contrastive learning so that the distance between multimodal vectors in the same state is smaller than the distance between multimodal vectors in different states.

6. The method of claim 1, wherein, In step S3, the multimodal feature vectors are weighted and fused, where the weights of each modal feature vector are determined by the task type and data quality.

7. The method of claim 1, wherein, In step S4, the fused state vector is quantized by symbol binarization to obtain the quantized fused state vector.

8. The method of claim 1, wherein, In step S5, the hash function adopts the locality-sensitive hashing algorithm, and the metadata of the multimodal perception data includes one or more of the following: agent ID, timestamp, and task tag.

9. An electronic device, comprising: The electronic device includes: One or more processors; Memory, which is used to store executable instructions; The one or more processors are configured to implement the method of any one of claims 1 to 8 via the executable instructions.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the method according to any one of claims 1 to 8.

Images