Environmental graph model generation method and device based on perception enhancement, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, provides an environment graph model generation method and device based on perception enhancement, equipment and a medium, can be applied to financial and medical health care service scenes, and can classify multi-source environment data according to entities so as to provide a targeted and effective data basis. Constructing an initial environment graph model according to a classification result, so as to convert the unstructured environment data into a structured graph model; performing feature initialization processing on nodes and edges in the initial environment graph model, so that the features of the nodes and the edges can accurately reflect basic information and relationships of entities; a graph attention network is adopted, feature propagation is carried out according to initialized data to obtain a target environment graph model, node features can be fused with information of neighbor nodes, and therefore attributes and relations of entities in the environment can be reflected more comprehensively and accurately; the obtained environment graph model can effectively solve the problem that environment perception in a vision-language-action scene is incomplete and inaccurate.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for generating environmental graph models based on enhanced perception. Background Technology

[0002] Currently, with the rapid development of artificial intelligence, many businesses involve vision-language-action (VLA) scenarios.

[0003] For example, in financial services, each bank branch has visual data, voice data, and motion data of personnel or equipment collected by surveillance cameras; in healthcare services, each operating room also has visual data, voice data, and motion data of personnel or equipment collected by surveillance cameras.

[0004] In the perception process of these scenarios, traditional methods struggle to effectively handle complex entity relationships and spatial structure information in the environment. For example, in multi-object interaction scenarios or dynamically changing environments, traditional methods cannot accurately capture the relationships between objects and the dynamic evolution of the environmental state, resulting in incomplete and inaccurate perception results, which adversely affects the execution of subsequent tasks. Summary of the Invention

[0005] In view of the above, it is necessary to provide a method, apparatus, device and medium for generating environmental graph models based on perception enhancement, which aims to solve the problem of incomplete and inaccurate environmental perception in vision-language-action scenarios.

[0006] A method for generating environment graph models based on perception enhancement, the method comprising:

[0007] In response to the command to generate an environment graph model based on the target environment, multi-source environment data of the target environment is collected;

[0008] The multi-source environmental data is classified according to entities to obtain classification results;

[0009] Construct an initial environment graph model based on the classification results;

[0010] The nodes and edges in the initial environment graph model are subjected to feature initialization processing to obtain initialization data;

[0011] A graph attention network is used to propagate features based on the initial data to obtain a graph model of the target environment.

[0012] An environment graph model generation device based on perception enhancement, the environment graph model generation device based on perception enhancement includes:

[0013] The acquisition unit is used to acquire multi-source environmental data of the target environment in response to the environmental graph model generation instruction based on the target environment;

[0014] A classification unit is used to classify the multi-source environmental data according to entities to obtain classification results;

[0015] The construction unit is used to construct an initial environment graph model based on the classification results;

[0016] An initialization unit is used to perform feature initialization processing on the nodes and edges in the initial environment graph model to obtain initialization data.

[0017] The propagation unit is used to perform feature propagation based on the initialization data using a graph attention network to obtain a target environment graph model.

[0018] A computer device, the computer device comprising:

[0019] Memory, storing at least one instruction; and

[0020] The processor executes the instructions stored in the memory to implement the perception-enhanced environment graph model generation method.

[0021] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the perception-enhanced environment graph model generation method.

[0022] As can be seen from the above technical solutions, this invention can classify multi-source environmental data according to entities to provide a targeted and effective data foundation; construct an initial environmental graph model based on the classification results, thereby transforming unstructured environmental data into a structured graph model; perform feature initialization processing on the nodes and edges in the initial environmental graph model, so that the features of the nodes and edges can accurately reflect the basic information and relationships of the entities; use a graph attention network to propagate features based on the initial data to obtain the target environmental graph model, so that the node features can be integrated with the information of neighboring nodes, thereby more comprehensively and accurately reflecting the attributes and relationships of entities in the environment. The resulting environmental graph model can effectively solve the problem of incomplete and inaccurate environmental perception in visual-language-action scenarios. Attached Figure Description

[0023] Figure 1 This is a flowchart of a preferred embodiment of the environmental graph model generation method based on perception enhancement of the present invention.

[0024] Figure 2 This is a functional block diagram of a preferred embodiment of the environmental graph model generation device based on perception enhancement of the present invention.

[0025] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the perception-enhanced environment graph model generation method of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the perception-enhanced environment graph model generation method of the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0028] The perception-enhanced environment graph model generation method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0029] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0030] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0031] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0032] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0035] S10, in response to the instruction to generate an environment graph model based on the target environment, collect multi-source environment data of the target environment.

[0036] In this embodiment, the target environment can be an environment with a vision-language-action (VLA) scene.

[0037] For example, the target environment can be a bank branch in a financial scenario, or an operating room in a medical and health scenario.

[0038] In this embodiment, the environment graph model generation command can be customized and triggered according to actual needs.

[0039] In this embodiment, the collection of multi-source environmental data of the target environment includes:

[0040] A multi-source data acquisition network is formed by deploying various environmental sensors such as cameras, microphones, and lidar in the target environment.

[0041] The multi-source environmental data is collected using the multi-source data acquisition network.

[0042] The environmental sensors may also include, but are not limited to, temperature sensors, humidity sensors, and light sensors.

[0043] For example, a camera can capture visual images of the target environment at a frequency of 30 frames per second; a microphone can collect audio signals of the target environment at a sampling rate of 16 kHz; a lidar can scan at a frequency of 10 Hz to obtain spatial distance information of the target environment; and environmental sensors can collect physical parameters at different sampling intervals (such as temperature sensors and humidity sensors every 10 seconds) depending on their type.

[0044] Through the above embodiments, multi-source environmental data of the target environment can be collected based on the constructed multi-source data acquisition network, providing a comprehensive data foundation for the subsequent construction of the environmental graph model.

[0045] S11, the multi-source environmental data is classified according to entities to obtain the classification results.

[0046] In this embodiment, after collecting the multi-source environmental data, it is also necessary to classify the data in order to determine the graph nodes and edges.

[0047] Specifically, the classification of the multi-source environmental data according to entities to obtain classification results includes:

[0048] For the visual images in the multi-source environmental data, a pre-trained object detection model is used to detect objects in the visual images, obtaining each object in the visual images and the object type of each object; and / or

[0049] For the audio data in the multi-source environmental data, sound source localization is performed on the audio data to obtain the sound source type of the audio data; and / or

[0050] For the physical parameter data collected by various environmental sensors in the multi-source environmental data, identify the environmental attribute type of the physical parameter data;

[0051] The classification result is generated based on each object in the visual image and the object type of each object, and / or the sound source type of the audio data, and / or the environmental attribute type of the physical parameter data.

[0052] For example, the object type of each object can include coffee table, robot, etc.; the sound source type can include human voice, equipment operation sound, etc.; the environmental attribute type can include temperature, humidity, etc.

[0053] Through the above embodiments, a clear data foundation can be provided for subsequent graph structure construction through classification processing, ensuring the relevance and effectiveness of the data and laying a good data foundation for subsequent steps.

[0054] S12, Construct an initial environment graph model based on the classification results.

[0055] In this embodiment, constructing the initial environment graph model based on the classification results includes:

[0056] Each type in the classification results is mapped as each entity to each node in the graph structure;

[0057] Identify the relationships between each node;

[0058] Connect the nodes according to the relationships between them to obtain each edge in the graph structure;

[0059] The initial environment graph model is generated based on each node and each edge in the graph structure;

[0060] Specifically, for spatial relationships, the spatial coordinate distance between every two entities is calculated; when it is detected that the spatial coordinate distance between two entities is less than a preset distance, the two detected entities are determined to be adjacent, and an edge is added between the two detected entities according to the adjacent relationship; when it is detected that a first entity is within the spatial range of a second entity, the first entity and the second entity are determined to be in an inclusion relationship, and an edge is added between the first entity and the second entity according to the inclusion relationship.

[0061] Specifically, for semantic relationships, the semantic similarity between any two entity names is calculated; when two entity names are detected to have a semantic similarity greater than a preset similarity, the two detected entity names are identified as having an association relationship, and an edge is added between the entities corresponding to the two detected entity names according to the association relationship.

[0062] Furthermore, each node can be configured with a unique identifier to accurately distinguish between different nodes.

[0063] The preset distance and the preset similarity can be optimal values ​​selected based on the experiment.

[0064] The node features of each node may include, but are not limited to: the appearance features of the entity (such as the color and shape of an object in a visual image), spatial coordinates (which can be obtained through LiDAR or image coordinate transformation), and attribute information (such as the material and state of the object).

[0065] The edge features of each edge may include, but are not limited to: relation type (e.g., adjacent relation is [1,0,0], containment relation is [0,1,0], association relation is [0,0,1]), relation strength (e.g., spatial relation strength is represented by the reciprocal of spatial distance, and semantic relation strength is represented by semantic similarity), etc.

[0066] Through the above embodiments, unstructured environmental data can be transformed into a structured graph model, clearly representing entities and their relationships, thereby providing a suitable data structure for graph neural network processing.

[0067] S13, Perform feature initialization processing on the nodes and edges in the initial environment graph model to obtain initialization data.

[0068] In this embodiment, the feature initialization processing of the nodes and edges in the initial environment graph model to obtain initialization data includes:

[0069] For each object node corresponding to each object type in the visual image, a visual feature vector is extracted from the visual image using the YOLOv7 model; the 3D spatial coordinates of the object corresponding to each object node are obtained; the visual feature vector and the 3D spatial coordinates are concatenated to obtain the initial feature vector of the object node; and / or

[0070] For each audio source node corresponding to each sound source type in the audio data, the Mel Frequency Cepstrum Coefficient (MFCC) feature of the audio data is extracted; the 3D coordinates obtained through the sound source localization are acquired; the MFCC feature and the 3D coordinates are concatenated to obtain the initial feature vector of the audio source node; and / or

[0071] For each environmental attribute type corresponding to the physical parameter node in the physical parameter data, the measured value of the physical parameter data is determined as the initial feature vector of the environmental physical parameter node; and / or

[0072] For the spatial edges in the initial environment graph model, the initial feature vector of the spatial edge is determined based on the spatial distance between the nodes corresponding to the spatial edge; and / or

[0073] For the semantic edges in the initial environment graph model, the relation descriptions corresponding to the semantic edges are encoded using a pre-trained semantic model to obtain the initial feature vectors of the semantic edges;

[0074] The initialization data is obtained by integrating each initial feature vector.

[0075] For example, when the YOLOv7 model extracts a 2048-dimensional visual feature vector, and the 3D spatial coordinates correspond to coordinates (x, y, z), then the initial feature vector of the object node is a 2051-dimensional feature vector.

[0076] The above embodiments can provide initial feature inputs for graph neural network computation, enabling the features of nodes and edges to accurately reflect the basic information and relationships of entities. The accuracy of feature initialization can ensure the effectiveness of feature propagation in subsequent processing.

[0077] S14, using a Graph Attention Network (GAT), feature propagation is performed based on the initialization data to obtain the target environment graph model.

[0078] In this embodiment, the step of employing a graph attention network to perform feature propagation based on the initialization data to obtain the target environment graph model includes:

[0079] For any node in each layer of the graph attention network, calculate the attention coefficient between the arbitrary node and its neighboring nodes; normalize the attention coefficient using a normalized exponential function to obtain normalized attention weights; and aggregate the features of the neighboring nodes using the normalized attention weights to update the features of the arbitrary node.

[0080] For any edge in each layer of the graph attention network, update the features of the edge according to the attention relationships of the connected nodes of the edge;

[0081] The target environment graph model is generated based on the updated features of each node and each edge.

[0082] The target environment graph model can effectively model the complex spatial and semantic relationships between entities in the environment, enabling a more accurate understanding of the environmental structure in VLA scenarios and improving the ability to perceive complex scenes. Simultaneously, the graph structure allows for unified representation and processing of multi-source heterogeneous data, reducing data processing complexity, improving information processing efficiency, and enhancing the model's comprehensive processing capabilities for different types of environmental information.

[0083] In the above embodiments, the attention mechanism enables the effective propagation and updating of node features, allowing node features to integrate information from neighboring nodes and more comprehensively reflect the attributes and relationships of entities in the environment. The updated model can significantly improve the accuracy of entity relationship recognition.

[0084] In this embodiment, after obtaining the target environment graph model, the method further includes:

[0085] Real-time monitoring of changes in the target environment based on the data dimensions of the multi-source environmental data;

[0086] Update the target environment map model based on the changed data;

[0087] Specifically, for the visual image, the inter-frame difference method is used to calculate the pixel difference between two adjacent frames; when the area corresponding to the pixel difference is greater than a preset area, it is determined that entity movement has been detected.

[0088] Specifically, for the lidar data in the physical parameter data, the point cloud change at adjacent time points is calculated; when the point cloud change is greater than a preset change, it is determined that an entity change has been detected; wherein, the entity change includes the addition of an entity, the disappearance of an entity, and the movement of an entity's position.

[0089] The preset area and the preset change amount can be selected as optimal values ​​based on experiments.

[0090] Specifically, when updating the target environment graph model based on the changed data, when a new entity is detected, a new node is created for it. Initial features are assigned to the new node according to the node feature initialization method. The spatial and semantic relationships between the new node and existing nodes are calculated, and corresponding edges are added and their features are initialized. For a disappeared entity, the corresponding node and all edges connected to that node are deleted from the target environment graph model. For changes in entity position, the spatial coordinate features of the node are updated, the spatial relationships between the node and other nodes are recalculated, and the features of the corresponding edges (such as relationship strength) are updated. For changes in entity attributes, the attribute features of the node are directly updated (e.g., if the object's color changes, the node's color features are updated). After the target environment graph model is updated, the graph neural network is re-run for feature propagation and updating to ensure that the model reflects the latest environmental state.

[0091] Through the above embodiments, various changes in the environment can be monitored in real time, and the addition, disappearance, location change and attribute change of entities can be detected in a timely manner. This enables the environmental graph model to adapt to the dynamic changes of the environment in real time, ensuring the accuracy and timeliness of the environmental graph model's perception of the environment, and making it more suitable for dynamic and ever-changing application scenarios.

[0092] In this embodiment, after obtaining the target environment graph model, the method further includes:

[0093] In response to a task execution instruction based on the target environment graph model, the task execution instruction is semantically understood to obtain instruction keywords;

[0094] The specified instruction keywords are used to match the target environment graph model to obtain the matching results.

[0095] Task decision data is generated based on the matching results.

[0096] For example, when the task execution instruction corresponds to a service optimization task based on environment awareness in a financial scenario, if the instruction keyword is "ATM location", then "ATM location" can be used to match in the target environment graph model to generate corresponding decisions, such as guiding customers to idle self-service equipment to optimize the branch service process.

[0097] For example, when the task execution instruction corresponds to the patient monitoring and abnormal response task in the ward based on environmental perception in a medical and health scenario, if the instruction keyword is "infusion finished", then "infusion finished" can be used to match in the target environment graph model to generate action decisions, such as notifying the nurse to remove the needle or adjusting the infusion pump status, so as to improve the response speed.

[0098] As can be seen from the above technical solutions, this invention can classify multi-source environmental data according to entities to provide a targeted and effective data foundation; construct an initial environmental graph model based on the classification results, thereby transforming unstructured environmental data into a structured graph model; perform feature initialization processing on the nodes and edges in the initial environmental graph model, so that the features of the nodes and edges can accurately reflect the basic information and relationships of the entities; use a graph attention network to propagate features based on the initial data to obtain the target environmental graph model, so that the node features can be integrated with the information of neighboring nodes, thereby more comprehensively and accurately reflecting the attributes and relationships of entities in the environment. The resulting environmental graph model can effectively solve the problem of incomplete and inaccurate environmental perception in visual-language-action scenarios.

[0099] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the perception-enhanced environment graph model generation device of the present invention. The perception-enhanced environment graph model generation device 11 includes a data acquisition unit 110, a classification unit 111, a construction unit 112, an initialization unit 113, and a propagation unit 114. The module / unit referred to in this invention is a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0100] The acquisition unit 110 is used to acquire multi-source environmental data of the target environment in response to an environmental graph model generation instruction based on the target environment.

[0101] In this embodiment, the target environment can be an environment with a vision-language-action (VLA) scene.

[0102] For example, the target environment can be a bank branch in a financial scenario, or an operating room in a medical and health scenario.

[0103] In this embodiment, the environment graph model generation command can be customized and triggered according to actual needs.

[0104] In this embodiment, the acquisition unit 110 acquires multi-source environmental data of the target environment, including:

[0105] A multi-source data acquisition network is formed by deploying various environmental sensors such as cameras, microphones, and lidar in the target environment.

[0106] The multi-source environmental data is collected using the multi-source data acquisition network.

[0107] The environmental sensors may also include, but are not limited to, temperature sensors, humidity sensors, and light sensors.

[0108] For example, a camera can capture visual images of the target environment at a frequency of 30 frames per second; a microphone can collect audio signals of the target environment at a sampling rate of 16 kHz; a lidar can scan at a frequency of 10 Hz to obtain spatial distance information of the target environment; and environmental sensors can collect physical parameters at different sampling intervals (such as temperature sensors and humidity sensors every 10 seconds) depending on their type.

[0109] Through the above embodiments, multi-source environmental data of the target environment can be collected based on the constructed multi-source data acquisition network, providing a comprehensive data foundation for the subsequent construction of the environmental graph model.

[0110] The classification unit 111 is used to classify the multi-source environmental data according to entities to obtain classification results.

[0111] In this embodiment, after collecting the multi-source environmental data, it is also necessary to classify the data in order to determine the graph nodes and edges.

[0112] Specifically, the classification unit 111 classifies the multi-source environmental data according to entities, and the classification results include:

[0113] For the visual images in the multi-source environmental data, a pre-trained object detection model is used to detect objects in the visual images, obtaining each object in the visual images and the object type of each object; and / or

[0114] For the audio data in the multi-source environmental data, sound source localization is performed on the audio data to obtain the sound source type of the audio data; and / or

[0115] For the physical parameter data collected by various environmental sensors in the multi-source environmental data, identify the environmental attribute type of the physical parameter data;

[0116] The classification result is generated based on each object in the visual image and the object type of each object, and / or the sound source type of the audio data, and / or the environmental attribute type of the physical parameter data.

[0117] For example, the object type of each object can include coffee table, robot, etc.; the sound source type can include human voice, equipment operation sound, etc.; the environmental attribute type can include temperature, humidity, etc.

[0118] Through the above embodiments, a clear data foundation can be provided for subsequent graph structure construction through classification processing, ensuring the relevance and effectiveness of the data and laying a good data foundation for subsequent steps.

[0119] The construction unit 112 is used to construct an initial environment graph model based on the classification results.

[0120] In this embodiment, the construction unit 112 constructs an initial environment graph model based on the classification results, including:

[0121] Each type in the classification results is mapped as each entity to each node in the graph structure;

[0122] Identify the relationships between each node;

[0123] Connect the nodes according to the relationships between them to obtain each edge in the graph structure;

[0124] The initial environment graph model is generated based on each node and each edge in the graph structure;

[0125] Specifically, for spatial relationships, the spatial coordinate distance between every two entities is calculated; when it is detected that the spatial coordinate distance between two entities is less than a preset distance, the two detected entities are determined to be adjacent, and an edge is added between the two detected entities according to the adjacent relationship; when it is detected that a first entity is within the spatial range of a second entity, the first entity and the second entity are determined to be in an inclusion relationship, and an edge is added between the first entity and the second entity according to the inclusion relationship.

[0126] Specifically, for semantic relationships, the semantic similarity between any two entity names is calculated; when two entity names are detected to have a semantic similarity greater than a preset similarity, the two detected entity names are identified as having an association relationship, and an edge is added between the entities corresponding to the two detected entity names according to the association relationship.

[0127] Furthermore, each node can be configured with a unique identifier to accurately distinguish between different nodes.

[0128] The preset distance and the preset similarity can be optimal values ​​selected based on the experiment.

[0129] The node features of each node may include, but are not limited to: the appearance features of the entity (such as the color and shape of an object in a visual image), spatial coordinates (which can be obtained through LiDAR or image coordinate transformation), and attribute information (such as the material and state of the object).

[0130] The edge features of each edge may include, but are not limited to: relation type (e.g., adjacent relation is [1,0,0], containment relation is [0,1,0], association relation is [0,0,1]), relation strength (e.g., spatial relation strength is represented by the reciprocal of spatial distance, and semantic relation strength is represented by semantic similarity), etc.

[0131] Through the above embodiments, unstructured environmental data can be transformed into a structured graph model, clearly representing entities and their relationships, thereby providing a suitable data structure for graph neural network processing.

[0132] The initialization unit 113 is used to perform feature initialization processing on the nodes and edges in the initial environment graph model to obtain initialization data.

[0133] In this embodiment, the initialization unit 113 performs feature initialization processing on the nodes and edges in the initial environment graph model to obtain initialization data including:

[0134] For each object node corresponding to each object type in the visual image, a visual feature vector is extracted from the visual image using the YOLOv7 model; the 3D spatial coordinates of the object corresponding to each object node are obtained; the visual feature vector and the 3D spatial coordinates are concatenated to obtain the initial feature vector of the object node; and / or

[0135] For each audio source node corresponding to each sound source type in the audio data, the Mel Frequency Cepstrum Coefficient (MFCC) feature of the audio data is extracted; the 3D coordinates obtained through the sound source localization are acquired; the MFCC feature and the 3D coordinates are concatenated to obtain the initial feature vector of the audio source node; and / or

[0136] For each environmental attribute type corresponding to the physical parameter node in the physical parameter data, the measured value of the physical parameter data is determined as the initial feature vector of the environmental physical parameter node; and / or

[0137] For the spatial edges in the initial environment graph model, the initial feature vector of the spatial edge is determined based on the spatial distance between the nodes corresponding to the spatial edge; and / or

[0138] For the semantic edges in the initial environment graph model, the relation descriptions corresponding to the semantic edges are encoded using a pre-trained semantic model to obtain the initial feature vectors of the semantic edges;

[0139] The initialization data is obtained by integrating each initial feature vector.

[0140] For example, when the YOLOv7 model extracts a 2048-dimensional visual feature vector, and the 3D spatial coordinates correspond to coordinates (x, y, z), then the initial feature vector of the object node is a 2051-dimensional feature vector.

[0141] The above embodiments can provide initial feature inputs for graph neural network computation, enabling the features of nodes and edges to accurately reflect the basic information and relationships of entities. The accuracy of feature initialization can ensure the effectiveness of feature propagation in subsequent processing.

[0142] The propagation unit 114 is used to perform feature propagation based on the initialization data using a graph attention network (GAT) to obtain a target environment graph model.

[0143] In this embodiment, the propagation unit 114 employs a graph attention network to perform feature propagation based on the initialization data, obtaining the target environment graph model including:

[0144] For any node in each layer of the graph attention network, calculate the attention coefficient between the arbitrary node and its neighboring nodes; normalize the attention coefficient using a normalized exponential function to obtain normalized attention weights; and aggregate the features of the neighboring nodes using the normalized attention weights to update the features of the arbitrary node.

[0145] For any edge in each layer of the graph attention network, update the features of the edge according to the attention relationships of the connected nodes of the edge;

[0146] The target environment graph model is generated based on the updated features of each node and each edge.

[0147] The target environment graph model can effectively model the complex spatial and semantic relationships between entities in the environment, enabling a more accurate understanding of the environmental structure in VLA scenarios and improving the ability to perceive complex scenes. Simultaneously, the graph structure allows for unified representation and processing of multi-source heterogeneous data, reducing data processing complexity, improving information processing efficiency, and enhancing the model's comprehensive processing capabilities for different types of environmental information.

[0148] In the above embodiments, the attention mechanism enables the effective propagation and updating of node features, allowing node features to integrate information from neighboring nodes and more comprehensively reflect the attributes and relationships of entities in the environment. The updated model can significantly improve the accuracy of entity relationship recognition.

[0149] In this embodiment, after obtaining the target environment map model, the changes in the target environment are monitored in real time according to the data dimensions of the multi-source environment data;

[0150] Update the target environment map model based on the changed data;

[0151] Specifically, for the visual image, the inter-frame difference method is used to calculate the pixel difference between two adjacent frames; when the area corresponding to the pixel difference is greater than a preset area, it is determined that entity movement has been detected.

[0152] Specifically, for the lidar data in the physical parameter data, the point cloud change at adjacent time points is calculated; when the point cloud change is greater than a preset change, it is determined that an entity change has been detected; wherein, the entity change includes the addition of an entity, the disappearance of an entity, and the movement of an entity's position.

[0153] The preset area and the preset change amount can be selected as optimal values ​​based on experiments.

[0154] Specifically, when updating the target environment graph model based on the changed data, when a new entity is detected, a new node is created for it. Initial features are assigned to the new node according to the node feature initialization method. The spatial and semantic relationships between the new node and existing nodes are calculated, and corresponding edges are added and their features are initialized. For a disappeared entity, the corresponding node and all edges connected to that node are deleted from the target environment graph model. For changes in entity position, the spatial coordinate features of the node are updated, the spatial relationships between the node and other nodes are recalculated, and the features of the corresponding edges (such as relationship strength) are updated. For changes in entity attributes, the attribute features of the node are directly updated (e.g., if the object's color changes, the node's color features are updated). After the target environment graph model is updated, the graph neural network is re-run for feature propagation and updating to ensure that the model reflects the latest environmental state.

[0155] Through the above embodiments, various changes in the environment can be monitored in real time, and the addition, disappearance, location change and attribute change of entities can be detected in a timely manner. This enables the environmental graph model to adapt to the dynamic changes of the environment in real time, ensuring the accuracy and timeliness of the environmental graph model's perception of the environment, and making it more suitable for dynamic and ever-changing application scenarios.

[0156] In this embodiment, after obtaining the target environment graph model, in response to the task execution instruction based on the target environment graph model, the task execution instruction is semantically understood to obtain instruction keywords;

[0157] The specified instruction keywords are used to match the target environment graph model to obtain the matching results.

[0158] Task decision data is generated based on the matching results.

[0159] For example, when the task execution instruction corresponds to a service optimization task based on environment awareness in a financial scenario, if the instruction keyword is "ATM location", then "ATM location" can be used to match in the target environment graph model to generate corresponding decisions, such as guiding customers to idle self-service equipment to optimize the branch service process.

[0160] For example, when the task execution instruction corresponds to the patient monitoring and abnormal response task in the ward based on environmental perception in a medical and health scenario, if the instruction keyword is "infusion finished", then "infusion finished" can be used to match in the target environment graph model to generate action decisions, such as notifying the nurse to remove the needle or adjusting the infusion pump status, so as to improve the response speed.

[0161] As can be seen from the above technical solutions, this invention can classify multi-source environmental data according to entities to provide a targeted and effective data foundation; construct an initial environmental graph model based on the classification results, thereby transforming unstructured environmental data into a structured graph model; perform feature initialization processing on the nodes and edges in the initial environmental graph model, so that the features of the nodes and edges can accurately reflect the basic information and relationships of the entities; use a graph attention network to propagate features based on the initial data to obtain the target environmental graph model, so that the node features can be integrated with the information of neighboring nodes, thereby more comprehensively and accurately reflecting the attributes and relationships of entities in the environment. The resulting environmental graph model can effectively solve the problem of incomplete and inaccurate environmental perception in visual-language-action scenarios.

[0162] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device that implements the method for generating environment graph models based on perception enhancement according to the present invention.

[0163] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a perception-enhanced environment graph model generation program.

[0164] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0165] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0166] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as code for a perception-enhanced environment graph model generation program, but also to temporarily store data that has been output or will be output.

[0167] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing a perception-enhanced environment graph model generation program) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0168] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the various embodiments of the perception-enhanced environment graph model generation method described above, for example... Figure 1 The steps are shown.

[0169] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into a collection unit 110, a classification unit 111, a construction unit 112, an initialization unit 113, and a propagation unit 114.

[0170] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, a computer device, or a network device, etc.) or processor to execute portions of the perception-enhanced environment graph model generation method described in the various embodiments of this invention.

[0171] If the modules / units integrated in the computer device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware devices. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above.

[0172] The computer program includes computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory, etc.

[0173] Furthermore, the computer-readable storage medium may primarily include a stored program area and a stored data area, wherein the stored program area may store the operating system, an application program required for at least one function, etc.; and the stored data area may store data created based on the use of blockchain nodes, etc.

[0174] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0175] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, in... Figure 3 The bus is represented by only one straight line, but this does not mean that there is only one bus or one type of bus. The bus is configured to enable communication between the memory 12 and at least one processor 13, etc.

[0176] Although not shown, the computer device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 13 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The computer device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0177] Furthermore, the computer device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish a communication connection between the computer device 1 and other computer devices.

[0178] Optionally, the computer device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the computer device 1 and to display a visual user interface.

[0179] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0180] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0181] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a method for generating an environment graph model based on perception enhancement, and the processor 13 can execute the multiple instructions to achieve the following:

[0182] In response to the command to generate an environment graph model based on the target environment, multi-source environment data of the target environment is collected;

[0183] The multi-source environmental data is classified according to entities to obtain classification results;

[0184] Construct an initial environment graph model based on the classification results;

[0185] The nodes and edges in the initial environment graph model are subjected to feature initialization processing to obtain initialization data;

[0186] A graph attention network is used to propagate features based on the initial data to obtain a graph model of the target environment.

[0187] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0188] It should be noted that all data involved in this case was legally obtained. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0189] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0190] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0191] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0193] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0194] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0195] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in this invention can also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0196] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating environmental graph models based on perception enhancement, characterized in that, The method for generating environment graph models based on perception enhancement includes: In response to the command to generate an environment graph model based on the target environment, multi-source environment data of the target environment is collected; The multi-source environmental data is classified according to entities to obtain classification results; Construct an initial environment graph model based on the classification results; The nodes and edges in the initial environment graph model are subjected to feature initialization processing to obtain initialization data; A graph attention network is used to perform feature propagation based on the initialization data to obtain a graph model of the target environment.

2. The method for generating an environment graph model based on perception enhancement as described in claim 1, characterized in that, The classification of the multi-source environmental data according to entities to obtain the classification results includes: For the visual images in the multi-source environmental data, a pre-trained object detection model is used to detect objects in the visual images, obtaining each object in the visual images and the object type of each object; and / or For the audio data in the multi-source environmental data, sound source localization is performed on the audio data to obtain the sound source type of the audio data; and / or For the physical parameter data collected by various environmental sensors in the multi-source environmental data, identify the environmental attribute type of the physical parameter data; The classification result is generated based on each object in the visual image and the object type of each object, and / or the sound source type of the audio data, and / or the environmental attribute type of the physical parameter data.

3. The method for generating an environment graph model based on perception enhancement as described in claim 1, characterized in that, The step of constructing the initial environment graph model based on the classification results includes: Each type in the classification results is mapped as each entity to each node in the graph structure; Identify the relationships between each node; Connect the nodes according to the relationships between them to obtain each edge in the graph structure; The initial environment graph model is generated based on each node and each edge in the graph structure; Specifically, for spatial relationships, the spatial coordinate distance between every two entities is calculated; when it is detected that the spatial coordinate distance between two entities is less than a preset distance, the two detected entities are determined to be adjacent, and an edge is added between the two detected entities according to the adjacent relationship; when it is detected that a first entity is within the spatial range of a second entity, the first entity and the second entity are determined to be in an inclusion relationship, and an edge is added between the first entity and the second entity according to the inclusion relationship. Specifically, for semantic relationships, the semantic similarity between any two entity names is calculated; when two entity names are detected to have a semantic similarity greater than a preset similarity, the two detected entity names are identified as having an association relationship, and an edge is added between the entities corresponding to the two detected entity names according to the association relationship.

4. The method for generating an environment graph model based on perception enhancement as described in claim 2, characterized in that, The feature initialization process for nodes and edges in the initial environment graph model to obtain initialization data includes: For each object node corresponding to each object type in the visual image, a visual feature vector is extracted from the visual image using the YOLOv7 model; the 3D spatial coordinates of the object corresponding to each object node are obtained; the visual feature vector and the 3D spatial coordinates are concatenated to obtain the initial feature vector of the object node; and / or For each audio source node corresponding to each sound source type in the audio data, the Mel-frequency cepstral coefficient features of the audio data are extracted; the 3D coordinates obtained through the sound source localization are acquired; the Mel-frequency cepstral coefficient features and the 3D coordinates are concatenated to obtain the initial feature vector of the audio source node; and / or For each environmental attribute type corresponding to the physical parameter node in the physical parameter data, the measured value of the physical parameter data is determined as the initial feature vector of the environmental physical parameter node; and / or For the spatial edges in the initial environment graph model, the initial feature vector of the spatial edge is determined based on the spatial distance between the nodes corresponding to the spatial edge; and / or For the semantic edges in the initial environment graph model, the relation descriptions corresponding to the semantic edges are encoded using a pre-trained semantic model to obtain the initial feature vectors of the semantic edges; The initialization data is obtained by integrating each initial feature vector.

5. The method for generating an environment graph model based on perception enhancement as described in claim 1, characterized in that, The step of employing a graph attention network to perform feature propagation based on the initialization data to obtain the target environment graph model includes: For any node in each layer of the graph attention network, calculate the attention coefficient between the arbitrary node and its neighboring nodes; normalize the attention coefficient using a normalized exponential function to obtain normalized attention weights; and aggregate the features of the neighboring nodes using the normalized attention weights to update the features of the arbitrary node. For any edge in each layer of the graph attention network, update the features of the edge according to the attention relationships of the connected nodes of the edge; The target environment graph model is generated based on the updated features of each node and each edge.

6. The method for generating an environment graph model based on perception enhancement as described in claim 2, characterized in that, After obtaining the target environment map model, the method further includes: Real-time monitoring of changes in the target environment based on the data dimensions of the multi-source environmental data; Update the target environment map model based on the changed data; Specifically, for the visual image, the inter-frame difference method is used to calculate the pixel difference between two adjacent frames; when the area corresponding to the pixel difference is greater than a preset area, it is determined that entity movement has been detected. Specifically, for the lidar data in the physical parameter data, the point cloud change at adjacent time points is calculated; when the point cloud change is greater than a preset change, it is determined that an entity change has been detected; wherein, the entity change includes the addition of an entity, the disappearance of an entity, and the movement of an entity's position.

7. The method for generating an environment graph model based on perception enhancement as described in claim 1, characterized in that, After obtaining the target environment map model, the method further includes: In response to a task execution instruction based on the target environment graph model, the task execution instruction is semantically understood to obtain instruction keywords; The specified instruction keywords are used to match the target environment graph model to obtain the matching results. Task decision data is generated based on the matching results.

8. An environment graph model generation device based on perception enhancement, characterized in that, The perception-enhanced environment graph model generation device includes: The acquisition unit is used to acquire multi-source environmental data of the target environment in response to the environmental graph model generation instruction based on the target environment; A classification unit is used to classify the multi-source environmental data according to entities to obtain classification results; The construction unit is used to construct an initial environment graph model based on the classification results; An initialization unit is used to perform feature initialization processing on the nodes and edges in the initial environment graph model to obtain initialization data. The propagation unit is used to perform feature propagation based on the initialization data using a graph attention network to obtain a target environment graph model.

9. A computer device, characterized in that, The computer device includes: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the perception-enhanced environment graph model generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the perception-enhanced environment graph model generation method as described in any one of claims 1 to 7.

Images