A multi-task scene understanding method based on graph attention network

Abstract

The present invention provides a multi-task scene understanding method based on a graph attention network. The method includes the following steps: collecting data and annotating and recording to obtain a multimodal data set; obtaining the category information of all objects based on the multimodal data set; then finding the relative position information of each object and converting it into global position information, while adding the current time environmental information to generate a comprehensive data representation; designing preset rules to generate a scene model; using the loss function and the graph attention neural network to iteratively train the scene model, obtain a trained scene model and perform a variety of scene analysis tasks, and infer the overall situation of the scene through the scene analysis results. The present invention can improve the efficiency of multimodal data fusion, simplify the data alignment and synchronization processing process, fully integrate and analyze environmental information, enhance the global scene understanding ability, and perform a variety of scene understanding tasks at the same time, so as to achieve comprehensive perception and intelligent reasoning of complex scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of environment perception and scene understanding, and in particular to a multi-task scene understanding method based on a graph attention network. Background Art

[0002] In the current technological context, intelligent mobile devices (such as automated guided vehicles (AGVs)) are increasingly used in industrial and commercial environments. At present, AGVs are usually equipped with a variety of sensors for data collection to achieve inspection work with autonomous obstacle avoidance and navigation functions. Typical AGV systems include several types of sensors such as laser radar (LiDAR), cameras, and GPS sensors. Although these sensors can provide rich scene information, the existing technology mainly focuses on the processing and application of sensor data, and lacks the ability to conduct comprehensive analysis and global understanding of multi-sensor data. For example, AGV can recognize that there is an obstacle ahead, but it cannot evaluate the degree of congestion, danger level, or other special circumstances of the environment as a whole. In addition, existing technical solutions rarely involve the content of overall understanding of the perceived environment, and only stay at the level of machine perception and understanding of the environment. It is difficult for humans to grasp the scene perceived by the machine as a whole.

[0003] At present, scene understanding technology is rarely used in smart mobile devices, and is mainly limited to basic environmental perception and navigation tasks. Some technologies have begun to try to use the Transformer structure for scene understanding tasks, but there are still shortcomings such as high computational complexity, limited multimodal data fusion effect, insufficient adaptability to environmental changes, and limited global scene understanding capabilities. Therefore, the present invention proposes a multi-task scene understanding method based on a graph attention network to solve the above problems. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a multi-task scene understanding method based on a graph attention network, which can improve the efficiency of multimodal data fusion, simplify data alignment and synchronization processing, comprehensively integrate and analyze environmental information, enhance global scene understanding capabilities, and perform multiple scene understanding tasks at the same time, so that the present invention can achieve comprehensive perception and intelligent reasoning of complex scenes, and improve the application effect and decision-making ability of smart mobile devices in industrial and commercial environments.

[0005] To achieve the above object, the present invention provides the following solution: a multi-task scene understanding method based on graph attention network, comprising the following steps:

[0006] Set a unified collection interval, use an automated guided vehicle to synchronously collect data in the application scenario, annotate the collected data, and then record the collected data in time series to obtain a multimodal data set;

[0007] Based on the multimodal data set, object features are extracted using a multimodal data fusion method to implement object detection training in the application scene and obtain category information of all objects in the application scene;

[0008] Based on the category information of all detected objects, the relative position information of each object is found in time series, and the relative position information is converted into global position information, and the current time environment information is added to the global position to generate a comprehensive data representation of each object;

[0009] Design preset rules, perform scene modeling based on the preset rules and the comprehensive data representation of all objects in the scene, and generate a scene model;

[0010] Iteratively training the scene model using the loss function and the graph attention neural network to obtain a trained scene model; the trained scene model is used to establish the relationship between objects, the relationship between objects and the environment, scene comprehensive information, and the connection between different scenes;

[0011] Perform various scenario analysis tasks based on the trained scenario model, and infer the overall situation of the scenario through the scenario analysis results.

[0012] Optionally, the generation process of the multimodal dataset is:

[0013] The automatic guided vehicle is equipped with a laser radar, a camera, an environmental information sensor, a GPS sensor, and an IMU sensor, and data is synchronously collected at a set collection interval to obtain an original data set;

[0014] The image data in the original data set is annotated in 2D to obtain the 2D bounding box of the object, the point cloud data in the original data set is annotated in 3D to obtain the 3D bounding box of the object and the relative position of the object and the automatic guided vehicle, and then the GPS data, IMU data and environmental sensor data are recorded in time series to obtain the multimodal data set.

[0015] Optionally, the process of extracting object features using a multimodal data fusion method is:

[0016] The deep neural network is used to extract features from the collected image data and point cloud data respectively, and then the first generalized eigenvalue decomposition theory and principal component analysis technology are used to reduce the dimension of the extracted features to obtain image data features and point cloud data features;

[0017] The image data features and the point cloud data features are fused using a multimodal data fusion strategy to retain the information content of the original data; in the feature fusion process, different backbone networks are selected for different data, attention weights are calculated, and then the main feature vectors are updated respectively according to the attention weights.

[0018] Optionally, the data processing process represented by the comprehensive data is:

[0019] Based on the category information of all detected objects, each entity in the scene is detected using a detection time alignment scheme to generate scene detection data, and then one-hot encoding and global coordinate conversion are performed based on the scene detection data, and environmental information of the current time is added to generate a comprehensive data representation of each object; wherein the one-hot encoding determines the number of object categories and constructs a unique hot encoding vector for each object category to learn the characteristics of each object category; the global coordinate conversion combines GPS data and the posture information of the automatic guided vehicle, and applies a coordinate conversion method to convert the local 3D position of each object into a global coordinate system position to achieve the positioning of the target object.

[0020] Optionally, the process of scene modeling is:

[0021] Using the GNN network model, objects in the scene are set as nodes; the relationship between the nodes is set as an adjacency matrix;

[0022] The nodes, the adjacency matrix and the environmental feature information are input into the GNN network model for training in the form of node feature vectors, and the attention mechanism is used for dynamic adjustment during the training process to comprehensively process multimodal data to obtain the scene model.

[0023] Optionally, the graph attention neural network comprises:

[0024] Input layer, used to input node feature matrix and adjacency matrix;

[0025] A graph attention layer, which is used to extract node features and relationships and enhance the expression capability of the scene model through a multi-head attention mechanism;

[0026] Feature aggregation, which is used to aggregate features through the results of multiple attention heads and concatenate or average the outputs of multiple attention heads;

[0027] Different task headers are set for different tasks; the task headers are used to output label information and degree information.

[0028] Optionally, the process of using the loss function and graph attention neural network to train the model to generate scene description and scene understanding is:

[0029] Using the attention mechanism, the weights are dynamically adjusted in the graph attention layer according to the feature similarity between nodes;

[0030] Various information of objects in the scene, environmental sensor data, and real labels of different scenes are input into the scene model, and the graph attention neural network is used to enable the scene model to learn the connection between the input information and the scene situation, and to make predictions and inferences for similar scenes, thereby realizing multi-task learning;

[0031] In the multi-task learning, different loss functions are designed according to different tasks, and then the loss functions are weighted summed based on the weights to obtain a total loss function for model training;

[0032] The trained scene model is used to generate scene description and scene understanding.

[0033] Optionally, multiple tasks include label tasks and degree tasks:

[0034] The labeling task is set as a binary classification task, indicating the category of the scene, and the category of the scene includes whether it is a populated scene, whether it is a construction scene, whether it is a crowded scene, whether it is an emergency situation, and whether it is a high-risk area; the degree task is set as a regression task, indicating the degree of a certain situation, and the degree of the situation includes the degree of danger, the degree of congestion, the noise level, the degree of environmental abnormality, and the air quality index.

[0035] Optionally, the process of scenario analysis and reasoning is:

[0036] Obtaining the label information and degree information output by the task head, and fusing the label information and the degree information to obtain fused information; the fused information is used for scene analysis and reasoning;

[0037] Analyze the tag information and the degree information respectively to obtain a scene analysis result; the scene analysis result includes the type of the current scene and the state of the current scene;

[0038] The scene analysis results are integrated to generate descriptive text to infer the current scene situation.

[0039] The present invention discloses the following technical effects by providing a multi-task scene understanding method based on a graph attention network:

[0040] 1. Improved efficiency of multimodal data fusion: The present invention uses the graph attention network (GAT) to achieve efficient data fusion, which can simultaneously process data from different sensors and improve the real-time and accuracy of data processing.

[0041] 2. Simplified data alignment and synchronization processing: Through the structure of graph neural network, the present invention can more naturally realize the alignment and synchronization of multimodal data, reducing the complexity and computational burden of data preprocessing.

[0042] 3. Comprehensive environmental information integration and analysis: Through a multi-task learning model, the present invention can make full use of the various data provided by environmental sensors to achieve comprehensive integration and dynamic analysis of environmental information, and improve the ability to identify environmental changes and potential risks.

[0043] 4. Enhanced global scene understanding capability: By combining local detection results with global environmental information, the present invention can perform comprehensive scene analysis and reasoning, improve the understanding and decision-making capabilities of the overall scene, and solve the problem of being limited to local target detection in the prior art.

[0044] 5. Application of multi-task learning: The present invention adopts a multi-task learning method, which can perform multiple scene understanding tasks (such as hazard assessment, congestion analysis, etc.) at the same time, improves the versatility and adaptability of the model, and can better cope with complex and changing environments.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 A flow chart of a method provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of a design framework provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] like Figure 1-2As shown, the present invention provides a multi-task scene understanding method based on graph attention network. The design framework is mainly divided into two stages: scene detection stage and scene understanding stage. The scene detection stage is divided into three parts: raw data collection and preprocessing, target detection network training, and scene target data processing; the scene understanding stage is divided into three parts: scene modeling, scene understanding network training, and scene analysis and reasoning. Specifically, the following steps are included:

[0052] 1. Raw data collection: Use AGVs equipped with LiDAR, cameras, environmental information sensors, GPS sensors, and IMU sensors in the application scenario, set a unified collection interval (e.g. once per second) to simplify the data alignment and splicing process, improve system efficiency and data consistency. Realize synchronous data collection, and record GPS data, IMU data, and environmental sensor data in time series to generate raw data sets.

[0053] 1.1 Image raw data: Name the image data according to the acquisition time point.

[0054] 1.2. Point cloud raw data: Name the point cloud data according to the acquisition time.

[0055] 1.3GPS and IMU sensors record the position and attitude information of the car. Among them, latitude, longitude, and altitude describe the global position information of the car in the map, which are latitude, longitude, and altitude respectively; pitch, roll, and yaw describe the attitude information of the car, which are pitch angle, roll angle, and yaw angle respectively.

[0056] gps":[latitude,longitude,altitude]

[0057] "imu":[pitch,roll,yaw]

[0058] 1.4 Environmental information sensors include specific information on temperature, humidity, sound, oxygen concentration, CO concentration, and harmful gas concentration. An example of environmental sensor data at a certain moment is as follows:

[0059] "temperature": 20.5

[0060] "humidity": 55

[0061] "noise": 50

[0062] "O2": 21

[0063] "CO": 0.05

[0064] "hazardous_gas": 0.01

[0065] 2. Raw data preprocessing (annotation):

[0066] 2.1 Perform 2D annotation on the collected image data to obtain the 2D bounding box of the object.

[0067] Get the 2D bounding_box of the object to be tested. Where class is the category of the object to be tested; x and y are the horizontal and vertical coordinates of the center point of the standardized 2D bounding_box; w and h are the width and height of the standardized 2D bounding_box:

[0068] "class":"person"

[0069] "2d_bounding_box":[x,y,w,h]

[0070] 2.2 Perform 3D annotation on the collected point cloud data to obtain the 3D bounding box of the object and the relative position of the object and the AGV car;

[0071] Get the 3D bounding_box of the object to be measured. Among them, x, y, z represent the coordinates of the center point of the object to be measured in the local coordinate system; length, width, height represent the size of the object to be measured in three-dimensional space, that is, the length, width and height of the object in three different directions:

[0072] "class":"person"

[0073] "3d_position":[x,y,z]

[0074] "3d_dimensions":[length,width,height]

[0075] 2.3 Record GPS data, IMU data, and environmental sensor data in time series to form a complete multimodal dataset.

[0076] 3. Target detection network training: Based on the multimodal data set, the image data and point cloud data are preprocessed, and the multimodal data fusion method is used to extract object features, thereby realizing the detection of objects in the scene and obtaining the category information of all objects in the scene.

[0077] 3.1 Large-scale fusion data feature processing

[0078] The main research process is: for the collected image data and point cloud data, deep neural networks are used to extract features respectively. Then, the enhanced feature vectors of image information and point cloud information are obtained through feature enhancement technology. Finally, these feature vectors are fused using advanced fusion strategies to obtain a comprehensive feature representation.

[0079] 3.11 Data feature extraction: The image data and point cloud data collected by the visible light camera and lidar for the scene are respectively subjected to feature extraction through neural network structures (such as ResNet-50 and PointNet).

[0080] Data feature enhancement: Based on the first generalized eigenvalue decomposition theory and principal component analysis technology, the extracted large-scale data features are reduced in dimension to obtain feature enhanced data and achieve feature enhancement.

[0081] Data feature fusion: Use appropriate methods to fuse the enhanced image data features and point cloud data features, and retain the information of the original data as much as possible in the process to improve model performance and efficiency.

[0082] Considering the realistic conditions of limited resources and high real-time requirements in certain specific tasks, as well as the problem of loss or distortion of single modal information that may be encountered in complex environments. At the same time, considering the disadvantages of early fusion increasing the dimension and complexity of the data, and the disadvantage that late fusion may not fully utilize the potential correlation between modalities. This module selects the feature layer fusion strategy as the application method and conducts subsequent scheme design. By fusing multimodal data at the feature level using a multimodal data fusion strategy, the enhanced image data features and point cloud data features are fused to retain the information content of the original data, which can more effectively promote the interaction between information and improve the perception ability of the system.

[0083] 3.22 Object Detection Network Model Structure

[0084] This module selects different backbone networks (such as ResNet-50 and PointNet) for different data, and introduces the Mutimodel Cross-Attention multimodal data fusion strategy. The specific process is: in the feature fusion process, one feature is used as the query, and the other feature is used as the key and value to calculate the attention weight, and the main feature vector is updated according to the weight. This method can significantly improve the interactivity of the two modes and enhance the effect of target detection.

[0085] In the backbone network, the main function of the Head part is to process input features and output classification predictions and bounding box predictions. It extracts and transforms the input features through a series of neural network layers, and finally maps them to category predictions and bounding box coordinates. The classification head is used to determine which category the input belongs to, while the regression head is used to predict the location of the object's bounding box. The model also includes batch normalization and Dropout layers to improve training stability and prevent overfitting, thereby enhancing the overall detection performance.

[0086] 4. Scene target data processing: Based on the category information of all detected objects, the relative position information of each object is found in time series, and the relative position information is converted into global position information according to GPS data. At the same time, the current environmental information is added to generate a comprehensive data representation for each object.

[0087] 4.1 By deploying the target detection algorithm on the AGV, the category information of all objects in the scene can be obtained during the scene detection phase. Based on the category information of all detected objects, for each entity in the actual scene, the scene detection data is generated based on the detection time alignment scheme. Take the data information of an entity with the category "person" as an example:

[0088] "class":"person"

[0089] "2d_bounding_box":[x,y,w,h]

[0090] "3d_position":[x,y,z]

[0091] "3d_dimensions":[length,width,height]

[0092] "gps":[latitude,longitude,altitude]

[0093] "imu":[pitch,roll,yaw]

[0094] "temperature": 20.5

[0095] "humidity": 55

[0096] "noise": 50

[0097] "O2": 21

[0098] "CO": 0.05

[0099] "hazardous_gas": 0.01

[0100] timestamp:20240527153245

[0101] Based on the above scene detection data, the scene target information is post-processed according to subsequent algorithms and actual needs, including one-hot encoding and global coordinate transformation.

[0102] 4.2 One-hot encoding

[0103] In order to enable the subsequent scene understanding algorithm to perform effective measurement calculations in the vector space, and considering that one-hot encoding can well solve the problem of classifier processing attribute data, this module designs a data processing solution for one-hot encoding of different object categories. The specific steps are as follows:

[0104] Determine the number of object categories: Assume that there are N types of objects in the scene, and each object category is represented by a unique integer ID ranging from 0 to N-1.

[0105] Construct one-hot encoding vector: For each object category, construct a binary encoding vector of length N, in which only one position is 1 and the rest are 0. The specific encoding method is: if the object category is k, assign 1 to the kth position of the encoding vector and assign 0 to the rest.

[0106] Encoding example: Assume that there are 4 types of objects in the scene (N=4), represented by 0, 1, 2, and 3 respectively, then the corresponding one-hot encoding vectors are as follows:

[0107] Class 0: [1,0,0,0]

[0108] Class 1: [0,1,0,0]

[0109] Category 2: [0,0,1,0]

[0110] Category 3: [0,0,0,1]

[0111] 4.3 Global Coordinate Transformation

[0112] In the detection phase, the relative position of the target to the vehicle body has been obtained through point cloud data. In order to meet the needs of the subsequent scene understanding algorithm, it is necessary to combine GPS data and the posture information of the vehicle, and apply the coordinate transformation method to convert the local 3D position of each object into the global coordinate system position to achieve accurate target positioning.

[0113] Input data:

[0114] The local 3D position of the object: P local =[x local y local z local ]

[0115] The initial GPS position of the car: GPS origin = [latitude origin , longitude origin , altitude origin ]

[0116] The current GPS location of the car: GPS current = [latitude current , longitude current , altitude current ]

[0117] Car's attitude information: IMU = [pitch, roll, yaw]

[0118] Output data:

[0119] The global 3D position of the object: P global =[x global y global z global ]

[0120] The specific coordinate conversion method is as follows:

[0121] Get translation vector: T = GPS current -GPS origin

[0122] Calculate the rotation matrix R x (Roll, φ), R y (Pitch, θ), R z (Yaw, ψ) and the comprehensive rotation matrix R

[0123]

[0124]

[0125] R=R z ·R y ·R x

[0126] Construct the homogeneous rotation matrix T global

[0127]

[0128] Where R is a 3×3 rotation matrix and T is a 3×1 translation vector.

[0129] Convert local coordinates to global coordinates

[0130] The local coordinate P localExpressed as homogeneous coordinates (i.e. add 1 at the end of it), and then use the transformation matrix T global To convert, the specific formula is as follows:

[0131]

[0132] P global =[x global y global z global ]

[0133] This step is crucial for global scene understanding, as it ensures that the position of objects in the global coordinate system is accurately reflected, thereby improving the accuracy of scene analysis and understanding.

[0134] After processing the data using the above method, the data set is reconstructed. The following example takes a data set with 10 detection targets, the category is "person", and the encoding is [0,0,0,0,0,0,0,0,0,1]:

[0135] "class":[0,0,0,0,0,0,0,0,0,1]

[0136] "global_position":[x,y,z]

[0137] "3d_dimensions":[length,width,height]

[0138] "temperature": 20.5

[0139] "humidity": 55

[0140] "noise": 50

[0141] "O2": 21

[0142] "CO": 0.05

[0143] "hazardous_gas": 0.01

[0144] timestamp:20240527153245

[0145] 5. Scene modeling. Design preset rules, perform scene modeling based on the preset rules and the comprehensive data representation of all objects in the scene, and generate a scene model.

[0146] 5.1 Traditional CNN and other methods often need to flatten the data into vectors when processing graph-structured data, which is prone to losing spatial relationship information. GNN is naturally suitable for graph-structured data and can accurately model the spatial and category relationships between objects. For scene understanding tasks, objects in the scene can be regarded as nodes, and the relationships between objects (such as distance, category similarity) can be regarded as edges. The GNN network model structure can well describe the object relationships in the scene, and then understand and reason about the scene. In addition, in scene understanding tasks, single-modal information is often insufficient to fully describe the scene. GNN can fuse object features (such as object category, position, shape) and environmental features to provide a more comprehensive scene description. This information (i.e., node, adjacency matrix, and environmental feature information) is input into the network model for training in the form of node feature vectors, and the attention mechanism is used for dynamic adjustment during the training process to achieve comprehensive processing of complex multimodal data and obtain the scene model.

[0147] 5.2 Initialize the adjacency matrix:

[0148] In the GNN model, objects are regarded as nodes in the graph, and the adjacency matrix is ​​used to describe the connection relationship between nodes in the graph, and is used as a parameter input into the model training. The relationship between nodes can be initialized by object category and object distance, and dynamically adjusted through the attention mechanism during the model training stage.

[0149] Object category: Objects of the same category have higher initial weights.

[0150] Object distance: The closer the object is, the higher the initial weight.

[0151] The initial adjacency matrix A can be calculated by the following formula:

[0152] A ij =exp(-α·distance(i,j))+β·class(i,j)

[0153] Among them, α and β are weight adjustment coefficients; distance(i,j) is the distance between objects i and j; class(i,j) is a function indicating whether the object categories are the same. When objects i and j belong to the same category, the value is 1, otherwise it is 0.

[0154] 5.3 Node feature design

[0155] The features of each node should contain the following information to ensure that the object and its surrounding environment are fully described, and are used as the node feature matrix X to input the model training:

[0156] Object category: represented by one-hot encoding (for example, assuming there are n object categories, the length of the category feature is n);

[0157] Object position: Use 3D global coordinates (x, y, z) as the representation of the object in space;

[0158] Object size: Attributes that describe the size of an object using length, width, and height;

[0159] Environmental information: such as temperature, humidity, sound, gas concentration (such as O 2 , CO 2 , harmful gases) and other types of sensors describe the surrounding environment of an object.

[0160] Then the input feature dimension of each node is:

[0161] n+3+3+m=n+m+6

[0162] 6. Scene understanding network training. Use the loss function and graph attention neural network to iteratively train the scene model to obtain a trained scene model; the trained scene model is used to establish the relationship between objects, the relationship between objects and the environment, the scene comprehensive information, and the connection between different scenes.

[0163] 6.1 Network Model Structure

[0164] Graph neural network is a kind of neural network structure based on graph structure and running on graph. Graph attention network introduces attention mechanism based on graph neural network (GNN). The correlation between node features is better integrated into the model, thus achieving better neighbor aggregation. The model architecture based on GAT mainly includes the following parts:

[0165] Input layer: Input node feature matrix X and adjacency matrix A.

[0166] Graph Attention Layer: Use the GAT layer to extract node features and relationships, and enhance the expressiveness of the model through a multi-head attention mechanism.

[0167] Feature aggregation: Feature aggregation is performed through the results of multiple attention heads, and the outputs of multiple attention heads are concatenated or averaged.

[0168] Task header: Set different task headers for different tasks.

[0169] 6.2 Dynamically adjust weights:

[0170] In GAT, the attention mechanism is used to dynamically adjust the weights according to the feature similarity between nodes. The attention coefficient can be calculated by the following formula:

[0171]

[0172] Among them, e ijis the attention score between nodes i and j, calculated by a single-layer neural network:

[0173] e ij =LeakyReLU(a T [W i ||W j ])

[0174] Among them, a is the weight vector; W is the learnable weight matrix.

[0175] 6.3 Multi-task Learning

[0176] By inputting various information about objects in the scene (such as object category, location, size, shape, etc.) and environmental sensor data, and providing real labels in different scenes, and using the graph attention neural network, the model can learn the connection between this information and the scene situation, and make predictions and inferences about similar scenes. In scene understanding, using the graph attention neural network and rich object and environmental information, a variety of scene inferences can be performed, which can be mainly divided into label type and degree type.

[0177] 6.31 Labeling tasks. These inferences are usually binary classification tasks, with the output being 0 or 1, indicating the category of the scene, such as:

[0178] Is it a scene with people? This can be done by detecting pedestrians and moving crowds.

[0179] Is it a construction scene?: Determine whether there is construction activity in the scene by detecting specific construction equipment, workers, and obstacles.

[0180] Is it a crowded scene? Determine whether there are areas in the scene with many objects. This can be inferred by object category and object location.

[0181] Is it an emergency?: Determine whether there is an emergency in the scene, such as fire, explosion, etc., through data such as flame detection and smoke sensors.

[0182] Is it a high-risk area? Determine whether there are high-risk factors in the scene, such as excessive concentration of harmful gases, etc., and infer it through data from multiple sensors.

[0183] 6.32 Degree tasks. These inferences are usually regression tasks, and the output is a continuous value from 0 to 1, indicating the degree of a certain situation, such as:

[0184] Danger level: The danger level of the scene is comprehensively assessed based on multiple factors such as gas concentration, flame detection, temperature, etc. It can be expressed as a decimal between 0 and 1, with 0 indicating complete safety and 1 indicating extreme danger.

[0185] Crowding: Assess the crowding level of the scene based on the number and distribution of objects. It can be expressed as a decimal between 0 and 1, where 0 means no crowding at all and 1 means extremely crowded.

[0186] Noise Level: Assess the noise level of the scene based on the data from the sound sensor. This can be expressed as a decimal between 0 and 1, where 0 means quiet and 1 means very noisy.

[0187] Environmental abnormality level: Evaluates the environmental abnormality level of the scene based on the data from the temperature / humidity sensor. It can be represented by a decimal between 0 and 1, where 0 represents a normal environment and 1 represents a severe environment.

[0188] Air Quality Index: Assess air quality based on data from a variety of air sensors (such as PM2.5, CO2 concentration, etc.), which can be expressed as a decimal between 0 and 1, where 0 indicates very good air quality and 1 indicates extremely poor air quality.

[0189] 6.4 Loss Function Design In the multi-task learning, different loss functions are designed according to different tasks, and then the weighted sum of each loss function is performed based on the weight to obtain the total loss function for model training.

[0190] In multi-task learning, different tasks usually have different loss functions. In order to effectively train a multi-task model, it is necessary to design an overall loss function that integrates the losses of each task. For label tasks, such as classification tasks, the commonly used loss function is cross-entropy loss. For degree tasks, such as regression tasks, the commonly used loss function is mean squared error loss (MSE).

[0191] Label tasks are usually classification tasks, and the cross entropy loss function is used to measure the gap between the model output and the true label. For each subtask, an independent loss function is designed as follows:

[0192] Is it a scene with people?

[0193]

[0194] Is it a construction scene?

[0195]

[0196] Is it a crowded scene?

[0197]

[0198] Is it an emergency?

[0199]

[0200] Is it a high-risk area?

[0201]

[0202] Therefore, the overall loss function for the label class task is:

[0203]

[0204] Among them, λ is_people ,λ is_constructtion ,λ is_congestion ,λ is_emergency ,λ is_highrisk is the weight value of each loss in the design.

[0205] Degree tasks are usually regression tasks, and the mean square error loss (MSE) is used to measure the gap between the model output and the true value. For each subtask, an independent loss function is designed as follows:

[0206] Danger level:

[0207]

[0208] Crowding level:

[0209]

[0210] Noise Level:

[0211]

[0212] Environmental abnormality:

[0213]

[0214] Air Quality Index:

[0215]

[0216] Therefore, the overall loss function for degree tasks is:

[0217] Loss regression =λ danger ·Loss danger +λ crowding ·Loss crowding +λ noise ·Loss noise

[0218] +λ environment ·Loss environment +λ air_quality ·Loss air_quality

[0219] where λ danger ,λ crowding ,λ noise ,λ environment ,λ air_quality is the weight value of each loss in the design.

[0220] In multi-task learning, the loss functions of each task can be weighted and summed to obtain an overall loss function. The choice of weights can be adjusted according to the importance of the task or the experience during training.

[0221] The basic form of the overall loss function is:

[0222] Loss total =λ classification ·Loss classification +λ regression ·Loss regression

[0223] Among them, Loss classification is the loss function for label-based tasks (such as cross entropy loss); Loss regression is the loss function of the degree task (such as mean square error loss); λ classification and λ regression is the weight coefficient of each loss function.

[0224] 7. Scene analysis and reasoning. Perform various scene analysis tasks based on the trained scene model, and infer the overall situation of the scene through the scene analysis results.

[0225] The label information and degree information output by the multi-task head are fused to obtain fused information; the fused information is used for scene analysis and reasoning. It helps users quickly understand the overall situation of the scene and make better decisions. The following is a specific implementation of this comprehensive description:

[0226] 7.1 Fusion of label information and degree information

[0227] The system first obtains the results output by each task head, including five label tasks and five degree tasks. This information is integrated together as the basis for scene analysis and reasoning.

[0228] 7.2 Scene type identification and description

[0229] By analyzing various types of tag information, the system can comprehensively determine the type of the current scene. The degree information provides a detailed assessment of the scene status.

[0230] 7.3 Comprehensive Description Generation

[0231] The system combines the above analysis results to generate a descriptive text to help users quickly understand the status of the current scene.

[0232] In practical applications, through the above steps, the system can perform scenario model analysis for different scenarios, and perform scenario reasoning according to the designed model labels to achieve overall judgment of the scenario, including scenario type recognition, risk assessment, congestion assessment, emergency warning, etc. These reasoning results can not only improve the adaptability of the AGV system to complex environments, but also significantly improve the application effect and decision-making ability of smart mobile devices in industrial and commercial environments, provide solid technical support for more efficient and safe automated operations, and also provide a reliable method for people to grasp the overall environment.

[0233] Therefore, the present invention provides a multi-task scene understanding method based on graph attention network, which can improve the efficiency of multimodal data fusion, simplify data alignment and synchronization processing, comprehensively integrate and analyze environmental information, enhance global scene understanding capabilities, and perform multiple scene understanding tasks at the same time, so that the present invention can achieve comprehensive perception and intelligent reasoning of complex scenes, and improve the application effect and decision-making ability of smart mobile devices in industrial and commercial environments.

[0234] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0235] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multi-task scene understanding method based on graph attention network, characterized in that: The following steps are involved: Set a unified collection interval, use an automated guided vehicle to synchronously collect data in the application scenario, annotate the collected data, and then record the collected data in time series to obtain a multimodal data set; Based on the multimodal data set, object features are extracted using a multimodal data fusion method to implement object detection training in the application scene and obtain category information of all objects in the application scene; Based on the category information of all detected objects, the relative position information of each object is found in time series, and the relative position information is converted into global position information, and the current time environment information is added to the global position to generate a comprehensive data representation of each object; Design preset rules, perform scene modeling based on the preset rules and the comprehensive data representation of all objects in the scene, and generate a scene model; Iteratively training the scene model using the loss function and the graph attention neural network to obtain a trained scene model; the trained scene model is used to establish the relationship between objects, the relationship between objects and the environment, scene comprehensive information, and the connection between different scenes; Perform various scenario analysis tasks based on the trained scenario model, and infer the overall situation of the scenario through the scenario analysis results; The generation process of multimodal datasets is as follows: The automatic guided vehicle is equipped with a laser radar, a camera, an environmental information sensor, a GPS sensor, and an IMU sensor, and data is synchronously collected at a set collection interval to obtain an original data set; Performing 2D annotation on the image data in the original data set to obtain a 2D bounding box of the object, performing 3D annotation on the point cloud data in the original data set to obtain a 3D bounding box of the object and a relative position of the object and the automatic guided vehicle, and then recording GPS data, IMU data, and environmental sensor data in time series to obtain the multimodal data set; The process of scene modeling is: Using the GNN network model, objects in the scene are set as nodes; the relationship between the nodes is set as an adjacency matrix; The node, the adjacency matrix and the environmental feature information are input into the GNN network model for training by means of node feature vectors, and the attention mechanism is used for dynamic adjustment during the training process to comprehensively process multimodal data to obtain the scene model; The process of using loss function and graph attention neural network to train the model to generate scene description and scene understanding is: Using the attention mechanism, the weights in the graph attention layer are dynamically adjusted according to the feature similarity between nodes; Various information of objects in the scene, environmental sensor data, and real labels of different scenes are input into the scene model, and the graph attention neural network is used to enable the scene model to learn the connection between the input information and the scene situation, and to make predictions and inferences for similar scenes, thereby realizing multi-task learning; In the multi-task learning, different loss functions are designed according to different tasks, and then the loss functions are weighted summed based on the weights to obtain a total loss function for model training; Generate scene description and scene understanding using the trained scene model; Multi-tasks include label tasks and degree tasks: The labeling task is set as a binary classification task, indicating the category of the scene, including whether it is a scene with people, whether it is a construction scene, whether it is a crowded scene, whether it is an emergency, and whether it is a high-risk area; the degree task is set as a regression task, indicating the degree of a certain situation, including the degree of danger, the degree of crowding, the noise level, the degree of environmental abnormality, and the air quality index; The process of scene analysis and reasoning is: Obtaining label information and degree information output by the task head, and fusing the label information and the degree information to obtain fused information; the fused information is used for scene analysis and reasoning; Analyze the label information and the degree information respectively to obtain a scene analysis result; The scene analysis result includes the type of the current scene and the state of the current scene; The scene analysis results are integrated to generate descriptive text to infer the current scene situation.

2. According to claim 1, a multi-task scene understanding method based on graph attention network is characterized in that: The process of extracting object features using multimodal data fusion methods is as follows: The deep neural network is used to extract features from the collected image data and point cloud data respectively, and then the first generalized eigenvalue decomposition theory and principal component analysis technology are used to reduce the dimension of the extracted features to obtain image data features and point cloud data features; The image data features and the point cloud data features are fused using a multimodal data fusion strategy to retain the information content of the original data; in the feature fusion process, different backbone networks are selected for different data, attention weights are calculated, and then the main feature vectors are updated respectively according to the attention weights.

3. According to claim 2, a multi-task scene understanding method based on graph attention network is characterized in that: The data processing process represented by the comprehensive data is as follows: Based on the category information of all detected objects, each entity in the scene is detected using a detection time alignment scheme to generate scene detection data, and then one-hot encoding and global coordinate conversion are performed based on the scene detection data, and environmental information of the current time is added to generate a comprehensive data representation of each object; wherein the one-hot encoding determines the number of object categories and constructs a unique hot encoding vector for each object category to learn the characteristics of each object category; the global coordinate conversion combines GPS data and the posture information of the automatic guided vehicle, and applies a coordinate conversion method to convert the local 3D position of each object into a global coordinate system position to achieve the positioning of the target object.

4. According to claim 3, a multi-task scene understanding method based on graph attention network is characterized in that: The graph attention neural network includes: Input layer, used to input node feature matrix and adjacency matrix; A graph attention layer, which is used to extract node features and relationships and enhance the expression capability of the scene model through a multi-head attention mechanism; Feature aggregation, which is used to aggregate features through the results of multiple attention heads and concatenate or average the outputs of multiple attention heads; Different task headers are set for different tasks; the task headers are used to output label information and degree information.

Images