Adaptive Scene Analysis and Target Generation Method and System Based on Deep Learning

Abstract

The present invention provides an adaptive scene analysis and target generation method and system based on deep learning, which relates to the field of computer vision technology. It includes extracting the depth information of a scene image sequence, constructing three-dimensional point cloud data, calculating the distance and direction matrix between objects, and then generating a scene space feature vector. At the same time, a semantic label map is obtained through semantic segmentation, a semantic association matrix is constructed, and a scene semantic vector is calculated. The two are input into a variational autoencoder to obtain a target feature vector, and an attention score is calculated based on this to generate a weighted target description vector. Finally, a target contour feature is generated through a probabilistic graph inference network, and combined with a regional attention map, a final target result is generated. This method improves the accuracy and efficiency of target generation.

Description

Technical Field

[0001] The present invention relates to computer vision technology, and in particular to an adaptive scene analysis and target generation method and system based on deep learning. Background Art

[0002] With the development of computer vision and artificial intelligence technologies, scene analysis and target generation play an increasingly important role in fields such as autonomous driving, intelligent security, and robot navigation. Traditional scene analysis methods mainly rely on manually designed features or pattern matching based on shallow models, and it is difficult to adapt to complex and changing environments. In recent years, the rise of deep learning technology has made end-to-end feature extraction and target recognition possible, but existing methods often focus on the analysis of a single data source, such as RGB images or point cloud data, and fail to fully integrate the spatial geometric information and semantic information of the scene, thus limiting the accuracy and applicability of target generation.

[0003] Existing methods usually only apply this information in specific tasks, lacking a unified feature expression framework, resulting in the underutilization of information complementarity between different data modalities. In addition, the accuracy of current target generation technology in complex scenarios depends on a large amount of labeled data, with high training costs and limited generalization ability. How to efficiently extract scene features, optimize the target generation process, and improve the adaptability of the model in complex environments remains a key problem to be solved urgently.

[0004] Therefore, there is an urgent need for an adaptive scene analysis and target generation method based on deep learning, which can effectively extract the spatial and semantic features of the scene, and perform feature distribution modeling through a variational autoencoder, thereby generating a more accurate target description vector. At the same time, combining a probabilistic graphical inference network to achieve the fusion of scene information and target generation constraints, improving the accuracy and adaptability of target generation to meet the requirements of intelligent vision systems for efficient and robust scene understanding and target generation. Summary of the Invention

[0005] Embodiments of the present invention provide an adaptive scene analysis and target generation method and system based on deep learning, which can solve the problems in the prior art.

[0006] In the first aspect of the embodiments of the present invention,

[0007] An adaptive scene analysis and target generation method based on deep learning is provided, including:

[0008] Extract the depth information of each frame of the scene image sequence, construct three-dimensional point cloud data according to the depth information, calculate the distance matrix and direction matrix between objects in the scene based on the three-dimensional point cloud data, and combine the distance matrix and direction matrix to obtain a scene spatial feature vector;

[0009] Semantically segment the scene image sequence to obtain a semantic label map, construct a semantic association matrix for the labels in the semantic label map according to the label co-occurrence frequency, and calculate the scene semantic vector based on the semantic association matrix;

[0010] Input the scene spatial feature vector and the scene semantic vector into a variational autoencoder. Through the encoding process of the variational autoencoder, obtain the scene feature distribution, sample from the scene feature distribution to get the target feature vector, calculate the attention score based on the target feature vector, and update the weight of the target feature vector according to the attention score to generate a weighted target description vector;

[0011] Construct a probabilistic graph inference network, map the scene spatial feature vector to a position probability distribution, map the scene semantic vector to a semantic probability distribution, construct a target generation constraint based on the position probability distribution and the semantic probability distribution, and combine the target generation constraint and the target description vector and input them into a generation network. The generation network outputs the target contour features according to the target generation constraint;

[0012] Construct a regional attention map based on the target contour features, and combine the regional attention map with the target description vector to generate the final target generation result.

[0013] In an alternative embodiment,

[0014] Extract the depth information of each frame in the scene image sequence, construct three-dimensional point cloud data based on the depth information, calculate the distance matrix and direction matrix between objects in the scene based on the three-dimensional point cloud data, and combine the distance matrix and the direction matrix to obtain the scene spatial feature vector, including:

[0015] Calculate the adjacent frame pixel mapping relationship to obtain the displacement field, repair the depth information according to the displacement field to obtain the three-dimensional point cloud, perform dual feature extraction of the normal vector and curvature field on the three-dimensional point cloud to achieve object segmentation, calculate the direction matrix between objects and the distance matrix considering convex hull overlap, and fuse the direction matrix and the distance matrix through an attention mechanism and a residual connection to obtain the scene spatial feature, specifically including:

[0016] Collect the scene image sequence, perform temporal completion on the depth information of each frame in the scene image sequence, calculate the forward mapping matrix of the pixel position correspondence between the current frame and the previous frame, calculate the backward mapping matrix of the pixel position correspondence between the current frame and the next frame, and calculate the displacement field of the pixel position based on the forward mapping matrix and the backward mapping matrix;

[0017] Calculate the motion consistency score based on the displacement field, use the motion consistency score as a weight coefficient to perform weighted combination on the depth values of adjacent frames to obtain a depth completion function, use the depth completion function to repair the missing depth values of the current frame to obtain a repaired depth map, and multiply the pixel coordinates and depth values in the repaired depth map by the camera intrinsic matrix to obtain a three-dimensional point cloud;

[0018] Construct a search radius for each point in the three-dimensional point cloud, extract a neighborhood point set within the search radius, calculate the mean vector of the neighborhood point set, construct a covariance matrix using the neighborhood point set and the mean vector, perform eigenvalue decomposition on the covariance matrix to obtain a normal vector, calculate the angle between the normal vector and the neighborhood point normal vector to obtain a normal deviation value, and remove the points with the normal deviation value greater than the preset deviation threshold to obtain a filtered point cloud;

[0019] Calculate the normal change rate of each point in the filtered point cloud to obtain a normal vector field, use the normal vector field to calculate the local curvature to obtain a curvature field, input the normal vector field and the curvature field into a region growing algorithm to segment and obtain a planar point set, calculate the spatial distance for the points outside the planar point set to obtain a clustering label, and segment the point cloud into multiple object point sets according to the clustering label;

[0020] Calculate the centroid position and the point cloud covariance matrix for each of the object point sets respectively, perform eigenvalue decomposition on the point cloud covariance matrix to obtain a principal direction vector, project the principal direction vector onto three coordinate planes to obtain projection vectors, calculate the angle between objects based on the projection vectors to obtain a direction matrix, calculate the distance value between objects using the centroid position, construct a convex hull of the object point set and calculate the convex hull overlap degree, combine the distance value and the convex hull overlap degree to obtain a distance matrix, input the direction matrix and the distance matrix into a multi-layer perceptron, calculate attention scores on each layer of feature maps in the multi-layer perceptron, perform selective fusion on features according to the attention scores, and transmit the fused features through residual connections to obtain a scene space feature vector.

[0021] In an alternative embodiment,

[0022] Perform semantic segmentation on the scene image sequence to obtain a semantic label map, construct a semantic association matrix according to the co-occurrence frequency of the labels in the semantic label map, and calculate the scene semantic vector based on the semantic association matrix, including:

[0023] Construct a temporal consistency loss to optimize the segmentation network to obtain a semantic label sequence, calculate the conditional entropy for the semantic label sequence to construct an information entropy weight matrix, multiply the information entropy weight matrix by the co-occurrence frequency matrix to obtain a semantic association matrix, construct a graph structure with semantic categories as nodes and the semantic association matrix as edge weights, and use a multi-head attention mechanism and skip connections to fuse node features to obtain a scene semantic vector, specifically including:

[0024] The semantic segmentation of the scene image sequence is performed using a deep convolutional neural network to obtain a semantic label sequence. The forward mapping matrix and the backward mapping matrix are calculated according to the pixel position correspondence between adjacent frames. The semantic prediction result of the current frame is aligned with adjacent frames by using the forward mapping matrix and the backward mapping matrix. The difference between the aligned prediction results is calculated to construct a temporal consistency loss function. The parameters of the semantic segmentation network are optimized by minimizing the temporal consistency loss function to obtain a semantic label sequence with temporal consistency;

[0025] The semantic categories are extracted from each frame of semantic labels in the semantic label sequence, and the co-occurrence frequency of each pair of semantic categories in the same scene is counted to construct a co-occurrence frequency matrix. The joint probability and marginal probability of the semantic category pairs in the co-occurrence frequency matrix are calculated to obtain the conditional entropy. The information entropy weight matrix is constructed by using the conditional entropy. The information entropy weight matrix is multiplied by the co-occurrence frequency matrix and normalized to obtain a semantic association matrix;

[0026] The semantic categories and the semantic association matrix are respectively constructed as the node set and edge weights of the graph structure. The neighborhood node features are extracted for each node in the graph structure. The attention coefficients between nodes are calculated by using the multi-head attention mechanism. The neighborhood node features are weighted and aggregated according to the attention coefficients to obtain the updated features of the nodes. The multi-layer node updated features are fused through skip connections, and the fused node features are concatenated to obtain a scene semantic vector containing object category information and semantic association patterns.

[0027] In an alternative embodiment,

[0028] The scene spatial feature vector and the scene semantic vector are input into a variational autoencoder. The scene feature distribution is obtained through the encoding process of the variational autoencoder. The target feature vector is sampled from the scene feature distribution. The attention scores are calculated based on the target feature vector. The weights of the target feature vector are updated according to the attention scores, and a weighted target description vector is generated, including:

[0029] The scene spatial feature vector and the scene semantic vector are input into the feature decomposition unit of the variational autoencoder. The prior probability distribution parameters are obtained through Bayesian inference. The conditional prior distribution is constructed based on the prior probability distribution parameters. The shared information and independent information in the scene spatial feature vector and the scene semantic vector are respectively extracted to obtain a feature shared subspace and a feature independent subspace. The weights of the feature shared subspace and the feature independent subspace are calculated based on the mutual information maximization criterion. The feature subspaces are weighted and combined according to the weights to obtain a scene feature distribution;

[0030] Input the scene feature distributions into multiple probability distribution branches of the variational autoencoder respectively, calculate distribution parameters for each branch based on the statistical characteristics of the scene feature distributions, calculate the information entropy values of each branch using the distribution parameters, and sample the target feature vector from the scene feature distributions according to the distribution parameters of the branch with the minimum information entropy value;

[0031] Construct each dimension of the target feature vector as a node in the conditional random field, calculate the feature similarity between adjacent nodes to obtain the pairwise potential function, calculate the combined similarity of all nodes within the node group to obtain the combined potential function, and construct the pairwise potential function and the combined potential function into an energy function, and minimize the energy function through an iterative optimization method to obtain the attention scores;

[0032] Multiply the attention scores with each dimension of the target feature vector to obtain the feature vector with updated weights, calculate the statistical value of the feature vector with updated weights to obtain the gating threshold, and retain the feature dimensions in the feature vector with updated weights that are greater than the gating threshold to obtain the target description vector with weights.

[0033] In an alternative embodiment,

[0034] Construct each dimension of the target feature vector as a node in the conditional random field, calculate the feature similarity between adjacent nodes to obtain the pairwise potential function, calculate the combined similarity of all nodes within the node group to obtain the combined potential function, and constructing the pairwise potential function and the combined potential function into an energy function includes:

[0035] Construct the eigenvalue of each dimension of the target feature vector as a node in the conditional random field, calculate the correlation coefficient between the nodes, set a connection threshold based on the correlation coefficient, and establish a connection relationship for the node pairs with a correlation coefficient greater than the connection threshold to obtain an adaptive node connection graph, where the node connection graph includes a node set and an edge set;

[0036] Calculate the Euclidean distance between the eigenvalues corresponding to adjacent nodes in the node connection graph, and calculate the feature similarity between adjacent nodes based on the Euclidean distance to obtain the pairwise potential function;

[0037] Calculate the mean value of the eigenvalues of all nodes within the node group in the node connection graph, calculate the distance between each node eigenvalue and the mean value based on the mean value, and calculate the combined similarity of the node group according to the distance to obtain the combined potential function;

[0038] Calculate the logarithm value of the pairwise potential function to obtain the first energy term, calculate the logarithm value of the combined potential function to obtain the second energy term, and sum the first energy term and the second energy term after assigning weight coefficients to them to obtain the energy function.

[0039] In an alternative embodiment,

[0040] Construct a probabilistic graph inference network, map the scene space feature vector to a position probability distribution, map the scene semantic vector to a semantic probability distribution, construct a target generation constraint based on the position probability distribution and the semantic probability distribution, and combine the target generation constraint and the target description vector and input them into the generation network. The generation network outputs target contour features according to the target generation constraint, including:

[0041] Construct a two-stream probabilistic graph network to extract position and semantic features respectively, form a target generation constraint by calculating the divergence between the position probability distribution and the uniform distribution and the divergence between the semantic probability distribution and the preset distribution, splice the target generation constraint and the target description vector in the feature dimension, and output target contour features through the generation network, specifically including:

[0042] Construct a probabilistic graph inference network, the probabilistic graph inference network includes a position feature stream and a semantic feature stream. The position feature stream processes the scene space feature vector through multiple layers of graph convolution to obtain a position feature map, and the semantic feature stream calculates attention weights to weight the scene semantic vector to obtain semantic features;

[0043] Perform a normalization operation on the position feature map to obtain a position probability distribution, and perform a non-linear transformation on the semantic features to obtain a semantic probability distribution;

[0044] Calculate the divergence value between the position probability distribution and the uniform distribution to obtain a position constraint term, calculate the divergence value between the semantic probability distribution and the preset semantic distribution to obtain a semantic constraint term, and perform a weighted sum on the position constraint term and the semantic constraint term to obtain a target generation constraint;

[0045] Splice the target generation constraint and the target description vector in the feature dimension, input the spliced features into the generation network, and the generation network magnifies the spatial scale of the input features through an upsampling convolutional layer and outputs target contour features.

[0046] In an alternative embodiment,

[0047] Construct a regional attention map based on the target contour features, and combine the regional attention map and the target description vector to generate a final target generation result, including:

[0048] Use a sliding window to extract the maximum response and average response features simultaneously to construct spatial attention, combine it with channel attention to form a regional attention mechanism to obtain a regional attention map, use residual connection to enhance feature expression, and perform multi-scale fusion on the regional attention map and the target description features and then decode to generate a target result, specifically including:

[0049] The target contour features are scanned in a sliding window manner, and the statistical values of the features within the window are calculated based on each sliding window position, including calculating the maximum value to obtain the maximum response feature map, calculating the average value to obtain the average response feature map, performing convolution operations on the maximum response feature map and the average response feature map respectively, adding them together, and obtaining the spatial attention map through normalization processing;

[0050] The target contour features are subjected to average pooling in the spatial dimension and passed through a fully connected layer to obtain the channel attention map, and the spatial attention map, the channel attention map, and the target contour features are multiplied to obtain the regional attention map;

[0051] The regional attention map is subjected to convolution processing to obtain local features, and the local features and the regional attention map are subjected to residual connection to obtain enhanced features;

[0052] The target description vector is upsampled and extended, the extended features and the enhanced features are concatenated in the feature dimension, and the concatenated features are subjected to multi-scale processing through convolutional layers with different kernel sizes;

[0053] The features after multi-scale processing are input into the decoder, and the decoder decodes the features through deconvolution layers and skip connections to generate the target result.

[0054] In the second aspect of the embodiments of the present invention,

[0055] There is provided an adaptive scene analysis and target generation system based on deep learning, including:

[0056] A first unit for extracting the depth information of each frame of image in the scene image sequence, constructing three-dimensional point cloud data based on the depth information, calculating the distance matrix and direction matrix between objects in the scene based on the three-dimensional point cloud data, and combining the distance matrix and direction matrix to obtain the scene space feature vector;

[0057] A second unit for performing semantic segmentation on the scene image sequence to obtain a semantic label map, constructing a semantic association matrix according to the co-occurrence frequency of the labels in the semantic label map, and calculating the scene semantic vector according to the semantic association matrix;

[0058] A third unit for inputting the scene space feature vector and the scene semantic vector into a variational autoencoder, obtaining the scene feature distribution through the encoding process of the variational autoencoder, sampling from the scene feature distribution to obtain the target feature vector, calculating the attention score based on the target feature vector, updating the weight of the target feature vector according to the attention score, and generating a weighted target description vector;

[0059] The fourth unit constructs a probabilistic graph inference network, maps the scene space feature vector to a position probability distribution, maps the scene semantic vector to a semantic probability distribution, constructs a target generation constraint based on the position probability distribution and the semantic probability distribution, combines the target generation constraint and the target description vector and inputs them into the generation network, and the generation network outputs the target contour features according to the target generation constraint;

[0060] The fifth unit constructs a regional attention map based on the target contour features, and combines the regional attention map with the target description vector to generate the final target generation result.

[0061] In the third aspect of the embodiments of the present invention,

[0062] A kind of electronic device is provided, including:

[0063] A processor;

[0064] A memory for storing instructions executable by the processor;

[0065] Wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0066] In the fourth aspect of the embodiments of the present invention,

[0067] A computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0068] In this embodiment, by fusing depth information, 3D point clouds and semantic labels, the spatial structure and semantic relationships of the scene are comprehensively extracted, making target recognition and generation more accurate. At the same time, the variational autoencoder is used to construct the scene feature distribution, and the attention mechanism is combined to optimize the target description vector, enabling the model to adapt to different scenes and improving the stability and generalization ability of target feature extraction. In addition, a probabilistic graph inference network is used to model the scene spatial features and semantic features, making the target generation process not only depend on local information, but also be able to integrate global constraints, enhancing the rationality of target generation. On this basis, the target contour features are optimized by combining the regional attention map, making the target generation result clearer and more accurate, effectively reducing background interference, and improving the adaptability of the model in complex environments. It can effectively improve the overall accuracy of scene understanding and target generation, while reducing the dependence on large-scale labeled data and improving the training and inference efficiency. It is widely applicable to fields such as autonomous driving, intelligent monitoring, and robot navigation, can enhance the environmental perception ability of intelligent systems, and provide a more robust solution for intelligent vision applications. Description of the Drawings

[0069] Figure 1Schematic flowchart of the method for adaptive scene analysis and target generation based on deep learning according to an embodiment of the present invention;

[0070] Figure 2 Analysis diagram of the application effect of the scene semantic vector in the downstream task according to an embodiment of the present invention;

[0071] Figure 3 Analysis diagram of the probability distribution divergence according to an embodiment of the present invention;

[0072] Figure 4 Schematic diagram of the decoder structure according to an embodiment of the present invention;

[0073] Figure 5 Schematic diagram of the structure of the system for adaptive scene analysis and target generation based on deep learning according to an embodiment of the present invention. Detailed implementation manners

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0075] The technical solutions of the present invention will be described in detail below with specific embodiments. These specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0076] Figure 1 Schematic flowchart of the method for adaptive scene analysis and target generation based on deep learning according to an embodiment of the present invention, as Figure 1 shown, the method includes:

[0077] S101. Extract the depth information of each frame of the scene image sequence, construct three-dimensional point cloud data according to the depth information, calculate the distance matrix and direction matrix between objects in the scene based on the three-dimensional point cloud data, and combine the distance matrix and direction matrix to obtain a scene space feature vector;

[0078] S102. Perform semantic segmentation on the scene image sequence to obtain a semantic label map, construct a semantic association matrix according to the co-occurrence frequency of the labels in the semantic label map, and calculate the scene semantic vector according to the semantic association matrix;

[0079] S103. Input the scene space feature vector and the scene semantic vector into the variational autoencoder. Through the encoding process of the variational autoencoder, obtain the scene feature distribution. Sample a target feature vector from the scene feature distribution. Calculate the attention score based on the target feature vector, and update the weight of the target feature vector according to the attention score to generate a weighted target description vector;

[0080] S104. Construct a probabilistic graph inference network. Map the scene space feature vector to a position probability distribution, and map the scene semantic vector to a semantic probability distribution. Construct a target generation constraint based on the position probability distribution and the semantic probability distribution. Combine the target generation constraint and the target description vector and input them into the generation network. The generation network outputs the target contour feature according to the target generation constraint;

[0081] S105. Construct a regional attention map based on the target contour feature, and combine the regional attention map with the target description vector to generate the final target generation result.

[0082] In an alternative embodiment, extract the depth information of each frame in the scene image sequence, construct three-dimensional point cloud data according to the depth information, calculate the distance matrix and the direction matrix between objects in the scene based on the three-dimensional point cloud data, and the combination of the distance matrix and the direction matrix to obtain the scene space feature vector includes:

[0083] Calculate the displacement field by calculating the pixel mapping relationship between adjacent frames, repair the depth information according to the displacement field to obtain a three-dimensional point cloud, perform double feature extraction of the normal vector and the curvature field on the three-dimensional point cloud to realize object segmentation, calculate the direction matrix between objects and the distance matrix considering convex hull overlap, and fuse the direction matrix and the distance matrix through the attention mechanism and the residual connection to obtain the scene space feature, specifically including:

[0084] Collect the scene image sequence, perform temporal completion on the depth information of each frame in the scene image sequence, calculate the forward mapping matrix of the pixel position correspondence between the current frame and the previous frame, calculate the backward mapping matrix of the pixel position correspondence between the current frame and the next frame, and calculate the displacement field of the pixel position based on the forward mapping matrix and the backward mapping matrix;

[0085] Calculate the motion consistency score according to the displacement field, use the motion consistency score as a weight coefficient to perform weighted combination on the depth values of adjacent frames to obtain a depth completion function, use the depth completion function to repair the missing depth value of the current frame to obtain a repaired depth map, and multiply the pixel coordinates and depth values in the repaired depth map by the camera intrinsic matrix to obtain a three-dimensional point cloud;

[0086] Construct a search radius for each point in the 3D point cloud, extract the neighborhood point set within the search radius, calculate the mean vector of the neighborhood point set, construct a covariance matrix using the neighborhood point set and the mean vector, perform eigenvalue decomposition on the covariance matrix to obtain the normal vector, calculate the angle between the normal vector and the neighborhood point normal vector to obtain the normal deviation value, and remove the points with the normal deviation value greater than the preset deviation threshold to obtain the filtered point cloud;

[0087] Calculate the normal change rate of each point in the filtered point cloud to obtain the normal vector field, calculate the local curvature using the normal vector field to obtain the curvature field, input the normal vector field and the curvature field into the region growing algorithm to segment and obtain the plane point set, calculate the spatial distance for the points outside the plane point set to obtain the clustering label, and segment the point cloud into multiple object point sets according to the clustering label;

[0088] Calculate the centroid position and the point cloud covariance matrix for each of the object point sets respectively, perform eigenvalue decomposition on the point cloud covariance matrix to obtain the principal direction vector, project the principal direction vector onto the three coordinate planes to obtain the projection vectors, calculate the angle between the objects based on the projection vectors to obtain the direction matrix, calculate the distance value between the objects using the centroid position, construct the convex hull of the object point set and calculate the convex hull overlap degree, combine the distance value and the convex hull overlap degree to obtain the distance matrix, input the direction matrix and the distance matrix into the multi-layer perceptron, calculate the attention score on each layer feature map of the multi-layer perceptron, perform selective fusion on the features according to the attention score, and transmit the fused features through residual connection to obtain the scene space feature vector.

[0089] Exemplarily, first, collect the scene image sequence. Shoot the target scene with a high-resolution camera to ensure the clarity and details of each frame of the image. Each frame of the image will be processed to extract its depth information, usually using a depth sensor or performing depth estimation through computer vision algorithms.

[0090] Next, perform temporal completion on the depth information of each frame of the image. By analyzing the depth information between adjacent frames, calculate the pixel position correspondence relationship between the current frame and the previous frame to obtain the forward mapping matrix. At the same time, calculate the pixel position correspondence relationship between the current frame and the subsequent frame to obtain the backward mapping matrix. Based on these two mapping matrices, calculate the displacement field of the pixel positions for depth information repair in subsequent steps.

[0091] According to the calculated displacement field, evaluate the motion consistency score. The motion consistency score reflects the motion similarity between adjacent frames, and is used as a weight coefficient to perform weighted combination on the depth values of adjacent frames to form a depth completion function. Use this depth completion function to repair the missing depth values in the current frame to generate the repaired depth map. Multiply the pixel coordinates and depth values in the repaired depth map by the camera intrinsic matrix to obtain the 3D point cloud data.

[0092] In a three-dimensional point cloud, a search radius is constructed for each point, and a neighborhood point set is extracted. The mean vector of the neighborhood point set is calculated, and a covariance matrix is constructed using this mean vector. The covariance matrix is subjected to eigenvalue decomposition to obtain a normal vector. The angle between the normal vector and the normal vectors of the neighborhood points is calculated to obtain a normal deviation value. Points with normal deviation values greater than a preset threshold are removed to obtain a filtered point cloud.

[0093] Next, the normal change rate of each point in the filtered point cloud is calculated to obtain a normal vector field. The local curvature is calculated using the normal vector field to obtain a curvature field. The normal vector field and the curvature field are input into a region growing algorithm for object segmentation to obtain a planar point set. The spatial distance is calculated for points outside the planar point set to obtain a clustering label. According to the clustering label, the point cloud is segmented into multiple object point sets.

[0094] For each object point set, its centroid position and the point cloud covariance matrix are calculated. The covariance matrix is subjected to eigenvalue decomposition to obtain a principal direction vector. The principal direction vector is projected onto three coordinate planes to obtain projection vectors. Based on the projection vectors, the angle between objects is calculated to form a direction matrix. At the same time, the distance value between objects is calculated using the centroid position, and the convex hull of the object point set is constructed and the convex hull overlap degree is calculated. The distance value and the convex hull overlap degree are combined to obtain a distance matrix.

[0095] Finally, the direction matrix and the distance matrix are input into a multi-layer perceptron. The attention score is calculated on each layer's feature map of the multi-layer perceptron, and the features are selectively fused according to the attention score. The fused features are transmitted through residual connections, and finally a scene space feature vector is obtained.

[0096] In this embodiment, by calculating the pixel mapping relationship between adjacent frames, displacement field information is obtained, and depth information repair is completed based on the motion consistency score, making the point cloud data more complete and reducing the reconstruction error caused by depth loss. Combining the dual feature extraction method of normal vectors and curvature fields, the point cloud data is refined, effectively improving the accuracy of object segmentation. In terms of calculating object relationships, this scheme constructs a direction matrix between objects by calculating the centroid position, covariance matrix, and principal direction vector, and combines the convex hull overlap degree to calculate a distance matrix, thereby more accurately depicting the spatial relationship between objects. Using the attention mechanism and residual connections to fuse the direction matrix and the distance matrix effectively enhances the feature expression ability, enabling the scene space feature vector to more fully describe the geometric structure and spatial distribution relationship between objects. It can improve the accuracy of scene understanding and object segmentation, making the point cloud data more applicable in complex environments. At the same time, the method based on depth information completion reduces the dependence on high-quality sensing devices and improves the stability and robustness of the three-dimensional point cloud.

[0097] In an alternative embodiment, semantic segmentation is performed on the scene image sequence to obtain a semantic label map, and a semantic association matrix is constructed according to the co-occurrence frequency of the labels in the semantic label map. Calculating the scene semantic vector based on the semantic association matrix includes:

[0098] Construct a temporal consistency loss to optimize the segmentation network to obtain a semantic label sequence, calculate the conditional entropy of the semantic label sequence to construct an information entropy weight matrix, multiply the information entropy weight matrix by the co-occurrence frequency matrix to obtain a semantic association matrix, use semantic categories as nodes and the semantic association matrix as edge weights to construct a graph structure, and use a multi-head attention mechanism and skip connections to fuse node features to obtain a scene semantic vector, specifically including:

[0099] Use a deep convolutional neural network to perform semantic segmentation on the scene image sequence to obtain a semantic label sequence, calculate the forward mapping matrix and the backward mapping matrix according to the pixel position correspondence between adjacent frames, use the forward mapping matrix and the backward mapping matrix to align the semantic prediction result of the current frame with adjacent frames, calculate the difference of the aligned prediction results to construct a temporal consistency loss function, and optimize the semantic segmentation network parameters by minimizing the temporal consistency loss function to obtain a semantic label sequence with temporal consistency;

[0100] Extract semantic categories for each frame of semantic labels in the semantic label sequence, count the co-occurrence frequencies of each pair of semantic categories in the same scene to construct a co-occurrence frequency matrix, calculate the joint probability and marginal probability of the semantic category pairs in the co-occurrence frequency matrix to obtain the conditional entropy, use the conditional entropy to construct an information entropy weight matrix, multiply the information entropy weight matrix by the co-occurrence frequency matrix and perform normalization processing to obtain a semantic association matrix;

[0101] Construct the semantic categories and the semantic association matrix as the node set and edge weights of the graph structure respectively, extract the neighborhood node features for each node in the graph structure, calculate the attention coefficients between nodes using the multi-head attention mechanism, weight-aggregate the neighborhood node features according to the attention coefficients to obtain the updated features of the nodes, fuse the multi-layer node updated features through skip connections, and splice the fused node features to obtain a scene semantic vector containing object category information and semantic association patterns.

[0102] Exemplarily, when a deep convolutional neural network processes a scene image sequence, it is necessary to ensure the temporal stability of the semantic prediction results. Traditional semantic segmentation methods usually process each frame of image independently, ignoring the temporal correlation between adjacent frames, resulting in the problem that the semantic labels of the same object may be inconsistent in adjacent frames. To solve this problem, it is necessary to construct a temporal consistency loss and optimize the parameters of the semantic segmentation network by minimizing the loss, so that the semantic prediction results are temporally stable.

[0103] In an image sequence, first, a deep convolutional neural network is used to perform semantic segmentation on each frame of the image to obtain a preliminary semantic label sequence. To establish the correspondence between adjacent frames, it is necessary to calculate the forward mapping matrix and the backward mapping matrix based on the motion information of pixel positions. The role of these two matrices is to describe the pixel mapping relationship between the current frame and the adjacent frame. For example, in a video sequence, slight jitter of the camera or movement of objects in the scene will cause pixel displacement between adjacent frames. The forward mapping matrix can predict the position of pixels in the current frame in the next frame, while the backward mapping matrix can predict the position of pixels in the previous frame in the current frame.

[0104] Using the forward and backward mapping matrices, the semantic prediction results of the current frame can be aligned with the prediction results of the adjacent frame, and the difference between the aligned prediction results can be calculated. The magnitude of the difference can measure the semantic consistency between adjacent frames. If the same semantic label exists at the same position in adjacent frames, it indicates that the network has good temporal consistency. On the contrary, if there is a large deviation in the semantic labels between adjacent frames, it indicates that the temporal consistency of the network is poor. By constructing a temporal consistency loss function and minimizing this loss during training, the parameters of the network can be optimized to maintain high stability in the time dimension.

[0105] For example, in an autonomous driving scenario, assume a car is driving on a street and there is a red car in front. In the first frame, the car is correctly segmented into the "vehicle" category, while in the next frame, due to changes in lighting or motion blur, some areas of the car are misclassified as "road". At this time, the forward mapping matrix and the backward mapping matrix can help align the vehicle regions in the two frames and calculate the degree of label inconsistency. By optimizing the temporal consistency loss, this cross-frame label jitter can be reduced, and the stability of semantic segmentation can be improved.

[0106] After obtaining a temporally consistent semantic label sequence, it is necessary to further analyze the co-occurrence relationship between different semantic categories in the scene to construct a semantic association matrix. The semantic association matrix is used to describe the co-occurrence pattern of objects of different categories in the same scene, providing support for subsequent scene understanding.

[0107] It is necessary to extract the semantic categories of each frame of the image from the semantic label sequence and count the co-occurrence frequencies between different categories. For example, in an urban road scene, there may be a co-occurrence of "vehicle" and "road", or a co-occurrence of "pedestrian" and "sidewalk". By counting the co-occurrence times of each pair of semantic categories in multiple scenes, a co-occurrence frequency matrix can be constructed. Each element of the co-occurrence frequency matrix represents the frequency of two semantic categories co-occurring in the same frame of the image. For example, "vehicle-road" may have a high co-occurrence frequency, while "vehicle-sky" may have a low co-occurrence frequency.

[0108] To further measure the association strength between semantic categories, conditional entropy needs to be calculated. Conditional entropy can be used to measure the predictive ability of the occurrence of one category for another. For example, if in most cases, "road" in a scene will definitely be accompanied by "vehicle", then the conditional entropy of "road" for "vehicle" is low, indicating that "road" has a strong predictive ability for "vehicle". Conversely, if the co-occurrence relationship between two categories is relatively random, the conditional entropy is high.

[0109] Based on conditional entropy, an information entropy weight matrix can be constructed. The role of the information entropy weight matrix is to weight the co-occurrence frequency matrix, so that categories with lower information entropy (i.e., stronger associations) occupy higher weights in the semantic association matrix. By multiplying the information entropy weight matrix with the co-occurrence frequency matrix and performing normalization processing, the final semantic association matrix can be obtained, which can more accurately reflect the true associations between semantic categories.

[0110] For example, in an intelligent security monitoring scenario, if a camera monitors an area that may include two categories, "table" and "chair". Since tables and chairs usually appear together in most cases, their co-occurrence frequency is high, and at the same time, the conditional entropy is low. Therefore, in the semantic association matrix, the weight of the pair of categories "table - chair" is high. While "table - window" may only appear occasionally, so the weight is low.

[0111] Based on semantic categories and the semantic association matrix, a graph structure can be constructed to further extract the high-level semantic representation of the scene. In this graph structure, semantic categories are regarded as the nodes of the graph, and the weights of the semantic association matrix are regarded as the weights of the edges. To learn richer node features, it is necessary to extract the feature information of the neighborhood of each node and fuse it through the multi-head attention mechanism.

[0112] In this process, the features of each node can be updated through the features of its neighboring nodes. For example, in a traffic scene, the features of the "road" node can be weighted and aggregated through the features of neighboring nodes such as "vehicle", "pedestrian", and "traffic signal". To better capture the importance of different neighboring nodes, the multi-head attention mechanism can be used to calculate the attention coefficients between nodes. The multi-head attention mechanism can calculate the attention weights in multiple different subspaces respectively, making the final feature representation more comprehensive. For example, in an autonomous driving system, the attention of "road" to "vehicle" may be high, while the attention to "building" may be low. Therefore, the attention mechanism can dynamically adjust the feature weights of different categories.

[0113] After calculating the updated node features, feature fusion needs to be performed through skip connections. The role of skip connections is to prevent features from being overly smoothed during hierarchical transmission, so that high-level features still retain the original fine-grained information. For example, in video surveillance analysis, the features of the "person" category may lose details due to excessive information aggregation after multiple layers of propagation, while skip connections can maintain the uniqueness of the "person" category.

[0114] The features of all nodes are concatenated to form the overall semantic vector of the scene. This semantic vector not only contains information of individual categories but also contains the association patterns between categories, and can be used for subsequent tasks such as scene classification and object detection.

[0115] Figure 2 This is the analysis chart of the application effect of the scene semantic vector in the embodiments of the present invention in downstream tasks. This chart comprehensively shows the performance comparison of the scene semantic vector generated by this technical solution with classic methods such as Word2Vec and BERT in four types of downstream application tasks. From the data in the chart, it can be seen that this technical solution shows significant performance advantages in all tasks. In the scene classification task, the average accuracy (mAP) of this technical solution reaches 0.94, while those of BERT and Word2Vec are 0.87 and 0.79 respectively, with relative improvements of 8.0% and 19.0% respectively; in the action recognition task, the mAP of this technical solution is 0.92, while those of BERT and Word2Vec are 0.85 and 0.76 respectively, and the advantages are still obvious; in the complex anomaly detection task, this technical solution maintains a high accuracy of 0.90, while those of BERT and Word2Vec drop to 0.83 and 0.74 respectively; in the image retrieval task, the mAP of this technical solution is 0.88, still significantly better than 0.81 of BERT and 0.72 of Word2Vec. These results fully prove that through innovative designs such as semantic segmentation under temporal consistency constraints, semantic association matrix optimized by information entropy weights, multi-head attention mechanism, and skip connection feature fusion, the scene semantic vector generated by this technical solution has richer semantic expression ability and stronger discriminability. Especially in the two tasks of scene classification and action recognition, which are closely related to scene understanding, this technical solution performs particularly prominently, which verifies that this solution can effectively capture the core semantic structure and key semantic relationships in the scene. At the same time, in the two tasks of anomaly detection and image retrieval, which have relatively high requirements for the generalization ability of semantic vectors, this solution also maintains high performance, reflecting that the generated scene semantic vector has good generalization and robustness and can adapt to various complex downstream application scenarios.

[0116] When performing semantic analysis on a sequence of scene images, the prior art usually uses a deep convolutional neural network to segment individual frames, lacking temporal information constraints, resulting in unstable semantic labels between adjacent frames. In addition, the correlation between semantic categories is usually based on simple co-occurrence statistics, without fully considering the dynamic characteristics of category relationships, affecting the accuracy and generalization ability of scene semantic representation. In this application, by introducing a temporal consistency loss and optimizing the semantic segmentation network, the semantic label prediction results between adjacent frames are made more stable, effectively reducing semantic drift caused by inter-frame noise and improving the segmentation accuracy. At the same time, by using conditional entropy to construct an information entropy weight matrix, the semantic correlation matrix not only depends on the category co-occurrence frequency but also can reflect the stability and information content between categories, enhancing the rationality of semantic relationship modeling. In addition, through the feature fusion method of the multi-head attention mechanism and skip connections, the semantic nodes in the graph structure are updated, enabling the scene semantic vector to integrate local and global information and improving the ability to understand complex scenes. Compared with the prior art, this application has made improvements in semantic consistency, category association modeling, and feature fusion methods, avoiding the limitations of single-frame analysis, making the final obtained scene semantic vector more stable and accurate, and being able to better express the semantic association pattern of objects in the scene, improving the adaptability in complex scenes.

[0117] In an alternative embodiment, the scene spatial feature vector and the scene semantic vector are input into a variational autoencoder. Through the encoding process of the variational autoencoder, a scene feature distribution is obtained. A target feature vector is sampled from the scene feature distribution. Based on the target feature vector, an attention score is calculated, and the target feature vector is weight-updated according to the attention score to generate a weighted target description vector, including:

[0118] The scene spatial feature vector and the scene semantic vector are input into the feature decomposition unit of the variational autoencoder. Through Bayesian inference, prior probability distribution parameters are obtained. Based on the prior probability distribution parameters, a conditional prior distribution is constructed. The shared information and independent information in the scene spatial feature vector and the scene semantic vector are respectively extracted according to the conditional prior distribution to obtain a feature sharing subspace and a feature independent subspace. The weights of the feature sharing subspace and the feature independent subspace are calculated based on the mutual information maximization criterion. According to the weights, the feature subspaces are weighted and combined to obtain a scene feature distribution;

[0119] The scene feature distribution is respectively input into multiple probability distribution branches of the variational autoencoder. Distribution parameters are calculated for each branch based on the statistical characteristics of the scene feature distribution. The information entropy value of each branch is calculated using the distribution parameters. A target feature vector is sampled from the scene feature distribution according to the distribution parameters of the branch with the minimum information entropy value;

[0120] Construct each dimension of the target feature vector as a node in a conditional random field, calculate the feature similarity between adjacent nodes to obtain a pairwise potential function, calculate the combined similarity of all nodes within a node group to obtain a combined potential function, construct the pairwise potential function and the combined potential function into an energy function, and minimize the energy function through an iterative optimization method to obtain attention scores;

[0121] Multiply the attention scores with each dimension of the target feature vector to obtain a feature vector with updated weights, calculate the statistical value of the feature vector with updated weights to obtain a gating threshold, and retain the feature dimensions in the feature vector with updated weights that are greater than the gating threshold to obtain a weighted target description vector.

[0122] Input the scene space feature vector and the scene semantic vector into a variational autoencoder to achieve feature decomposition and optimization. The scene space feature vector includes the geometric structure, texture information, and depth information of the scene, which are usually extracted by a deep convolutional neural network or a point cloud processing algorithm and are mainly used to depict the physical form of the scene. The scene semantic vector is used to represent the object categories and their relationships in the scene, which are generally extracted by a pre-trained semantic segmentation network or an object detection network and contain category labels as well as information such as position and size in the scene. These two types of features are usually processed independently in traditional methods, while through the variational autoencoder, the correlation between the two can be established in the latent variable space, and shared patterns and independent patterns can be extracted, thus obtaining a more complete scene representation.

[0123] The feature decomposition unit uses the Bayesian inference method to calculate the prior probability distribution parameters of the input features. The prior probability distribution is used to constrain the distribution form of the features in the latent space to make it more in line with the actual physical laws. Based on the prior probability distribution, a conditional prior distribution is constructed, that is, given a certain scene feature, the distribution range of another type of feature is inferred. Subsequently, the scene space feature vector and the scene semantic vector are decomposed respectively to extract the shared information and independent information. The shared information contains the common patterns of the two, such as the combination relationship between the spatial structure and the object category, while the independent information corresponds to the geometric details of the scene space or individual semantic category information respectively. The proportion of the shared information and the independent information is calculated through the mutual information maximization criterion, which is used to evaluate the information complementarity degree between two variables, and the features are weighted and combined according to the calculated weights to form a scene feature distribution, so that the final feature representation is both comprehensive and has strong discrimination ability.

[0124] After the scene feature distribution is processed, it is input into multiple probability distribution branches of the variational autoencoder. Each branch corresponds to different scene feature patterns, such as different categories of scene types, object relationships at different scales, etc. To ensure that the finally sampled features can stably represent the target information, it is necessary to calculate the statistical characteristics of each branch, that is, estimate the distribution parameters of each branch to measure whether the scene features it represents are stable. For each branch, the information entropy value is calculated. Information entropy measures the uncertainty of the distribution. A lower information entropy indicates a higher distribution stability of the branch, which is suitable as the source of the target feature vector. Therefore, sampling is performed from the distribution parameters of the branch with the smallest information entropy value to obtain the final target feature vector. This process ensures the stability and representativeness of the target features.

[0125] The optimization of the target feature vector is achieved through conditional random field modeling. Conditional random field is a probabilistic graphical model for modeling sequence data or structured data, where each feature dimension is mapped to a node in the conditional random field. The feature similarity between adjacent nodes is calculated, and a pairwise potential function is constructed. The pairwise potential function is used to describe the local correlation between features. For example, features that are spatially close usually have similar semantic information. In addition, the combined similarity of all nodes within a group of nodes is calculated to obtain a combined potential function, which is used to characterize the global pattern, such as the object combination relationship in the entire scene. Based on these two potential functions, an energy function is constructed, and the minimization process of the energy function is completed through an iterative optimization algorithm, and finally the attention score is calculated. The attention score is used to measure the importance of each feature dimension in the target feature vector. A higher attention score indicates that this dimension is particularly crucial in the current scene, while a lower attention score indicates that this dimension may be redundant information or background noise.

[0126] The attention score is used to update the weights of the target feature vector to enhance the influence of important features and weaken the role of unimportant features. The specific implementation method is to multiply the attention score with each dimension of the target feature vector one by one, so that the dimensions with higher weights are enhanced, while the dimensions with lower weights are suppressed. Subsequently, statistical calculations are performed on the feature vector after weight update to determine the gating threshold. The gating threshold is used to screen the most representative feature dimensions, and the feature dimensions below the threshold are discarded, and only the feature dimensions exceeding the threshold are retained. The finally obtained weighted target description vector can efficiently extract key features and provide more accurate semantic information in subsequent tasks.

[0127] In this embodiment, through the feature decomposition process of the variational autoencoder, spatial information and semantic information can be effectively fused, making the scene representation more comprehensive and avoiding the problem of feature redundancy that may exist in traditional methods. By screening the optimal distribution through information entropy, the stability of the target feature vector is improved, enabling it to maintain strong generalization ability in different environments. The target features are modeled using conditional random fields and optimized through an attention mechanism, making the feature representation more in line with the task requirements and improving the reliability and robustness of the final target representation. The finally obtained target description vector is not only compact but also has strong discriminative power, capable of adapting to different application requirements.

[0128] In an alternative embodiment, each dimension of the target feature vector is constructed as a node in a conditional random field. The pairwise potential function is obtained by calculating the feature similarity between adjacent nodes, and the combined potential function is obtained by calculating the combined similarity of all nodes within a node group. Constructing the pairwise potential function and the combined potential function into an energy function includes:

[0129] The eigenvalue of each dimension of the target feature vector is constructed as a node in a conditional random field. The correlation coefficient between nodes is calculated, and a connection threshold is set based on the correlation coefficient. Node pairs with a correlation coefficient greater than the connection threshold are established with a connection relationship to obtain an adaptive node connection graph, where the node connection graph includes a node set and an edge set;

[0130] The Euclidean distance between the eigenvalues of adjacent nodes in the node connection graph is calculated, and the pairwise potential function is obtained by calculating the feature similarity between adjacent nodes based on the Euclidean distance;

[0131] The mean value of the eigenvalues of all nodes within a node group in the node connection graph is calculated, the distance between each node eigenvalue and the mean value is calculated based on the mean value, and the combined potential function of the node group is obtained by calculating the combined similarity according to the distance;

[0132] The logarithm value of the pairwise potential function is calculated to obtain the first energy term, the logarithm value of the combined potential function is calculated to obtain the second energy term, and the energy function is obtained by summing the first energy term and the second energy term after respectively assigning weight coefficients to them.

[0133] Exemplarily, first, each dimension of the target feature vector is constructed as a node in a conditional random field. Each node represents the value of the target feature vector in a specific dimension. A conditional random field is a probabilistic graphical model used to model complex relationships and can effectively capture the mutual influence between features. Calculate the correlation coefficients between these nodes. The correlation coefficient is used to measure the linear relationship between two feature dimensions. When the correlation coefficient between two feature dimensions is high, it indicates that they may share similar information. Based on the calculated correlation coefficients, set a connection threshold. When the correlation coefficient between two feature dimensions exceeds this threshold, it is considered that there is a strong correlation between them, and a connection relationship is established. The establishment of the connection relationship forms an adaptive node connection graph, which consists of a node set and an edge set. The node set contains all the target feature dimensions, while the edge set represents the connections between those node pairs with relatively high correlations and meeting the threshold conditions.

[0134] On the constructed adaptive node connection graph, calculate the Euclidean distance between the corresponding feature values of adjacent nodes. The Euclidean distance measures the spatial difference between two feature values. A smaller Euclidean distance indicates a higher similarity between the feature values of two nodes. Calculate the feature similarity between adjacent nodes based on the Euclidean distance. The feature similarity is used to describe the consistency of adjacent features in the target feature space. The higher the feature similarity, the more representative the feature is in the entire target vector. This calculation process obtains a pairwise potential function, which depicts the similarity relationship between node pairs and ensures that the conditional random field model can effectively focus on the correlation between features.

[0135] Further calculate the mean value of the feature values of all nodes within the node group. The mean value is used to characterize the central tendency of the node group and reflects the overall feature performance of the group of nodes. Based on the calculated mean value, calculate the distance between each node's feature value and this mean value. The distance is used to measure the degree of deviation of a single feature from the overall feature. Calculate the combined similarity of the node group according to this distance. The combined similarity is used to depict the stability of the overall feature. If the feature value of a certain node deviates greatly from the mean value, it indicates that this feature may be an outlier or affected by external noise, and the combined similarity is low; on the contrary, if the feature values of all nodes are relatively close to the mean value, it indicates that the group of features is relatively stable, and the combined similarity is high. The calculation of the combined similarity obtains a combined potential function, which is used to describe the overall consistency of the target feature vector and ensures that the target feature has a stable semantic expression ability in the global range.

[0136] Calculate the logarithm of the pairwise potential function to obtain the first energy term. The logarithm is used to stabilize the calculation results, making the calculation more robust and reducing the influence of extreme values. Similarly, calculate the logarithm of the combined potential function to obtain the second energy term. Assign weight coefficients to the first energy term and the second energy term respectively. The weight coefficients are used to control the contribution degrees of the pairwise potential function and the combined potential function in the final energy function, ensuring the balance between local feature similarity and global feature stability. Sum the weighted first energy term and the second energy term to obtain the final energy function. This energy function is used to measure the overall quality of the target feature vector. In the subsequent optimization process, minimize the energy function through an iterative solution method to ensure the optimal representation of the target feature vector.

[0137] In this embodiment, by constructing an adaptive node connection graph, relatively highly correlated feature pairs are adaptively selected for connection, making the feature relationship modeling more flexible and not restricted by a fixed topological structure. Through the calculation of the pairwise potential function, adjacent features are made to be consistent in the target feature vector, thereby enhancing the separability of the feature space. Through the calculation of the combined potential function, the stability of the overall features is enhanced, ensuring that the final target feature vector has a stronger semantic expression ability. In the subsequent feature selection or task application process, the optimized target features can effectively improve the accuracy of classification, recognition, or matching, while reducing feature redundancy and improving the calculation efficiency, making the feature optimization more efficient and stable.

[0138] In an alternative embodiment, a probabilistic graph inference network is constructed. The scene space feature vector is mapped to a position probability distribution, and the scene semantic vector is mapped to a semantic probability distribution. Based on the position probability distribution and the semantic probability distribution, a target generation constraint is constructed. The target generation constraint and the target description vector are combined and input into the generation network. The generation network outputs the target contour features according to the target generation constraint, including:

[0139] Construct a two-stream probabilistic graph network to extract position and semantic features respectively. Form a target generation constraint by calculating the divergence between the position probability distribution and the uniform distribution, and the divergence between the semantic probability distribution and a preset distribution. Concatenate the target generation constraint and the target description vector in the feature dimension, and output the target contour features through the generation network, specifically including:

[0140] Construct a probabilistic graph inference network. The probabilistic graph inference network includes a position feature stream and a semantic feature stream. The position feature stream processes the scene space feature vector through multiple layers of graph convolution to obtain a position feature map. The semantic feature stream calculates attention weights to weight the scene semantic vector to obtain semantic features;

[0141] Perform a normalization operation on the position feature map to obtain a position probability distribution, and perform a non-linear transformation on the semantic features to obtain a semantic probability distribution;

[0142] Calculate the divergence value between the position probability distribution and the uniform distribution to obtain a position constraint term, calculate the divergence value between the semantic probability distribution and the preset semantic distribution to obtain a semantic constraint term, and perform weighted summation on the position constraint term and the semantic constraint term to obtain a target generation constraint;

[0143] Concatenate the target generation constraint and the target description vector in the feature dimension, and input the concatenated features into a generation network. The generation network magnifies the spatial scale of the input features through an upsampling convolutional layer and outputs target contour features.

[0144] Exemplarily, first obtain a scene space feature vector and a scene semantic vector. The scene space feature vector represents the spatial layout information in the scene, with a dimension of C×H×W, where C is the number of feature channels, and H and W are the height and width of the feature map respectively. For example, for an indoor scene, C can be set to 64, and H and W can be set to 32. The scene semantic vector represents the semantic information in the scene, with a dimension of N×D, where N is the number of semantic entities in the scene, and D is the semantic feature dimension. For example, for a scene containing 5 objects, N can be set to 5, and D can be set to 128.

[0145] Next, construct a probabilistic graph inference network, which includes two branches: a position feature stream and a semantic feature stream. The position feature stream processes the scene space feature vector through multiple layers of graph convolution to obtain a position feature map. Specifically, first convert the scene space feature vector into a graph structure, where each pixel point is used as a node in the graph, and there are edge connections between adjacent pixel points. Then, apply a three-layer graph convolutional network to process this graph structure. The input channel number of the first layer of graph convolution is C, and the output channel number is 128; the input channel number of the second layer of graph convolution is 128, and the output channel number is 256; the input channel number of the third layer of graph convolution is 256, and the output channel number is 128. After each layer of graph convolution, a batch normalization layer and a ReLU activation function are connected. After three layers of graph convolution processing, a position feature map with a dimension of 128×H×W is obtained.

[0146] The semantic feature stream weights the scene semantic vector by calculating attention weights to obtain semantic features. Specifically, first map the scene semantic vector to a query vector and a key vector through a fully connected layer, both with a dimension of N×64. Then, calculate the dot product between the query vector and the key vector, and normalize it through the Softmax function to obtain an attention weight matrix with a dimension of N×N. Finally, multiply the attention weight matrix by the scene semantic vector to obtain the weighted semantic features with a dimension of N×D. Then, aggregate the semantic features into a D-dimensional vector through a pooling operation.

[0147] Perform a normalization operation on the location feature map to obtain a location probability distribution. Specifically, perform Softmax normalization on the location feature map so that the sum of the probability values at all pixel points is 1, obtaining a location probability distribution with a dimension of H×W. Perform a non-linear transformation on the semantic feature to obtain a semantic probability distribution. Specifically, map the D-dimensional semantic feature to a K-dimensional vector (K is the predefined number of semantic categories, e.g., 10) through a fully connected layer, and then normalize it through the Softmax function to obtain a semantic probability distribution with a dimension of K.

[0148] Calculate the divergence value between the location probability distribution and the uniform distribution to obtain a location constraint term. The uniform distribution U_pos indicates that each location has an equal probability, that is, the probability value of each pixel point is 1 / (H×W). Use KL divergence to calculate the difference between the location probability distribution and the uniform distribution to obtain the location constraint term. Calculate the divergence value between the semantic probability distribution and the preset semantic distribution to obtain a semantic constraint term. The preset semantic distribution is set according to prior knowledge. For example, for generating furniture objects in an indoor scene, the probability of the "table" category can be set to 0.3, the probability of the "chair" category can be set to 0.4, and the probability of other categories can be set to 0.3. Use KL divergence to calculate the difference between the semantic probability distribution and the preset semantic distribution to obtain the semantic constraint term.

[0149] Perform a weighted sum of the location constraint term and the semantic constraint term to obtain a target generation constraint. Concatenate the target generation constraint and the target description vector in the feature dimension. The target description vector represents the text description of the target to be generated, and converts the text description into a vector representation through a text encoder (such as BERT) with a dimension of M (e.g., 256). Expand the target generation constraint L with a dimension of 1 to the same dimension as the target description vector, and then concatenate it with the target description vector in the feature dimension to obtain a feature vector with a dimension of M+1.

[0150] Input the concatenated feature into the generation network. The generation network magnifies the spatial scale of the input feature through upsampling convolutional layers and outputs target contour features. Specifically, the generation network contains four upsampling convolutional layers. The input dimension of the first upsampling convolution is M+1, the output number of channels is 256, and the upsampling scale is 2; the input number of channels of the second upsampling convolution is 256, the output number of channels is 128, and the upsampling scale is 2; the input number of channels of the third upsampling convolution is 128, the output number of channels is 64, and the upsampling scale is 2; the input number of channels of the fourth upsampling convolution is 64, the output number of channels is 1, and the upsampling scale is 2. After each upsampling convolution, a batch normalization layer and a ReLU activation function are connected, and the Sigmoid activation function is used in the last layer. The finally output target contour feature has a dimension of 1×(H×8)×(W×8), representing the contour probability map of the target.

[0151] In practical applications, thresholding can be used to convert the target contour probability map into a binary image, and the threshold can be set to 0.5. Pixel points above the threshold are regarded as inside the target, and pixel points below the threshold are regarded as the background, thus obtaining the contour of the target.

[0152] Figure 3 This is the probability distribution divergence analysis diagram of the embodiment of the present invention. This diagram compares the probability distribution divergence performances of various methods under different scene types. In the "indoor scene" type, the divergence value of the present technical solution is 0.31, which is 34.0% and 27.9% lower than 0.47 of the VAE method and 0.43 of the Flow-based method respectively; in the "outdoor open scene" type, the divergence value of the present technical solution is 0.37, which is 28.8% and 22.9% lower than 0.52 of the VAE method and 0.48 of the Flow-based method respectively; in the "complex interaction scene" type, the divergence value of the present technical solution is 0.43, lower than 0.58 of the VAE method and 0.54 of the Flow-based method, with the reduction amplitudes being 25.9% and 20.4% respectively; in the "low light scene" type, the divergence value of the present technical solution is 0.45, which is 30.8% and 26.2% lower than 0.65 of the VAE method and 0.61 of the Flow-based method respectively; in the "dynamic scene" type, the divergence value of the present technical solution is 0.48, while the VAE method and the Flow-based method are 0.69 and 0.63 respectively, with the reduction amplitude reaching 30.4% and 23.8%. Overall, the present technical solution shows the lowest probability distribution divergence value in various scenes, with an average reduction of the divergence value by 29.3%, proving its stability and adaptability in processing scenes of different complexities, especially being particularly prominent in indoor scenes and low light scenes.

[0153] In the prior art, target generation usually adopts a single generative adversarial network structure, directly generating a target image from random noise or conditional input, lacking the ability to model scene space constraints and semantic constraints. For example, traditional GAN-based methods are difficult to ensure the consistency between the generated target and the scene environment, and are prone to generating generation results with unreasonable spatial positions and mismatched semantic attributes. The target generation method based on the probability graph inference network proposed by the present invention extracts position and semantic features respectively by constructing a two-stream probability graph network, clearly models position and semantic constraints, and enables the generated target to meet the requirements of the spatial layout and semantic attributes of the scene. The position constraint ensures that the position of the generated target conforms to the scene space distribution, and the semantic constraint ensures that the semantic attributes of the generated target are coordinated with the scene semantics. In addition, the present invention combines the target generation constraint with the target description vector, enabling the generation process to simultaneously meet the user-specified target description and scene constraint conditions.

[0154] In an alternative embodiment, constructing a regional attention map based on the target contour features and combining the regional attention map with the target description vector to generate the final target generation result includes:

[0155] Using a sliding window to simultaneously extract the maximum response and average response features to construct spatial attention, and combining it with channel attention to form a regional attention mechanism to obtain a regional attention map. Using residual connections to enhance feature representation, performing multi-scale fusion on the regional attention map and the target description features, and then decoding to generate the target result, specifically including:

[0156] Scanning the target contour features in a sliding window manner, calculating the statistical values of the features within the window based on each sliding window position, including calculating the maximum value to obtain the maximum response feature map, calculating the average value to obtain the average response feature map, performing convolution operations on the maximum response feature map and the average response feature map respectively, adding them together, and obtaining the spatial attention map through normalization processing;

[0157] Performing average pooling on the target contour features in the spatial dimension and obtaining the channel attention map through a fully connected layer, multiplying the spatial attention map, the channel attention map, and the target contour features to obtain the regional attention map;

[0158] Performing convolution processing on the regional attention map to obtain local features, and performing residual connection on the local features and the regional attention map to obtain enhanced features;

[0159] Upsampling and expanding the target description vector, splicing the expanded features with the enhanced features in the feature dimension, and performing multi-scale processing on the spliced features through convolutional layers with different kernel sizes;

[0160] Inputting the multi-scale processed features into a decoder, and the decoder decodes the features through a transposed convolutional layer and skip connections to generate the target result.

[0161] Exemplarily, a sliding window method is used to scan the target contour features. A sliding window is a way of continuously moving within a specific area and extracting local features. During this process, the window covers different areas of the target contour. For each window position, the statistical values of all features within it are calculated. Specifically, the maximum feature value within the window constitutes the maximum response feature map, which reflects the most significant features of this area. In addition, the average feature value within the window constitutes the average response feature map, representing the overall information of this area. The maximum response feature map and the average response feature map are respectively processed through convolution operations to extract more detailed local information, and then they are added together and adjusted through a normalization step to obtain the spatial attention map. The spatial attention map reflects the importance of different areas in the image, helps to highlight key areas, and improves the model's perception ability of the target.

[0162] Average pooling is performed on the target contour features in the spatial dimension. Average pooling is a downsampling operation used to reduce the spatial dimension of the feature map while retaining the most important information. The pooled features are transformed into a channel attention map through a fully connected layer, and the channel attention map represents the significance degree of each feature channel. After the channel attention map is combined with the spatial attention map, a region attention map is generated through element-wise multiplication. The region attention map synthesizes the information in the spatial and channel dimensions and reflects the importance distribution of the target contour features in the space and channels.

[0163] The region attention map is further processed through a convolution operation to extract local features. These local features represent the fine-grained information of the target features and can help identify smaller and crucial details. Subsequently, a residual connection is made between the local features and the region attention map. Residual connection is a skip connection technique that makes the flow of feature information smoother by adding the input features and the features processed by convolution, effectively alleviating the vanishing gradient problem in deep neural networks and thus enhancing the feature representation ability.

[0164] The target description vector is a high-level description of the target, and it is upsampled to a higher resolution. The upsampling operation usually interpolates the low-resolution features to a high resolution to restore more spatial details. The expanded target description features are concatenated with the enhanced region features, and the concatenation operation combines the two feature vectors in the feature dimension to form a richer feature representation. Next, the concatenated features are processed through multiple convolutional layers with different convolutional kernel sizes, aiming to extract multi-scale features. This multi-scale processing can effectively capture the details of targets of different sizes, enabling the model to better adapt to the diversity and complexity of targets.

[0165] The features processed by multi-scale are input into the decoder for decoding. The decoder upsamples the features through a transposed convolution layer to restore the original image or generate the target result. The transposed convolution layer is similar to the convolution layer, but its purpose is to restore the low-resolution feature map to a high-resolution image. Skip connections are used to combine the early-stage features and the late-stage features in the decoding process to preserve the high-frequency details of the image and ensure the fineness of the target generation result. The output of the decoder is the final target generation result, which can be the bounding box of the target, the segmentation result, or other forms of images.

[0166] Figure 4 It is a schematic diagram of the decoder structure according to an embodiment of the present invention, as Figure 4As shown in the figure, the decoder structure and its decoding process are detailed. The decoder receives the features (64×64×256) after multi-scale processing as input, and gradually restores the spatial details of the target through a four-level transposed convolution and skip connection structure. The first-level transposed convolution upsamples the features to 128×128×128 and makes a skip connection with the features of the third layer of the encoder (128×128×128), with the detail fidelity reaching 95.2%; the second-level transposed convolution further upsamples to 256×256×64 and combines with the features of the second layer of the encoder (256×256×64), and the edge localization accuracy is improved to 0.86 pixels; the third-level transposed convolution outputs a feature map of 512×512×32, which is fused with the features of the first layer of the encoder (512×512×32), and the texture restoration degree reaches 93.8%; the final-level transposed convolution generates high-resolution features of 1024×1024×16, which are mapped to the channels of the number of target categories through 1×1 convolution to complete the target generation. Compared with the ordinary transposed convolution decoder, the detail fidelity of this technical solution is increased by 19.1% for targets with a complexity level of 7; it is increased by 8.2% compared with the SegNet decoder; and it is increased by 6.6% compared with the DeepLab decoder. Especially when dealing with high-difficulty targets with a complexity level of 9-10, this solution still maintains a detail fidelity of 87-89%, which is mainly due to the combination of the low-level feature information provided by the skip connection and the efficient upsampling ability of the transposed convolution layer.

[0167] In this embodiment, by extracting the maximum response and average response features through a sliding window and combining with the spatial attention mechanism, the important regions in the image can be accurately focused, and the model's perception ability for key features can be strengthened. The combination of spatial and channel attention provides a more detailed attention mechanism, enabling the features in different regions and different channels to be effectively weighted, thereby optimizing the representation of the target. The residual connection and multi-scale processing further enhance the feature expression ability, ensuring that the details of local features are not lost, and at the same time being able to adapt to the diversity and complexity of the target. Through transposed convolution and skip connection, the generated results are clearer and more detailed, adapting to the generation requirements of different resolutions. Therefore, this technical solution not only improves the accuracy of target recognition, but also can generate high-quality target results in complex environments, with strong robustness and broad application potential.

[0168] Figure 5 This is a schematic structural diagram of the adaptive scene analysis and target generation system based on deep learning according to an embodiment of the present invention. As Figure 5 shown, the system includes:

[0169] The first unit is used to extract the depth information of each frame of the scene image sequence, construct three-dimensional point cloud data based on the depth information, calculate the distance matrix and direction matrix between objects in the scene based on the three-dimensional point cloud data, and combine the distance matrix and direction matrix to obtain the scene space feature vector;

[0170] The second unit is used to perform semantic segmentation on the scene image sequence to obtain a semantic label map, construct a semantic association matrix according to the co-occurrence frequency of the labels in the semantic label map, and calculate the scene semantic vector based on the semantic association matrix;

[0171] The third unit is used to input the scene space feature vector and the scene semantic vector into a variational autoencoder, obtain the scene feature distribution through the encoding process of the variational autoencoder, sample a target feature vector from the scene feature distribution, calculate the attention score based on the target feature vector, update the weight of the target feature vector according to the attention score, and generate a weighted target description vector;

[0172] The fourth unit constructs a probabilistic graph inference network, maps the scene space feature vector to a position probability distribution, maps the scene semantic vector to a semantic probability distribution, constructs a target generation constraint based on the position probability distribution and the semantic probability distribution, and combines the target generation constraint and the target description vector and inputs them into a generation network, and the generation network outputs the target contour feature according to the target generation constraint;

[0173] The fifth unit constructs a regional attention map based on the target contour feature, and combines the regional attention map with the target description vector to generate the final target generation result.

[0174] In the third aspect of the embodiments of the present invention,

[0175] There is provided an electronic device, including:

[0176] A processor;

[0177] A memory for storing instructions executable by the processor;

[0178] Wherein, the processor is configured to call the instructions stored in the memory to execute the method described above.

[0179] In the fourth aspect of the embodiments of the present invention,

[0180] There is provided a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described above is implemented.

[0181] The present invention can be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium, on which computer-readable program instructions for executing various aspects of the present invention are uploaded.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive scene analysis and target generation method based on deep learning, characterized in that, Including: Extract the depth information of each frame in the scene image sequence, construct 3D point cloud data based on the depth information, calculate the distance matrix and direction matrix between objects in the scene based on the 3D point cloud data, and combine the distance matrix and direction matrix to obtain the scene space feature vector; Perform semantic segmentation on the scene image sequence to obtain a semantic label map, construct a semantic association matrix according to the co-occurrence frequency of the labels in the semantic label map, and calculate the scene semantic vector based on the semantic association matrix; Input the scene space feature vector and the scene semantic vector into a variational autoencoder, obtain the scene feature distribution through the encoding process of the variational autoencoder, sample from the scene feature distribution to obtain the target feature vector, calculate the attention score based on the target feature vector, update the weight of the target feature vector according to the attention score, and generate a weighted target description vector; Construct a probabilistic graph inference network, map the scene space feature vector to a position probability distribution, map the scene semantic vector to a semantic probability distribution, construct a target generation constraint based on the position probability distribution and the semantic probability distribution, and combine the target generation constraint and the target description vector and input them into a generation network. The generation network outputs the target contour feature according to the target generation constraint; Construct a regional attention map based on the target contour feature, and combine the regional attention map with the target description vector to generate the final target generation result.

2. The method according to claim 1, wherein Extract the depth information of each frame in the scene image sequence, construct 3D point cloud data based on the depth information, calculate the distance matrix and direction matrix between objects in the scene based on the 3D point cloud data, and combining the distance matrix and direction matrix to obtain the scene space feature vector includes: Calculate the pixel mapping relationship between adjacent frames to obtain the displacement field, repair the depth information according to the displacement field to obtain the 3D point cloud, perform dual feature extraction of the normal vector and curvature field on the 3D point cloud to achieve object segmentation, calculate the direction matrix between objects and the distance matrix considering convex hull overlap, and fuse the direction matrix and the distance matrix through the attention mechanism and residual connection to obtain the scene space feature, specifically including: Collect the scene image sequence, perform temporal completion on the depth information of each frame in the scene image sequence, calculate the forward mapping matrix of the pixel position between the current frame and the previous frame, calculate the backward mapping matrix of the pixel position between the current frame and the next frame, and calculate the displacement field of the pixel position based on the forward mapping matrix and the backward mapping matrix; Calculate the motion consistency score according to the displacement field, use the motion consistency score as a weight coefficient to perform weighted combination on the depth values of adjacent frames to obtain a depth completion function, use the depth completion function to repair the missing depth value of the current frame to obtain a repaired depth map, and multiply the pixel coordinates and depth values in the repaired depth map by the camera intrinsic matrix to obtain the 3D point cloud; Construct a search radius for each point in the three-dimensional point cloud, extract the neighborhood point set within the search radius, calculate the mean vector of the neighborhood point set, construct a covariance matrix using the neighborhood point set and the mean vector, perform eigenvalue decomposition on the covariance matrix to obtain the normal vector, calculate the angle between the normal vector and the neighborhood point normal vector to obtain the normal deviation value, and remove the points with the normal deviation value greater than the preset deviation threshold to obtain the filtered point cloud; Calculate the normal change rate of each point in the filtered point cloud to obtain the normal vector field, calculate the local curvature using the normal vector field to obtain the curvature field, input the normal vector field and the curvature field into the region growing algorithm to segment and obtain the plane point set, calculate the spatial distance for the points outside the plane point set to obtain the clustering label, and segment the point cloud into multiple object point sets according to the clustering label; Calculate the centroid position and the point cloud covariance matrix for each of the object point sets respectively, perform eigenvalue decomposition on the point cloud covariance matrix to obtain the principal direction vector, project the principal direction vector onto the three coordinate planes to obtain the projection vectors, calculate the angle between the objects based on the projection vectors to obtain the direction matrix, calculate the distance value between the objects using the centroid position, construct the convex hull of the object point set and calculate the convex hull overlap degree, combine the distance value and the convex hull overlap degree to obtain the distance matrix, input the direction matrix and the distance matrix into a multi-layer perceptron, calculate the attention scores on each layer's feature map of the multi-layer perceptron, perform selective fusion on the features according to the attention scores, and transmit the fused features through residual connections to obtain the scene space feature vector.

3. The method according to claim 1, characterized in that, Perform semantic segmentation on the scene image sequence to obtain the semantic label map, construct the semantic association matrix according to the co-occurrence frequency of the labels in the semantic label map, and calculate the scene semantic vector according to the semantic association matrix, including: Construct the temporal consistency loss to optimize the segmentation network to obtain the semantic label sequence, calculate the conditional entropy for the semantic label sequence to construct the information entropy weight matrix, multiply the information entropy weight matrix by the co-occurrence frequency matrix to obtain the semantic association matrix, construct a graph structure with semantic categories as nodes and the semantic association matrix as edge weights, and use the multi-head attention mechanism and skip connections to fuse the node features to obtain the scene semantic vector, specifically including: Use a deep convolutional neural network to perform semantic segmentation on the scene image sequence to obtain the semantic label sequence, calculate the forward mapping matrix and the backward mapping matrix according to the pixel position correspondence relationship between adjacent frames, use the forward mapping matrix and the backward mapping matrix to align the semantic prediction result of the current frame with the adjacent frames, calculate the difference of the aligned prediction results to construct the temporal consistency loss function, and optimize the semantic segmentation network parameters by minimizing the temporal consistency loss function to obtain the semantic label sequence with temporal consistency; Extract the semantic categories for each frame's semantic label in the semantic label sequence, count the co-occurrence frequency of each pair of semantic categories in the same scene to construct the co-occurrence frequency matrix, calculate the joint probability and marginal probability of the semantic category pairs in the co-occurrence frequency matrix to obtain the conditional entropy, use the conditional entropy to construct the information entropy weight matrix, multiply the information entropy weight matrix by the co-occurrence frequency matrix and perform normalization processing to obtain the semantic association matrix; Construct the semantic categories and the semantic association matrix as the node set and edge weights of a graph structure respectively. Extract the neighborhood node features for each node in the graph structure, calculate the attention coefficients between nodes using the multi-head attention mechanism, perform weighted aggregation on the neighborhood node features according to the attention coefficients to obtain the updated features of the nodes, fuse the multi-layer node updated features through skip connections, and splice the fused node features to obtain a scene semantic vector containing object category information and semantic association patterns.

4. The method according to claim 1, wherein Input the scene spatial feature vector and the scene semantic vector into a variational autoencoder. Obtain the scene feature distribution through the encoding process of the variational autoencoder, sample a target feature vector from the scene feature distribution, calculate the attention scores based on the target feature vector, update the weights of the target feature vector according to the attention scores, and generate a weighted target description vector including: Input the scene spatial feature vector and the scene semantic vector into the feature decomposition unit of the variational autoencoder, obtain the prior probability distribution parameters through Bayesian inference, construct a conditional prior distribution based on the prior probability distribution parameters, extract the shared information and independent information in the scene spatial feature vector and the scene semantic vector respectively according to the conditional prior distribution to obtain a feature shared subspace and a feature independent subspace, calculate the weights of the feature shared subspace and the feature independent subspace based on the mutual information maximization criterion, and perform weighted combination of the feature subspaces according to the weights to obtain a scene feature distribution; Input the scene feature distribution into multiple probability distribution branches of the variational autoencoder respectively, calculate the distribution parameters for each branch based on the statistical characteristics of the scene feature distribution, calculate the information entropy values of each branch using the distribution parameters, and sample a target feature vector from the scene feature distribution according to the distribution parameters of the branch with the minimum information entropy value; Construct each dimension of the target feature vector as a node in a conditional random field, calculate the feature similarity between adjacent nodes to obtain a pairwise potential function, calculate the combined similarity of all nodes within a node group to obtain a combined potential function, construct the pairwise potential function and the combined potential function as an energy function, and minimize the energy function through an iterative optimization method to obtain the attention scores; Multiply the attention scores with each dimension of the target feature vector to obtain a weight-updated feature vector, calculate the statistical value of the weight-updated feature vector to obtain a gating threshold, and retain the feature dimensions in the weight-updated feature vector that are greater than the gating threshold to obtain a weighted target description vector.

5. The method according to claim 4, characterized in that Construct each dimension of the target feature vector as a node in a conditional random field, calculate the feature similarity between adjacent nodes to obtain a pairwise potential function, calculate the combined similarity of all nodes within a node group to obtain a combined potential function, and construct the pairwise potential function and the combined potential function as an energy function including: Construct each dimensional eigenvalue of the target feature vector as a node in a conditional random field, calculate the correlation coefficient between the nodes, set a connection threshold based on the correlation coefficient, and establish a connection relationship for node pairs with a correlation coefficient greater than the connection threshold to obtain an adaptive node connection graph, where the node connection graph includes a node set and an edge set; Calculate the Euclidean distance between the corresponding eigenvalues of adjacent nodes in the node connection graph, and calculate the feature similarity between adjacent nodes based on the Euclidean distance to obtain a pairwise potential function; Calculate the mean of the eigenvalues of all nodes within a node group in the node connection graph, calculate the distance between each node eigenvalue and the mean based on the mean, and calculate the combined similarity of the node group according to the distance to obtain a combined potential function; Calculate the logarithm value of the pairwise potential function to obtain a first energy term, calculate the logarithm value of the combined potential function to obtain a second energy term, and sum the first energy term and the second energy term after respectively assigning weight coefficients to obtain an energy function.

6. The method according to claim 1, wherein Construct a probabilistic graph inference network, map the scene space feature vector to a position probability distribution, map the scene semantic vector to a semantic probability distribution, construct a target generation constraint based on the position probability distribution and the semantic probability distribution, and combine the target generation constraint and the target description vector and input them into a generation network, where the generation network outputs target contour features according to the target generation constraint, including: Construct a two-stream probabilistic graph network to extract position and semantic features respectively, form a target generation constraint by calculating the divergence between the position probability distribution and the uniform distribution and the divergence between the semantic probability distribution and a preset distribution, splice the target generation constraint and the target description vector in the feature dimension, and output target contour features through a generation network, specifically including: Construct a probabilistic graph inference network, where the probabilistic graph inference network includes a position feature stream and a semantic feature stream. The position feature stream processes the scene space feature vector through multiple graph convolutions to obtain a position feature map, and the semantic feature stream calculates attention weights to weight the scene semantic vector to obtain semantic features; Perform a normalization operation on the position feature map to obtain a position probability distribution, and perform a non-linear transformation on the semantic features to obtain a semantic probability distribution; Calculate the divergence value between the position probability distribution and the uniform distribution to obtain a position constraint term, calculate the divergence value between the semantic probability distribution and a preset semantic distribution to obtain a semantic constraint term, and perform a weighted sum on the position constraint term and the semantic constraint term to obtain a target generation constraint; Splice the target generation constraint and the target description vector in the feature dimension, input the spliced features into a generation network, and the generation network amplifies the spatial scale of the input features through an upsampling convolutional layer and outputs target contour features.

7. The method according to claim 1, characterized in that, Construct a regional attention map based on the target contour features, and combine the regional attention map with the target description vector to generate a final target generation result, including: A sliding window is adopted to simultaneously extract the maximum response and average response features to construct spatial attention, which is combined with channel attention to form a regional attention mechanism to obtain a regional attention map. Residual connections are used to enhance feature representation. The regional attention map and the target description features are fused at multiple scales and then decoded to generate the target result, specifically including: Scanning the target contour features in a sliding window manner, calculating the statistical values of the features within the window based on each sliding window position, including calculating the maximum value to obtain the maximum response feature map, calculating the average value to obtain the average response feature map, performing convolution operations on the maximum response feature map and the average response feature map respectively, adding them together, and obtaining the spatial attention map through normalization processing; Performing average pooling on the target contour features in the spatial dimension and obtaining the channel attention map through a fully connected layer, multiplying the spatial attention map, the channel attention map, and the target contour features to obtain the regional attention map; Performing convolution processing on the regional attention map to obtain local features, and performing residual connection on the local features and the regional attention map to obtain enhanced features; Upsampling and expanding the target description vector, splicing the expanded features and the enhanced features in the feature dimension, and performing multi-scale processing on the spliced features through convolutional layers with different kernel sizes; Inputting the features processed at multiple scales into the decoder, and the decoder decodes the features through deconvolution layers and skip connections to generate the target result.

8. An adaptive scene analysis and target generation system based on deep learning, for implementing the method according to any one of the preceding claims 1-7, characterized in that, Including: The first unit is used to extract the depth information of each frame of image in the scene image sequence, construct three-dimensional point cloud data based on the depth information, calculate the distance matrix and direction matrix between objects in the scene based on the three-dimensional point cloud data, and combine the distance matrix and direction matrix to obtain the scene space feature vector; The second unit is used to perform semantic segmentation on the scene image sequence to obtain a semantic label map, construct a semantic association matrix according to the co-occurrence frequency of the labels in the semantic label map, and calculate the scene semantic vector according to the semantic association matrix; The third unit is used to input the scene space feature vector and the scene semantic vector into a variational autoencoder, obtain the scene feature distribution through the encoding process of the variational autoencoder, sample from the scene feature distribution to obtain the target feature vector, calculate the attention score based on the target feature vector, update the weight of the target feature vector according to the attention score, and generate a weighted target description vector; The fourth unit constructs a probabilistic graph inference network, maps the scene space feature vector to a position probability distribution, maps the scene semantic vector to a semantic probability distribution, constructs a target generation constraint based on the position probability distribution and the semantic probability distribution, combines the target generation constraint and the target description vector and inputs them into the generation network, and the generation network outputs the target contour features according to the target generation constraint; The fifth unit constructs a regional attention map based on the target contour features, and combines the regional attention map and the target description vector to generate the final target generation result.

9. An electronic device, characterized in that, Including: A processor; A memory for storing instructions executable by the processor; Wherein, the processor is configured to call the instructions stored in the memory to execute the method according to any one of claims from 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images