Robot scene understanding method and system based on visual deep learning

Abstract

The application provides a robot scene understanding method and system based on visual deep learning, relates to the technical field of robot visual recognition, and comprises the following steps: target relationship matrix is established through multi-layer pyramid feature decomposition and bidirectional feature transmission between local regions; and target transition probability matrix is calculated by using historical scene data to optimize current scene target attributes. The application realizes dynamic modeling of the relationship between targets in a scene and knowledge migration of historical scenes, and improves the accuracy and robustness of robot scene understanding in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot visual recognition, and in particular to a robot scene understanding method and system based on visual deep learning. BACKGROUND

[0002] Robot scene understanding is a key technology in intelligent robot systems, which enables robots to acquire environmental information through visual sensors, understand surrounding objects, spatial structures and mutual relationships, and thus support navigation, operation and decision-making tasks. With the development of deep learning technology, scene understanding methods based on visual deep learning have made significant progress in object detection, scene segmentation and semantic understanding. Traditional scene understanding methods mainly rely on hand-designed feature extractors and rules, while deep learning methods can automatically learn complex feature representations, with stronger robustness and generalization ability.

[0003] Currently, most methods adopt a global feature extraction strategy, which is difficult to handle the balance between local details and global semantics in complex scenes, resulting in poor performance in perceiving fine objects and understanding complex spatial relationships. In addition, current scene understanding systems generally lack an effective utilization mechanism for historical scene knowledge, and cannot transfer previously acquired scene understanding experience to new scenes, resulting in the system still needing to relearn when facing similar scenes, reducing recognition efficiency and adaptability. SUMMARY

[0004] The robot scene understanding method and system based on visual deep learning provided by the embodiments of the present application can solve the problems in the prior art.

[0005] In a first aspect, the robot scene understanding method based on visual deep learning is provided, comprising:

[0006] acquiring a scene image collected by a robot visual sensor;

[0007] dividing the scene image into a plurality of overlapping local regions, performing multi-layer pyramid feature decomposition on each local region to obtain local region features, establishing a bidirectional feature transmission path between adjacent local regions to realize interactive enhancement of local region features and global semantic features, and reorganizing the enhanced local region features according to spatial positions to obtain a scene feature map;

[0008] performing target detection on the scene feature map, selecting a plurality of scene targets with a dispersion less than a consistency threshold, and extracting semantic features of each scene target; calculating the relationship strength between the scene targets based on the position offset and semantic feature change between adjacent frames of the scene targets, iteratively optimizing the relationship strength to obtain a target relationship matrix, and storing the target relationship matrix in a scene database;

[0009] Extract a historical scene target relationship matrix from the scene database, calculate the similarity of the current scene target relationship matrix and the historical scene target relationship matrix, select the highest similarity historical scene to calculate its target transfer probability matrix; based on the target transfer probability matrix, optimize the current scene target attribute to obtain a scene recognition result.

[0010] Divide the scene image into a plurality of overlapping local regions, perform multi-layer pyramid feature decomposition on each local region to obtain local region features, including:

[0011] Calculate the image difference degree of each position point in the scene image and its adjacent position points, and calculate the local region density of each position point by a Gaussian weighting function; take the product of the local region density and a preset overlap coefficient as the overlap rate of each position point, and divide the scene image into a plurality of overlapping local regions based on the overlap rate;

[0012] Construct a multi-layer feature pyramid for each local region, take the scene image of the local region as a first layer pyramid image, perform Gaussian filtering and pixel extraction on the first layer pyramid image to obtain a second layer pyramid image, and repeatedly perform the above operations to obtain a plurality of hierarchical pyramid images;

[0013] Extract the feature vector of each hierarchical pyramid image using a local binary pattern operator and a gray level co-occurrence matrix, and calculate the information entropy thereof, and weight and fuse each layer feature vector according to the information entropy to obtain a local region feature.

[0014] Establish a bidirectional feature transmission path between adjacent local regions to realize interactive enhancement of local region features and global semantic features, reorganize the enhanced local region features according to spatial positions to obtain a scene feature map, including:

[0015] Calculate the Euclidean distance between the center position points of adjacent local regions, and map the Euclidean distance to a distance weight by a Gaussian function;

[0016] Calculate the weighted difference degree of each local region feature and the adjacent local region feature, divide the weighted difference degree by the sum of the distance weights to obtain a feature increment; add the feature increment to the local region feature and multiply it by the corresponding distance weight to obtain a bottom-up transmission feature;

[0017] Perform modulation operation on the global semantic feature and each local region feature, multiply the modulated feature by the corresponding distance weight to obtain a top-down transmission feature; weight and sum the bottom-up transmission feature and the top-down transmission feature to obtain an enhanced feature of the local region;

[0018] The position index matrix is obtained by counting local areas to which each position point belongs, the feature aggregation weight is obtained by mapping the position index matrix through a Gaussian function, the feature aggregation weight is multiplied by the enhanced feature of the corresponding local area and normalized to obtain a scene feature map.

[0019] Target detection is performed on the scene feature map, a plurality of scene targets with a dispersion less than a consistency threshold are selected, and semantic features of each scene target are extracted; a relationship strength between scene targets is calculated based on a position offset and a semantic feature change between adjacent frames of the scene targets, and the relationship strength is iteratively optimized to obtain a target relationship matrix, including:

[0020] Target monitoring is performed on the scene feature map to obtain a plurality of groups of target frame parameters and calculate a dispersion thereof, and target frame parameters with a dispersion less than a consistency threshold are selected as scene targets;

[0021] Class information and attribute information of each scene target are extracted, a mapping vector of the class information in a pre-trained semantic space is calculated, the mapping vector is fused with the attribute information to obtain semantic features, and a semantic dependency strength between scene targets is calculated based on the semantic features;

[0022] A position offset and a semantic feature change between each scene target in adjacent frames are calculated, a time sequence correlation between the scene targets is calculated according to a cooperative change of the position offset and the semantic feature change, and the time sequence correlation is weighted and fused with the semantic dependency strength to obtain a relationship strength between the scene targets, and an initial target relationship matrix is generated;

[0023] Based on the initial target relationship matrix, a relationship strength change rate in adjacent frames is calculated, a relationship strength with a continuous multi-frame relationship strength change rate less than a stability threshold is marked as a stable relationship, and the stable relationship is used as a global constraint to optimize the initial target relationship matrix to obtain a target relationship matrix.

[0024] A position offset and a semantic feature difference between each scene target in adjacent frames are calculated, a time sequence correlation between the scene targets is calculated according to a cooperative change of the position offset and the semantic feature difference, and the time sequence correlation is weighted and fused with the semantic dependency strength to obtain a relationship strength between the scene targets, including:

[0025] A position offset of position coordinates between each scene target in adjacent frames is calculated, and a semantic feature difference between each scene target in adjacent frames is calculated;

[0026] calculating a first influence gradient of a position offset of each scene target on a semantic feature difference value of an adjacent scene target and a second influence gradient of a position offset of the adjacent scene target on the semantic feature difference value of the each scene target, summing the first influence gradient and the second influence gradient to obtain a collaborative change intensity between scene targets;

[0027] dividing the collaborative change intensity into a plurality of local segments according to a time window, calculating a local correlation degree between each local segment to obtain a local correlation degree, calculating a continuity of the collaborative change intensity on the whole time sequence to obtain a global correlation degree, and performing weighted summation on the local correlation degree and the global correlation degree to obtain a time sequence correlation between scene targets;

[0028] performing weighted fusion on the time sequence correlation and the semantic dependency intensity to obtain a relationship intensity between scene targets.

[0029] extracting a historical scene target relationship matrix from the scene database, calculating a similarity between a current scene target relationship matrix and the historical scene target relationship matrix, and selecting a historical scene with the highest similarity to calculate a target transition probability matrix thereof, including:

[0030] extracting a historical scene target relationship matrix from the scene database; performing matrix product operation on a current scene target relationship matrix and the historical scene target relationship matrix to obtain a matrix trace, and dividing the matrix trace by a product of two matrix norms to obtain a structural similarity;

[0031] performing difference calculation and exponential mapping on the current scene target relationship matrix and the historical scene target relationship matrix to obtain a topological similarity; performing weighted combination on the structural similarity and the topological similarity to obtain a comprehensive similarity, and selecting a historical scene with the highest comprehensive similarity as an optimal historical scene;

[0032] in the optimal historical scene, extracting matching features between adjacent scene targets and calculating a target matching degree, and counting a transition frequency between adjacent scene targets in the optimal historical scene to obtain a historical transition frequency; performing normalization processing on a product of the target state matching degree and the historical transition frequency to obtain a target transition probability matrix.

[0033] based on the target transition probability matrix, optimizing a current scene target attribute to obtain a scene recognition result, including:

[0034] performing feature mapping on the target transition probability matrix and a current scene target attribute to obtain a predicted attribute, and calculating an Euclidean distance between the predicted attribute and the current scene target attribute to obtain an attribute difference value;

[0035] calculating a Euclidean distance between the predicted attribute and an optimal historical scene target attribute to obtain a transition difference value, and combining the attribute difference value and the transition difference value to obtain an attribute consistency constraint;

[0036] calculating an adaptive update rate according to the attribute consistency constraint, and combining the predicted attribute and a current scene target attribute according to the adaptive update rate to obtain an optimized scene target attribute;

[0037] inputting the optimized scene target attribute into a pre-trained classifier, and outputting a current scene category identifier as a scene recognition result.

[0038] A second aspect of the embodiment of the present application provides a robot scene understanding system based on visual deep learning, comprising:

[0039] a first unit configured to acquire a scene image collected by a robot visual sensor;

[0040] a second unit configured to divide the scene image into a plurality of overlapping local areas, perform multi-layer pyramid feature decomposition on each local area to obtain local area features, establish a bidirectional feature transmission path between adjacent local areas to realize interactive enhancement of the local area features and global semantic features, and recombine the enhanced local area features according to spatial positions to obtain a scene feature map;

[0041] a third unit configured to perform target detection on the scene feature map, select a plurality of scene targets with a discrete degree less than a consistency threshold, and extract semantic features of each scene target, calculate relationship strengths between the scene targets based on position offsets and semantic feature changes between adjacent frames of the scene targets, iteratively optimize the relationship strengths to obtain a target relationship matrix, and store the target relationship matrix in a scene database;

[0042] a fourth unit configured to extract a historical scene target relationship matrix from the scene database, calculate a similarity between a current scene target relationship matrix and the historical scene target relationship matrix, select a historical scene with a highest similarity to calculate a target transition probability matrix, and optimize a current scene target attribute based on the target transition probability matrix to obtain a scene recognition result.

[0043] A third aspect of the embodiment of the present application provides an electronic device, comprising:

[0044] a processor;

[0045] a memory for storing processor-executable instructions;

[0046] a memory for storing processor-executable instructions;

[0047] The processor is configured to invoke instructions stored in the memory to perform the method described above.

[0048] A fourth aspect of the embodiments of the present application,

[0049] A computer readable storage medium is provided, and computer program instructions are stored on the computer readable storage medium, and the computer program instructions are executed by a processor to implement the method described above.

[0050] The beneficial effects of the present application are as follows:

[0051] The robot scene understanding method based on visual deep learning provided by the present application realizes the interactive enhancement of local region features and global semantic features through multi-layer pyramid feature decomposition and bidirectional feature transmission path, improves the accuracy and robustness of scene feature extraction, and overcomes the problem of insufficient understanding ability of complex scenes in traditional methods.

[0052] The method constructs a scene representation model based on target relationship strength, establishes a target relationship matrix by calculating the position offset and semantic feature change between scene targets, can more comprehensively capture the spatio-temporal correlation between targets in the scene, and enhances the understanding ability of dynamic scenes.

[0053] The present application introduces a historical scene similarity comparison mechanism, calculates the target relationship matrix similarity of the current scene and the historical scene, optimizes the current scene target attribute by using the target transfer probability matrix, realizes the cumulative learning and migration application of scene knowledge, and significantly improves the scene understanding accuracy and adaptability of the robot in a complex and changing environment. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The flowchart of the robot scene understanding method based on visual deep learning of the embodiments of the present application is shown.

[0055] Figure 2 The architecture diagram of bidirectional feature transmission and scene feature map reorganization is shown. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0057] The technical solutions of the present application will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described in detail in some examples.

[0058] Figure 1 The flowchart of the scene understanding method of the robot based on visual deep learning of the embodiment of the present application is shown in Figure 1 The method comprises the following steps:

[0059] Obtaining the scene image collected by the robot visual sensor;

[0060] Dividing the scene image into multiple overlapping local regions, performing multi-layer pyramid feature decomposition on each local region to obtain local region features, establishing a bidirectional feature transmission path between adjacent local regions to realize interactive enhancement of local region features and global semantic features, and reorganizing the enhanced local region features according to spatial positions to obtain a scene feature map;

[0061] Performing target detection on the scene feature map, selecting multiple scene targets with a dispersion degree less than a consistency threshold, and extracting semantic features of each scene target; calculating the relationship strength between scene targets based on the position offset and semantic feature change between adjacent frames of the scene targets, iteratively optimizing the relationship strength to obtain a target relationship matrix, and storing the target relationship matrix in a scene database;

[0062] Extracting a historical scene target relationship matrix from the scene database, calculating the similarity between the current scene target relationship matrix and the historical scene target relationship matrix, selecting the historical scene with the highest similarity to calculate a target transition probability matrix; based on the target transition probability matrix, optimizing the current scene target attributes to obtain a scene recognition result.

[0063] In an optional implementation, dividing the scene image into multiple overlapping local regions, performing multi-layer pyramid feature decomposition on each local region to obtain local region features, comprises:

[0064] Calculating the image difference degree of each position point in the scene image and its adjacent position points, calculating the local region density of each position point through a Gaussian weighting function based on the image difference degree; taking the product of the local region density and a preset overlap coefficient as the overlap rate of each position point, and dividing the scene image into multiple overlapping local regions based on the overlap rate;

[0065] Constructing a multi-layer feature pyramid for each local region, taking the scene image of the local region as a first layer pyramid image, performing Gaussian filtering and extracting pixel points on the first layer pyramid image to obtain a second layer pyramid image, and repeatedly performing the above operations to obtain multiple hierarchical pyramid images;

[0066] The feature vectors of each level pyramid image are extracted by using a local binary pattern operator and a gray level co-occurrence matrix, and an information entropy thereof is calculated, and each level feature vector is weighted and fused according to the information entropy, to obtain a local region feature.

[0067] The embodiment provides a scene image processing method, which divides a scene image into a plurality of overlapping local regions, and performs multi-level pyramid feature decomposition on each local region, to obtain a local region feature.

[0068] Before the scene image is divided, the image difference degree of each position point in the scene image and its adjacent position points needs to be calculated. Assuming that the size of the scene image is MxN, for a position point (i, j), a 3x3 neighborhood around the position point is selected, and the gray value difference of the position point and other position points in the neighborhood is calculated. The image difference degree between the position point (i, j) and the position point (i+m, j+n) in the neighborhood can be expressed as the absolute difference of the gray values of the two points, that is, |I(i, j)-I(i+m, j+n)|, where m, n ∈ {-1, 0, 1} and are not 0 at the same time. After the image difference degrees of all adjacent position points of the position point (i, j) are calculated, the difference degrees are weighted by a Gaussian weighting function. Specifically, for the image difference degree d, the Gaussian weighting value is exp(-d 2 / 2σ 2 ), where σ is the standard deviation of the Gaussian function, which can be set to 10. The sum of all the weighted values is obtained to obtain the local region density D(i, j) of the position point (i, j).

[0069] In order to realize the overlapping division of the regions, the product of the local region density D(i, j) and a preset overlap coefficient α is taken as the overlap rate R(i, j) = αxD(i, j) of the position point (i, j), where α can be set to 0.5. The scene image is divided into a plurality of overlapping local regions based on the overlap rate R. Specifically, an initial point (i0, j0) is selected as the center of the first local region from the top left corner, and the size of the local region can be set to WxW, such as 64x64 pixels. The center position of the next local region is determined according to the overlap rate R(i0, j0), and if R(i0, j0) = 0.5, the center position of the next local region is (i0+W / 2, j0), indicating that there is 50% overlap in the horizontal direction. This process is repeated until the entire scene image is covered. For example, for a 512x512 image, using a local region of 64x64 and an overlap coefficient of 0.5, about 81 overlapping local regions can be obtained.

[0070] For each divided local region, a multi-layer feature pyramid is constructed. Taking a local region of 64x64 pixels as an example, the scene image of the local region is taken as the first layer pyramid image P1. A 5x5 Gaussian filter is performed on P1, and the weights of the filter can be set as [1, 4, 6, 4, 1] / 16, [4, 16, 24, 16, 4] / 64, [6, 24, 36, 24, 6] / 96, [4, 16, 24, 16, 4] / 64, [1, 4, 6, 4, 1] / 16. After filtering, the pixel points are extracted according to the ratio of 2:1 to obtain the second layer pyramid image P2 with a size of 32x32. This operation is repeated to obtain the third layer pyramid image P3 with a size of 16x16, the fourth layer pyramid image P4 with a size of 8x8, and so on, until the image size is less than a preset threshold, such as 4x4.

[0071] For each layer of pyramid image, a local binary pattern (LBP) operator and a gray level co-occurrence matrix (GLCM) are used to extract a feature vector. Taking the first layer pyramid image P1 as an example, when the LBP operator is applied, the center pixel point and its eight adjacent points are selected. If the gray value of the adjacent point is greater than or equal to the center point, the position is coded as 1, otherwise as 0. An 8-bit binary number is thus obtained, which is converted into a decimal number as the LBP value of the center point. The LBP value distribution of the entire image is counted to form a 59-dimensional feature vector F LBP1 . At the same time, the GLCM of P1 is calculated, the distance is set to 1, the direction is set to 0°, 45°, 90° and 135°, and four statistics of the GLCM, contrast, correlation, energy and homogeneity, are extracted to form a 16-dimensional feature vector F GLCM1 . F LBP1 and F GLCM1 are connected to obtain a 75-dimensional feature vector F1. Similarly, the same operation is performed on the other layers of pyramid images P2, P3, P4 to obtain feature vectors F2, F3, F4.

[0072] In order to reasonably fuse the features of each layer, the information entropy of each layer of feature vector is calculated. For the feature vector F i , the information entropy H i is calculated after normalization. Specifically, each element of F i is divided by its maximum value to obtain a normalized vector, the normalized vector is divided into 10 uniform intervals, the number of elements in each interval is counted, the probability distribution is calculated, and then the information entropy is calculated. For example, for a four-layer pyramid, the obtained information entropy values are H1=4.2, H2=3.8, H3=3.2, H4=2.5. According to the information entropy, the feature vectors of each layer are weighted and fused, and the weight w i =H i / ∑H j , to obtain the final feature vector F of the local region, F=w1F1+w2F2+w3F3+w4F4.

[0073] By the above method, the overlapping local region division and multi-layer pyramid feature decomposition of the scene image are realized, and the representation containing rich local feature information is obtained, which has important significance for image analysis and understanding of complex scenes.

[0074] Figure 2 The schematic diagram of the architecture of bidirectional feature transmission and scene feature map reorganization is shown. In an optional embodiment, the bidirectional feature transmission path is established between adjacent local regions, the interactive enhancement of local region features and global semantic features is realized, the enhanced local region features are reorganized according to the spatial positions, and the scene feature map is obtained, including:

[0075] The Euclidean distance between the center position points of adjacent local regions is calculated, and the distance weight is obtained by mapping the Euclidean distance through a Gaussian function;

[0076] The weighted difference degree of each local region feature and the adjacent local region feature is calculated, the feature increment is obtained by dividing the weighted difference degree by the sum of the distance weights, the feature increment and the local region feature are added, and the corresponding distance weight is multiplied to obtain the bottom-up transmission feature;

[0077] The global semantic feature and each local region feature are modulated, the modulated feature and the corresponding distance weight are multiplied to obtain the top-down transmission feature, and the bottom-up transmission feature and the top-down transmission feature are weighted and summed to obtain the enhanced feature of the local region.

[0078] The position index matrix of each position point belonging to the local region is counted, the feature aggregation weight is obtained by mapping the position index matrix through a Gaussian function, the feature aggregation weight and the enhanced feature of the corresponding local region are multiplied and normalized to obtain the scene feature map.

[0079] The embodiment of the application provides a method for establishing a bidirectional feature transmission path between adjacent local regions, realizing the interactive enhancement of local region features and global semantic features, reorganizing the enhanced local region features according to the spatial positions, obtaining a scene feature map, and improving the feature expression ability.

[0080] The input image is preprocessed. The preprocessing includes image size adjustment, standardization and enhancement, etc. For example, the input image is adjusted to 256x256 pixel size, and brightness and contrast adjustment is performed to ensure that the image quality meets the requirements of subsequent processing. After image preprocessing, the local region features of the image are extracted by a convolutional neural network. The network contains multiple convolutional layers and pooling layers, and the output feature map size is 32x32x128, which represents that the image is divided into 32x32 local regions, and each region is represented by a 128-dimensional feature vector. At the same time, the global semantic features of 1x1x256 are obtained through global average pooling and fully connected layers.

[0081] The Euclidean distance between the center position points of adjacent local regions is calculated. Assuming that the current processed local region is located at coordinates (i, j), its adjacent regions include regions in eight directions, i.e. (i-1, j-1), (i-1, j), (i-1, j+1), (i, j-1), (i, j+1), (i+1, j-1), (i+1, j), (i+1, j+1). Taking (i, j) and (i-1, j) as an example, the Euclidean distance between the center points of the two regions is 1.0. The distance weight is obtained by mapping the distance value through a Gaussian function. Specifically, using a Gaussian function with parameter σ=2.0, the calculated distance weight is 0.8825. For other adjacent regions, such as (i-1, j-1), the Euclidean distance is 1.414, and the corresponding distance weight is 0.6065.

[0082] The weighted difference degree of each local region feature and the feature of the adjacent local region is calculated. Assuming that the feature vector of the current local region (i, j) is [0.5, 0.3, 0.7,..., 0.2], and the feature vector of the adjacent region (i-1, j) is [0.6, 0.2, 0.8,..., 0.1], the difference value between the two is [-0.1, 0.1, -0.1,..., 0.1]. Multiply this difference value by the corresponding distance weight 0.8825 to get the weighted difference degree [-0.08825, 0.08825, -0.08825,..., 0.08825]. Similar calculations are performed for all adjacent regions, and the weighted difference degrees are accumulated and divided by the sum of the distance weights (about 5.5) to obtain the feature increment, for example, [-0.02, 0.015, -0.018,..., 0.01].

[0083] The above feature increment is added to the current local region feature to obtain [0.48, 0.315, 0.682,..., 0.21], which is then multiplied by the corresponding distance weight 0.8825 to obtain the bottom-up transmission feature [0.4236, 0.278, 0.6018,..., 0.1853].

[0084] When performing top-down feature transmission, the global semantic feature [0.4, 0.6, 0.5,..., 0.3] is modulated with the current local region feature [0.5, 0.3, 0.7,..., 0.2]. The modulation method is element-level multiplication, and the result is [0.2, 0.18, 0.35,..., 0.06]. Multiply the modulated feature by the distance weight 0.8825 to obtain the top-down transmission feature [0.1765, 0.1589, 0.3089,..., 0.053].

[0085] The top-down transmission feature and the bottom-up transmission feature are weighted and summed. The bottom-up weight is set to 0.6 and the top-down weight is set to 0.4 to obtain the enhanced feature of the local region [0.3246, 0.2304, 0.4833,..., 0.1325].

[0086] In order to reorganize the enhanced local region feature to form a scene feature map, it is necessary to count the local region to which each position point belongs to form a position index matrix. Considering that the original image size is 256x256, after network downsampling, 32x32 local regions are obtained. For a position point (x, y) in the original image, the local region index to which it belongs is (floor(x / 8), floor(y / 8)). For example, the position point (100, 150) belongs to the local region index (12, 18).

[0087] The position index matrix is mapped to the feature aggregation weight through a Gaussian function. For the position point (100, 150), the distance between the center point (12 8+4,18 8+4)=(100, 148) of the local region is 2.0, and the weight obtained by mapping through the Gaussian function with parameter σ=3.0 is 0.8521. Similarly, the distances of the position point from adjacent local regions are calculated and the corresponding weights are calculated.

[0088] The feature aggregation weight is multiplied by the enhanced feature of the corresponding local region. For example, the position point (100, 150) is mainly affected by the local regions (12, 18) and (12, 19), with weights of 0.8521 and 0.1479, respectively. Multiply these weights with the enhanced features of the corresponding local regions and perform normalization to obtain the final feature of the position point. Repeat the above operation for all position points to generate a 256x256x128-dimensional scene feature map.

[0089] Through the above bidirectional feature transmission path, the local region feature and the global semantic feature are interactively enhanced, effectively improving the expression ability of the scene feature map, and providing more accurate feature representation for subsequent scene understanding and analysis tasks.

[0090] In an optional embodiment, target detection is performed on the scene feature map, a plurality of scene targets with a dispersion less than a consistency threshold are selected, and semantic features of each scene target are extracted; based on a position offset and a semantic feature change amount between adjacent frames of the scene targets, a relationship strength between the scene targets is calculated, and iterative optimization is performed on the relationship strength to obtain a target relationship matrix, including:

[0091] Target monitoring is performed on the scene feature map to obtain a plurality of groups of target box parameters and calculate a dispersion thereof, and target box parameters with a dispersion less than a consistency threshold are selected as scene targets;

[0092] Category information and attribute information of each scene target are extracted, a mapping vector of the category information in a pre-trained semantic space is calculated, the mapping vector is fused with the attribute information to obtain semantic features, and a semantic dependency strength between the scene targets is calculated based on the semantic features;

[0093] A position offset and a semantic feature change amount between adjacent frames of each scene target are calculated, a time sequence correlation between the scene targets is calculated according to a cooperative change of the position offset and the semantic feature change amount, and a relationship strength between the scene targets is obtained by weighted fusion of the semantic dependency strength, to generate an initial target relationship matrix;

[0094] Based on the initial target relationship matrix, a relationship strength change rate in adjacent frames is calculated, a relationship strength with a relationship strength change rate less than a stability threshold in a plurality of continuous frames is marked as a stable relationship, and the stable relationship is used as a global constraint to optimize the initial target relationship matrix to obtain a target relationship matrix.

[0095] In a specific embodiment, scene feature maps are extracted from the obtained scene image sequence, and a target detection operation is performed. Specifically, a pre-trained target detection model (such as a Faster R-CNN architecture) is used to process the input scene feature map, and the boundary box coordinates, confidence, and category prediction of each potential target are output. For each target, its boundary box parameters (x, y, width, height) and confidence score are obtained. To evaluate the stability of the target box, a dispersion index of the target box between consecutive frames is calculated. This index measures the fluctuation degree of the target box in spatial position and size. For example, for a specific target, if the center coordinates of its boundary box change by less than 5% of the pixel value and the size changes by less than 10% in consecutive 5 frames, the dispersion is low. In practical applications, a consistency threshold of 0.25 can be set, and when the dispersion of the target box is lower than this threshold, it is considered as a stable scene target.

[0096] For each selected scene target, further extract its semantic features. Semantic features contain two parts: class information and attribute information. Class information such as "person", "car", "building" and other target identifiers, attribute information includes the color, size, texture and other characteristics of the target description. The class information is mapped into a pre-trained semantic space, using a pre-trained model such as Word2Vec or GloVe, the class label is converted into a 300-dimensional semantic vector. Attribute information is obtained through a feature extraction network, and a 128-dimensional attribute feature vector is output. The class mapping vector and the attribute feature vector are fused, and a simple vector splicing operation or an attention mechanism can be used for weighted fusion to obtain a 428-dimensional comprehensive semantic feature vector. Based on the obtained semantic features, the semantic dependence strength between scene targets is calculated, and the cosine similarity method can be used to calculate the similarity score of the semantic feature vectors of two targets, and the score range is between 0 and 1. The higher the score, the stronger the semantic dependence relationship between the two targets.

[0097] To capture the dynamic relationship between targets, the position offset and semantic feature change between adjacent frames of scene targets are calculated. The position offset is calculated by the Euclidean distance between the center points of the target bounding box in adjacent frames. For example, the center point of a target moves from (150, 200) to (155, 210) between two frames, and the position offset is 11.18 pixels. The semantic feature change is calculated by the L2 norm difference of the semantic feature vectors between adjacent frames. According to the cooperative change mode of the position offset and the semantic feature change, the time correlation between targets is evaluated. For example, when two targets show similar motion patterns (position offset direction and amplitude close) and consistent semantic feature change trends in consecutive multiple frames, they have an interaction relationship. The time correlation score is weighted and fused with the semantic dependence strength, and the fusion weight can be set to 0.7 and 0.3 to obtain the comprehensive relationship strength, which is used to construct the initial target relationship matrix.

[0098] For a scene composed of n targets, the initial target relationship matrix is an n x n matrix, where the matrix element R[i][j] represents the relationship strength between target i and target j. Further optimization is performed based on the initial matrix to calculate the change rate of the relationship strength between consecutive frames. Specifically, for any two targets i and j, the relationship strength change rate between them in adjacent frames t and t+1 is calculated, change rate = |R t [i][j]-R (t+1) [i][j]| / R t [i][j]。If the change rate is less than the stability threshold (such as 0.15) in consecutive multiple frames (such as 5 frames), the relationship is marked as a stable relationship. These stable relationships serve as global constraints to guide the optimization of the entire relationship matrix.

[0099] The optimization process is performed in an iterative manner, considering the transitivity and consistency of stable relationships. For example, if there is a stable relationship between target A and B, and a stable relationship between B and C, the relationship strength between A and C should also satisfy certain consistency constraints. In specific implementation, the strength value of non-stable relationships can be updated multiple times to maintain harmony with the structure formed by stable relationships. In each iteration, the relationship strength is updated based on the adjacent relationship until the matrix converges or the preset number of iterations (e.g., 10 times) is reached. The final optimized matrix is the target relationship matrix, where each element value represents the relationship strength between the corresponding target pair, providing a basis for subsequent scene understanding and event analysis.

[0100] Through the above method, the system can effectively identify and quantify the complex relationships between targets in the scene, providing more in-depth scene understanding capabilities, and being suitable for various application scenarios such as video monitoring, autonomous driving, and human-computer interaction.

[0101] In an optional implementation, the position offset of each scene target between adjacent frames and the semantic feature difference value are calculated, the time sequence correlation between the scene targets is calculated according to the cooperative change of the position offset and the semantic feature difference value, and the relationship strength between the scene targets is obtained by weighted fusion of the semantic dependency strength, including:

[0102] The position offset of each scene target between adjacent frames is calculated, and the semantic feature difference value of each scene target between adjacent frames is calculated.

[0103] The first influence gradient of the position offset of each scene target on the semantic feature difference value of the adjacent scene target, and the second influence gradient of the position offset of the adjacent scene target on the semantic feature difference value of each scene target are calculated, the first influence gradient and the second influence gradient are summed to obtain the cooperative change strength between the scene targets;

[0104] The cooperative change strength is divided into multiple local segments according to a time window, the similarity between each local segment is calculated to obtain a local correlation degree, the continuity of the cooperative change strength on the entire time sequence is calculated to obtain a global correlation degree, and the local correlation degree and the global correlation degree are weighted and summed to obtain the time sequence correlation between the scene targets;

[0105] The time sequence correlation and the semantic dependency strength are weighted and fused to obtain the relationship strength between the scene targets.

[0106] In one embodiment, the present application provides a method for calculating the relationship strength between scene targets. To calculate the position offset of the scene target between adjacent frames, the position coordinates of the scene target in consecutive video frames are obtained. Assuming that in the t-th frame, the position coordinates of target A are (x tA , ytA ), in the t+1th frame, the position coordinate of target A is (x t+1A , y t+1A ), then the position offset of target A between adjacent frames is equal to the difference between the two position coordinates. For example, if the coordinate of target A in the 1st frame is (100, 150) and the coordinate in the 2nd frame is (105, 160), then the position offset is (5, 10).

[0107] For the calculation of the semantic feature difference, the semantic feature vector of each scene target needs to be extracted. Suppose in the tth frame, the semantic feature vector of target A is F tA , in the t+1th frame, the semantic feature vector of target A is F t+1A , then the semantic feature difference between target A in adjacent frames can be obtained by calculating the difference between the feature vectors. For example, if the semantic feature vector of target A in the 1st frame is [0.5, 0.3, 0.2] and the semantic feature vector in the 2nd frame is [0.6, 0.25, 0.15], then the semantic feature difference is [0.1, -0.05, -0.05].

[0108] When calculating the collaborative change intensity between scene targets, the influence gradient in two directions is considered. The first influence gradient of the position offset on the semantic feature difference is calculated. Specifically, for each component of the position vector of target A, the degree of influence of the change of the component on the semantic feature difference between target A and target B is calculated. For example, when the x coordinate of target A changes from 150 to 151.5, if the first element of the semantic feature difference vector changes from -0.03 to -0.035, then the influence gradient of the x direction on this element is (-0.035-(-0.03)) / 1.5 = -0.0033. The influence gradients of all semantic feature difference elements are averaged to obtain the comprehensive influence gradient of the position component. Suppose the comprehensive influence gradient in the x direction is 0.025, in the y direction is 0.018, in the w direction is 0.010, and in the h direction is 0.008, then the first influence gradient vector of target A is [0.025, 0.018, 0.010, 0.008].

[0109] A semantic feature difference vector is calculated between target A and target B. For example, if target A is a rectangle with coordinates [100, 100, 200, 200] and target B is a circle with coordinates [200, 180, 40, 55], the semantic feature difference vector is [100, 80, 160, 45].

[0110] A position offset distance is calculated, which is the Euclidean distance between the center points of target A and target B. For example, if the center point of target A is (150, 200) and the center point of target B is (200, 180), the position offset distance is ((200-150) 2 +(180-200) 2 ) 0.5 = (50 2 +20 2 ) 0.5 ≈ 53.85 pixels.

[0111] The first and second influence gradients are weighted and summed to obtain the collaborative change strength between target A and target B. The weight can be set in inverse proportion to the position offset distance, with a greater weight for a closer distance. For example, if the position offset distance is 53.85 pixels, a weight factor of 100 / 53.85 ≈ 1.857 can be set. The collaborative change strength calculation formula is the sum of the elements of the first influence gradient vector multiplied by the weight factor, plus the sum of the elements of the second influence gradient vector multiplied by the weight factor. In this example, the collaborative change strength is (0.025+0.018+0.010+0.008)×1.857 + (0.015+0.022+0.009+0.011)×1.857 ≈ 0.113×1.857 + 0.057×1.857 ≈ 0.316. Repeat the above calculation process for all adjacent target pairs in the scene to obtain the complete collaborative change strength.

[0112] To calculate the temporal correlation between the scene targets, the collaborative change strength is divided into multiple local segments according to a time window. For example, in a video sequence containing 100 frames, it can be divided into a local segment every 10 frames, and a total of 10 local segments are obtained. For each two local segments, the similarity between them is calculated to obtain the local correlation degree. The similarity can be obtained by calculating the cosine similarity or correlation coefficient of the collaborative change strength sequence in the two segments. For example, if the similarity of the 1st segment and the 2nd segment is 0.85, the similarity of the 1st segment and the 3rd segment is 0.75, and so on, a 10x10 local correlation degree matrix can be obtained.

[0113] At the same time, the continuity of the collaborative change strength in the entire time sequence is calculated to obtain the global correlation degree. The continuity can be obtained by calculating the autocorrelation coefficient or the frequency spectrum characteristics of the Fourier transform of the collaborative change strength sequence. For example, if the autocorrelation coefficient is 0.92, indicating that the collaborative change strength has high continuity in the entire time sequence, then the global correlation degree is 0.92.

[0114] The local correlation degree and the global correlation degree are weighted and summed to obtain the temporal correlation between the scene targets. The weight can be adjusted according to the specific needs of the application scenario. For example, the weight of the local correlation degree can be set to 0.6, and the weight of the global correlation degree can be set to 0.4, then the temporal correlation is the local correlation degree multiplied by 0.6 plus the global correlation degree multiplied by 0.4. For target A and target B, if their local correlation degree is 0.83 and their global correlation degree is 0.92, then their temporal correlation is 0.83x0.6+0.92x0.4=0.87.

[0115] The temporal correlation and the semantic dependency strength are weighted and fused to obtain the relationship strength between the scene targets. The semantic dependency strength can be obtained by calculating the similarity between the semantic feature vectors of the scene targets. For example, if the cosine similarity of the semantic feature vectors of target A and target B is 0.78, then their semantic dependency strength is 0.78. Similarly, the weight of the weighted fusion can be adjusted according to the specific needs of the application scenario. For example, the weight of the temporal correlation can be set to 0.7, and the weight of the semantic dependency strength can be set to 0.3, then the relationship strength is the temporal correlation multiplied by 0.7 plus the semantic dependency strength multiplied by 0.3. For target A and target B, if their temporal correlation is 0.87 and their semantic dependency strength is 0.78, then their relationship strength is 0.87x0.7+0.78x0.3=0.84.

[0116] Through the above method, the system can comprehensively consider the cooperative changes of the scene targets in spatial positions and semantic features, and the correlation and semantic dependency relationship in time series, so as to accurately calculate the relationship strength between the scene targets, and provide an important basis for subsequent scene understanding and behavior prediction.

[0117] In an optional implementation, a historical scene target relationship matrix is extracted from the scene database, a similarity between the current scene target relationship matrix and the historical scene target relationship matrix is calculated, and a target transition probability matrix of a historical scene with the highest similarity is selected, including:

[0118] A historical scene target relationship matrix is extracted from the scene database, a matrix trace is obtained by performing matrix multiplication operation on the current scene target relationship matrix and the historical scene target relationship matrix, and a structural similarity is obtained by dividing the matrix trace by the product of two matrix norms;

[0119] A topological similarity is obtained by performing difference calculation and exponential mapping on the current scene target relationship matrix and the historical scene target relationship matrix, a comprehensive similarity is obtained by weighted combination of the structural similarity and the topological similarity, and an optimal historical scene is selected according to the highest comprehensive similarity;

[0120] In the optimal historical scene, matching features between adjacent scene targets are extracted and a target matching degree is calculated, a historical transition frequency is obtained by counting the transition times between adjacent scene targets in the optimal historical scene, and a target transition probability matrix is obtained by normalizing the product of the target state matching degree and the historical transition frequency.

[0121] In a specific implementation, in order to improve the accuracy of scene understanding and target prediction, the application provides a historical scene matching and target transition probability matrix calculation method. The method extracts a historical scene target relationship matrix, calculates the similarity between the current scene and the historical scene, and calculates the target transition probability based on the optimal matching historical scene, so as to effectively predict the future state of the target in the current scene.

[0122] In the embodiment, the scene database stores a plurality of historical scene information, each historical scene contains spatial relationship, interaction relationship and other information between targets in the scene, and these relationships are represented by a target relationship matrix. Each element of the target relationship matrix represents the relationship strength or interaction degree between two targets in the scene. When it is necessary to predict the future state of the target in the current scene, the target relationship matrix of the historical scene is extracted from the scene database.

[0123] For each extracted historical scene-target relationship matrix, the similarity between it and the current scene-target relationship matrix is calculated. The similarity calculation is divided into two parts: structural similarity and topological similarity. The structural similarity is achieved by matrix multiplication operation. Specifically, the current scene-target relationship matrix is multiplied by the historical scene-target relationship matrix, and the trace of the matrix product (i.e., the sum of the main diagonal elements) is calculated. Then the matrix trace is divided by the product of the norms of the two matrices to obtain the structural similarity. The matrix norm uses the Frobenius norm, which is the square root of the sum of the squares of all elements of the matrix.

[0124] For example, assume that the current scene-target relationship matrix is a 3x3 matrix with element values [[0.8, 0.5, 0.2], [0.5, 0.7, 0.4], [0.2, 0.4, 0.9]], and the historical scene-target relationship matrix is also a 3x3 matrix with element values [[0.7, 0.4, 0.3], [0.4, 0.8, 0.3], [0.3, 0.3, 0.8]]. The product of the two matrices is calculated, and the trace of the resulting matrix is 0.8x0.7+0.5x0.4+0.2x0.3+0.5x0.4+0.7x0.8+0.4x0.3+0.2x0.3+0.4x0.3+0.9x0.8=2.11. The Frobenius norm of the current scene matrix is 2.14, and the Frobenius norm of the historical scene matrix is 1.98. The structural similarity is 2.11 / (2.14x1.98)=0.50.

[0125] The topological similarity is obtained by calculating the difference between the two matrices and performing exponential mapping. Specifically, the absolute value of the difference between the corresponding elements of the current scene-target relationship matrix and the historical scene-target relationship matrix is calculated to obtain a difference matrix. Each element of the difference matrix is negatively exponentiated, i.e., exp(-|difference|), to obtain a topological similarity matrix. The topological similarity is the average of all elements of this matrix.

[0126] Based on the above example, the absolute value of the difference between the two matrices is [[0.1, 0.1, 0.1], [0.1, 0.1, 0.1], [0.1, 0.1, 0.1]]. After negative exponentiation, we get [[0.905, 0.905, 0.905], [0.905, 0.905, 0.905], [0.905, 0.905, 0.905]]. The topological similarity is 0.905.

[0127] The comprehensive similarity is obtained by weighting combination of the structural similarity and the topological similarity. A weight coefficient a is set, and the comprehensive similarity is calculated as a x the structural similarity + (1-a) x the topological similarity. In the embodiment, a is 0.6, and the comprehensive similarity is 0.6 x 0.50 + 0.4 x 0.905 = 0.662.

[0128] The comprehensive similarity is calculated for all historical scenes in the scene database, and the historical scene with the highest comprehensive similarity is selected as the optimal historical scene. It is assumed that the selected optimal historical scene contains 10 continuous time frames, and each frame contains 5 targets.

[0129] After the optimal historical scene is determined, the target transition probability matrix needs to be calculated. The target transition probability matrix represents the probability distribution of the target transition from the current state to the next state in the scene. The calculation process includes two steps: extracting the matching features between the targets of adjacent scenes and calculating the target matching degree; and counting the transition times between the targets of adjacent scenes in the optimal historical scene to obtain the historical transition frequency.

[0130] The target matching features include the position, velocity, size, appearance and other features of the target. For each pair of targets in adjacent time frames, the feature similarity between them is calculated as the target matching degree. For example, for the position feature, the reciprocal of the Euclidean distance can be calculated; for the velocity feature, the cosine similarity of the velocity vector can be calculated; and for the appearance feature, the intersection of the color histogram can be calculated. The comprehensive matching degree is obtained by weighting combination of these feature similarities.

[0131] It is assumed that in the adjacent two frames of the optimal historical scene, the position feature similarity between target A and target B in the next frame is 0.8, the velocity feature similarity is 0.7, and the appearance feature similarity is 0.9. The weights are set as 0.4, 0.3 and 0.3 respectively, and the comprehensive matching degree is 0.4 x 0.8 + 0.3 x 0.7 + 0.3 x 0.9 = 0.8.

[0132] The historical transition frequency is obtained by counting the transition times between the targets of adjacent scenes in the optimal historical scene. For example, in the 10 time frames of the optimal historical scene, the target type A is transitioned to the target type B for 7 times, to the target type C for 2 times, and to the target type D for 1 time, and the historical transition frequency of the target type A is [0, 0.7, 0.2, 0.1, 0] (assuming that there are 5 target types).

[0133] The calculation of the target transition probability matrix is to normalize the product of the target matching degree and the historical transition frequency. Specifically, for each target in the current scene, the matching degree with each target in the next scene is calculated, multiplied by the corresponding historical transition frequency, and normalized to make the probability sum of each row equal to 1, to obtain the target transition probability matrix.

[0134] Taking the above example, suppose the matching degree of the current target A and the 5 targets in the next scene is [0.2, 0.8, 0.5, 0.3, 0.1], the historical transition frequency is [0, 0.7, 0.2, 0.1, 0], and the product of the two is [0, 0.56, 0.1, 0.03, 0]. After normalization, the transition probability is [0, 0.81, 0.14, 0.05, 0]. Repeat this calculation for all targets to form a complete transition probability matrix.

[0135] Through the above steps, the transition probability matrix of each target in the current scene is successfully calculated, providing an important basis for subsequent target state prediction.

[0136] In an optional implementation, based on the target transition probability matrix, the target attribute of the current scene is optimized to obtain a scene recognition result, including:

[0137] Feature mapping is performed on the target transition probability matrix and the target attribute of the current scene to obtain a predicted attribute, and the Euclidean distance between the predicted attribute and the target attribute of the current scene is calculated to obtain an attribute difference value;

[0138] The Euclidean distance between the predicted attribute and the optimal historical scene target attribute is calculated to obtain a transition difference value, and the attribute difference value and the transition difference value are weighted and combined to obtain an attribute consistency constraint;

[0139] An adaptive update rate is calculated according to the attribute consistency constraint, and the predicted attribute and the target attribute of the current scene are weighted and combined according to the adaptive update rate to obtain an optimized scene target attribute;

[0140] The optimized scene target attribute is input into a pre-trained classifier, and the current scene category identifier is output as a scene recognition result.

[0141] The scene recognition method based on the target transition probability matrix relates to the fields of computer vision and target tracking. The target transition probability matrix refers to a matrix reflecting the transition rule of target attributes between different scenes, which is constructed by analyzing the historical scene target attributes. The method optimizes the target attribute of the current scene to obtain an accurate scene recognition result.

[0142] In actual implementation, after obtaining the target transition probability matrix, the matrix is used to optimize the target attribute of the current scene. Specifically, the target transition probability matrix and the target attribute of the current scene are feature mapped to obtain a predicted attribute. This mapping process can be realized by matrix multiplication, that is, the target attribute vector of the current scene is multiplied by the target transition probability matrix to obtain a predicted target attribute vector. For example, assuming that the target attribute of the current scene is a five-dimensional vector [0.72, 0.31, 0.56, 0.43, 0.68], and the target transition probability matrix is a 5x5 matrix, a predicted attribute vector [0.65, 0.37, 0.59, 0.41, 0.70] can be obtained through feature mapping.

[0143] After obtaining the predicted attribute, the Euclidean distance between the predicted attribute and the target attribute of the current scene is calculated to obtain an attribute difference value. The Euclidean distance calculation method is to sum the square of the difference value of the corresponding position elements of two vectors, and then take the square root. According to the above example, the calculated Euclidean distance is 0.11, which is taken as the attribute difference value. The Euclidean distance between the predicted attribute and the optimal historical scene target attribute is calculated to obtain a transition difference value. The optimal historical scene target attribute refers to the target attribute corresponding to the scene with the highest recognition accuracy in the historical scene. Assuming that the optimal historical scene target attribute is [0.67, 0.35, 0.58, 0.42, 0.69], the calculated Euclidean distance is 0.03, which is taken as the transition difference value.

[0144] The attribute difference value and the transition difference value are combined by weighting to obtain an attribute consistency constraint. The weighted combination can be realized by setting a weight coefficient α and calculating α times the attribute difference value plus (1-α) times the transition difference value. Assuming that the weight coefficient α is 0.6, the attribute consistency constraint is 0.6x0.11+0.4x0.03=0.078.

[0145] According to the attribute consistency constraint, an adaptive update rate is calculated. The adaptive update rate reflects the influence degree of the predicted attribute on the optimization result. The attribute consistency constraint is brought into an exponential function, for example, exp(-attribute consistency constraint), to obtain the adaptive update rate. According to the above example, the adaptive update rate is exp(-0.078)≈0.925.

[0146] The predicted attribute and the current scene target attribute are combined by weighting according to the adaptive update rate to obtain an optimized scene target attribute. Specifically, the optimized scene target attribute is equal to the adaptive update rate multiplied by the predicted attribute plus (1-adaptive update rate) multiplied by the current scene target attribute. According to the above example, the optimized scene target attribute is 0.925x[0.65, 0.37, 0.59, 0.41, 0.70]+0.075x[0.72, 0.31, 0.56, 0.43, 0.68]=[0.654, 0.366, 0.588, 0.410, 0.699].

[0147] The optimized scene target attribute is input into a pre-trained classifier to output a current scene category identifier as a scene recognition result. The classifier can use a machine learning model such as a support vector machine, a random forest, or a neural network to determine the category to which the scene belongs according to the input target attribute vector. Taking the support vector machine as an example, the classifier has been trained by a large number of labeled scene target attribute samples and can effectively distinguish different types of scenes. Assuming that the classifier supports the identification of five scene categories, indoor, outdoor, traffic, nature, and crowd, the optimized scene target attribute [0.654, 0.366, 0.588, 0.410, 0.699] is input into the classifier, and the output scene category identifier is "nature", indicating that the current scene is identified as a natural scene.

[0148] In addition, to improve the recognition accuracy, the target transition probability matrix can be updated regularly in actual application. The system can record the results of each scene recognition and the corresponding target attribute, and when a certain number of new samples are accumulated, the target transition probability matrix is recalculated to better reflect the latest transition rules of the target attribute. For example, the system can recalculate the target transition probability matrix using the last 500 scene samples after processing 100 new scenes.

[0149] Through the above optimization process, the method fully utilizes the transition rules of the target attribute between scenes, significantly improves the accuracy and robustness of scene recognition, and especially in the case of noise or partial occlusion of the target attribute, the optimized scene target attribute can better reflect the characteristics of the real scene, thereby obtaining more reliable scene recognition results.

[0150] The robot scene understanding system based on visual deep learning according to the embodiment of the application comprises:

[0151] The first unit is configured to acquire a scene image collected by a robot vision sensor.

[0152] The second unit is configured to divide the scene image into a plurality of overlapping local areas, perform multi-layer pyramid feature decomposition on each local area to obtain local area features, establish a bidirectional feature transmission path between adjacent local areas, realize interactive enhancement of the local area features and global semantic features, recombine the enhanced local area features according to spatial positions to obtain a scene feature map.

[0153] The third unit is configured to perform target detection on the scene feature map, select a plurality of scene targets with a dispersion less than a consistency threshold, and extract semantic features of each scene target; calculate relationship strengths between the scene targets based on position offsets and semantic feature changes between adjacent frames of the scene targets, iteratively optimize the relationship strengths to obtain a target relationship matrix, and store the target relationship matrix in a scene database.

[0154] The fourth unit is configured to extract a historical scene target relationship matrix from the scene database, calculate a similarity between the current scene target relationship matrix and the historical scene target relationship matrix, select a historical scene with the highest similarity to calculate a target transition probability matrix; and optimize a current scene target attribute based on the target transition probability matrix to obtain a scene recognition result.

[0155] In a third aspect, an electronic device is provided, including:

[0156] a processor;

[0157] a memory for storing processor-executable instructions;

[0158] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0159] In a fourth aspect, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0160] The present application can be a method, device, system and / or computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein, which are used to perform various aspects of the present application.

[0161] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot scene understanding method based on visual deep learning, characterized in that, include: Acquire scene images captured by the robot's vision sensors; The scene image is divided into multiple overlapping local regions. Multi-level pyramid feature decomposition is performed on each local region to obtain local region features. A bidirectional feature transfer path is established between adjacent local regions to realize the interactive enhancement of local region features and global semantic features. The enhanced local region features are reorganized according to spatial location to obtain the scene feature map. Target detection is performed on the scene feature map, selecting multiple scene targets with a dispersion less than the consistency threshold, and extracting the semantic features of each scene target; Based on the position offset and semantic feature change between adjacent frames of the scene target, the relationship strength between the scene targets is calculated, the relationship strength is iteratively optimized to obtain the target relationship matrix, and the target relationship matrix is ​​stored in the scene database. Extract the target relationship matrix of historical scenes from the scene database, calculate the similarity between the target relationship matrix of the current scene and the target relationship matrix of the historical scenes, and select the historical scene with the highest similarity to calculate its target transition probability matrix; Based on the target transition probability matrix, the target attributes of the current scene are optimized to obtain the scene recognition result.

2. The method according to claim 1, characterized in that, The scene image is divided into multiple overlapping local regions. Multi-level pyramid feature decomposition is performed on each local region to obtain local region features, including: The image difference between each location point and its neighboring locations in the scene image is calculated. The image difference is then used to calculate the local region density of each location point through a Gaussian weighting function. The product of the local region density and a preset overlap coefficient is used as the overlap rate of each location point. Based on the overlap rate, the scene image is divided into multiple overlapping local regions. For each local region, a multi-layer feature pyramid is constructed. The scene image of the local region is used as the first layer pyramid image. Gaussian filtering is performed on the first layer pyramid image and pixels are extracted to obtain the second layer pyramid image. The above operation is repeated to obtain multiple layers of pyramid images. By using the local binary mode operator and the gray-level co-occurrence matrix, feature vectors of each level of the pyramid image are extracted and their information entropy is calculated. Based on the information entropy, the feature vectors of each level are weighted and fused to obtain local region features.

3. The method according to claim 1, characterized in that, A bidirectional feature transfer path is established between adjacent local regions to achieve interactive enhancement of local region features and global semantic features. The enhanced local region features are then reorganized according to spatial location to obtain a scene feature map, including: Calculate the Euclidean distance between the center points of adjacent local regions, and obtain the distance weight by mapping the Euclidean distance through a Gaussian function; Calculate the weighted difference between each local region feature and the features of its neighboring local regions, divide the weighted difference by the sum of the distance weights to obtain the feature increment; add the feature increment to the local region feature, and multiply it by the corresponding distance weight to obtain the bottom-up propagation feature; The global semantic features are modulated with each local region feature, and the modulated features are multiplied by the corresponding distance weights to obtain the top-down propagation features; the bottom-up propagation features and the top-down propagation features are weighted and summed to obtain the enhanced features of the local region. A location index matrix is ​​formed by statistically analyzing the local regions to which each location point belongs. The location index matrix is ​​then mapped using a Gaussian function to obtain feature aggregation weights. These feature aggregation weights are multiplied by the enhanced features of the corresponding local regions and normalized to obtain a scene feature map.

4. The method according to claim 1, characterized in that, Target detection is performed on the scene feature map, selecting multiple scene targets with a dispersion less than a consistency threshold, and extracting semantic features for each scene target; based on the positional offset and semantic feature changes between adjacent frames of the scene targets, the relationship strength between the scene targets is calculated, and the relationship strength is iteratively optimized to obtain a target relationship matrix, including: Target detection is performed on the scene feature map to obtain multiple sets of target box parameters and calculate their dispersion. Target box parameters with dispersion less than the consistency threshold are selected as scene targets. Extract the category information and attribute information of each scene target, calculate the mapping vector of the category information in the pre-trained semantic space, fuse the mapping vector with the attribute information to obtain semantic features, and calculate the semantic dependency strength between scene targets based on the semantic features; Calculate the position offset and semantic feature change of each scene target between adjacent frames. Based on the coordinated change of the position offset and semantic feature change, calculate the temporal correlation between the scene targets and perform weighted fusion with the semantic dependency strength to obtain the relationship strength between scene targets and generate an initial target relationship matrix. Based on the initial target relation matrix, the rate of change of relation strength in adjacent frames is calculated. Relationships with a rate of change of relation strength less than a stability threshold in multiple consecutive frames are marked as stable relations. The stable relations are used as global constraints to optimize the initial target relation matrix and obtain the target relation matrix.

5. The method according to claim 4, characterized in that, Calculate the positional offset and semantic feature difference of each scene target between adjacent frames. Based on the coordinated changes of the positional offset and semantic feature difference, calculate the temporal correlation between the scene targets, and perform weighted fusion with the semantic dependency strength to obtain the strength of the relationship between scene targets, including: Calculate the position offset of each scene target between adjacent frames, and calculate the semantic feature difference of each scene target between adjacent frames; Calculate the first influence gradient of the position offset of each scene target on the semantic feature difference of adjacent scene targets, and the second influence gradient of the position offset of the adjacent scene targets on the semantic feature difference of each scene target. Summate the first influence gradient and the second influence gradient to obtain the cooperative change intensity between scene targets. The intensity of the coordinated change is divided into multiple local segments according to the time window. The similarity between each local segment is calculated to obtain the local correlation. The continuity of the intensity of the coordinated change over the entire time series is calculated to obtain the global correlation. The local correlation and the global correlation are weighted and summed to obtain the temporal correlation between scene targets. The temporal correlation and the semantic dependency strength are weighted and fused to obtain the relationship strength between scene targets.

6. The method according to claim 1, characterized in that, Extract historical scene target relationship matrices from the scene database, calculate the similarity between the current scene target relationship matrix and the historical scene target relationship matrix, and select the historical scene with the highest similarity to calculate its target transition probability matrix, including: Extract the historical scene target relationship matrix from the scene database; perform matrix multiplication on the current scene target relationship matrix and the historical scene target relationship matrix to obtain the matrix trace; divide the matrix trace by the product of the two matrix norms to obtain the structural similarity. The topological similarity is obtained by calculating the difference between the current scene target relationship matrix and the historical scene target relationship matrix and performing an exponential mapping; the structural similarity and the topological similarity are weighted and combined to obtain the comprehensive similarity, and the historical scene with the highest comprehensive similarity is selected as the optimal historical scene; In the optimal historical scenario, matching features between adjacent scenario targets are extracted and the target state matching degree is calculated. The number of transitions between adjacent scenario targets in the optimal historical scenario is counted to obtain the historical transition frequency. The product of the target state matching degree and the historical transition frequency is normalized to obtain the target transition probability matrix.

7. The method according to claim 1, characterized in that, Based on the target transition probability matrix, the target attributes of the current scene are optimized to obtain the scene recognition result, including: The target transition probability matrix is ​​mapped to the target attribute of the current scene to obtain the predicted attribute, and the Euclidean distance between the predicted attribute and the target attribute of the current scene is calculated to obtain the attribute difference value. The Euclidean distance between the predicted attribute and the optimal historical scene target attribute is calculated to obtain the transition difference value. The attribute difference value and the transition difference value are weighted and combined to obtain the attribute consistency constraint. The adaptive update rate is calculated based on the attribute consistency constraint, and the predicted attribute and the current scene target attribute are weighted and combined according to the adaptive update rate to obtain the optimized scene target attribute. The optimized scene target attributes are input into a pre-trained classifier, and the current scene category identifier is output as the scene recognition result.

8. A robot scene understanding system based on visual deep learning, used to implement the method as described in any one of claims 1-7, characterized in that, include: The first unit is used to acquire scene images collected by the robot's vision sensors; The second unit is used to divide the scene image into multiple overlapping local regions, perform multi-level pyramid feature decomposition on each local region to obtain local region features, establish a bidirectional feature transfer path between adjacent local regions to realize the interactive enhancement of local region features and global semantic features, and reorganize the enhanced local region features according to spatial location to obtain a scene feature map. The third unit is used to perform target detection on the scene feature map, select multiple scene targets with a dispersion less than the consistency threshold, and extract the semantic features of each scene target. Based on the position offset and semantic feature change between adjacent frames of the scene target, the relationship strength between the scene targets is calculated, the relationship strength is iteratively optimized to obtain the target relationship matrix, and the target relationship matrix is ​​stored in the scene database. The fourth unit is used to extract the historical scene target relationship matrix from the scene database, calculate the similarity between the current scene target relationship matrix and the historical scene target relationship matrix, and select the historical scene with the highest similarity to calculate its target transition probability matrix. Based on the target transition probability matrix, the target attributes of the current scene are optimized to obtain the scene recognition result.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images