A robot vision control method and system based on spatio-temporal context perception

Abstract

The application relates to a robot vision control method and system based on spatiotemporal context perception and belongs to the field of vision control. The method comprises the following steps: acquiring an original image sequence, performing spatiotemporal context feature coding, and outputting a spatiotemporal context feature map; mapping the spatiotemporal context feature map into an initial attention map through a lightweight convolution network, introducing an attention diffusion process to output an attention potential field; obtaining a resource allocation matrix based on the attention potential field and performing calculation power allocation; obtaining a historical attention potential field based on the attention potential field, performing attention dynamics prediction according to the historical attention potential field, and obtaining a predicted attention potential field; and realizing prospective vision control based on the attention potential field and the predicted attention potential field. Through the introduction of spatiotemporal context perception coding, gated fusion, attention diffusion, nonlinear resource allocation and the like, the intelligent level of robot vision control is comprehensively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of visual control, and particularly relates to a robot visual control method and system based on spatiotemporal context perception. BACKGROUND

[0002] With the wide application of robot technology in automatic driving, industrial automation and intelligent monitoring, the performance of the visual control system is crucial. The existing robot visual control method mainly relies on static image feature extraction or simple time sequence processing, such as single-frame target detection and classification based on convolutional neural network (CNN) and motion detection based on optical flow or frame difference method. These methods are effective in some scenarios, but have obvious shortcomings: first, they often lack a deep understanding of the spatiotemporal context of the scene, and cannot dynamically integrate static features (such as texture, color) and dynamic features (such as motion patterns), resulting in attention dispersion or loss of key targets in complex environments; second, the resource allocation strategy often uses uniform or heuristic fixed allocation, which cannot adaptively adjust the computing power according to the importance of the scene, causing waste of computing resources or insufficient processing of key areas; in addition, traditional attention mechanisms are mostly limited to the current moment, lacking modeling and prediction of attention shift patterns, resulting in delayed or jittered control instructions, especially unstable robot tracking in high-speed motion scenarios. These shortcomings seriously limit the autonomy, real-time performance and robustness of robots in dynamic environments. SUMMARY

[0003] To solve the above problems in the prior art, the application provides a robot visual control method and system based on spatiotemporal context perception.

[0004] The object of the application can be achieved by the following technical solutions:

[0005] A robot visual control method based on spatiotemporal context perception, the implementation of the robot visual control method based on spatiotemporal context perception includes the following steps:

[0006] Step S1: Obtain the original image sequence and perform spatiotemporal context feature encoding, outputting a spatiotemporal context feature map;

[0007] Step S2: Map the spatiotemporal context feature map to an initial attention map through a lightweight convolutional network, and introduce an attention diffusion process to output an attention potential field;

[0008] Step S3: Obtain a resource allocation matrix based on the attention potential field, and perform computing power allocation;

[0009] Step S4: Obtain a historical attention potential field based on the attention potential field, and perform attention dynamics prediction according to the historical attention potential field to obtain a predicted attention potential field;

[0010] Step S5: achieving prospective visual control based on the attention potential field and the predicted attention potential field.

[0011] Preferably, the spatio-temporal context feature encoding in the step S1 is specifically:

[0012] Obtaining the original image sequence;

[0013] Feature extraction on the original image sequence by a deep neural network encoder to obtain a spatial feature map and a temporal feature map, the deep neural network encoder comprising a spatial encoder and a temporal encoder;

[0014] Fusing the spatial feature map and the temporal feature map by a spatio-temporal context gated fusion formula to obtain the spatio-temporal context feature map, the mathematical description of the spatio-temporal context gated fusion formula being , wherein, is the spatio-temporal context feature map at time t, is a Sigmoid activation function, is a gating weight matrix, is a spatial feature map, is a temporal feature map, represents concatenation along the feature channel dimension, is a gating bias, and is a transformation weight matrix, and is a transformation bias, is a hyperbolic tangent function, is a gating weight map, is an element-wise multiplication.

[0015] Preferably, the attention diffusion process in the step S2 is specifically:

[0016] Mapping the spatio-temporal context feature map into the initial attention map, the mathematical description being wherein, is the initial attention map at time t, is an attention weight, is an attention bias, is the spatio-temporal context feature map at time t;

[0017] Converting the initial attention map into a probability distribution map by spatial Softmax normalization;

[0018] Obtaining the attention potential field based on the initial attention map and the probability distribution map, the mathematical description being wherein, is the attention potential field at time t, is the probability distribution map, is the spatial Softmax normalization, is the diffusion coefficient, is the Plancherel operator, is the spatial discretization step.

[0019] Preferably, the step S3 specifically comprises:

[0020] Obtaining total computing resource and presetting minimum computing resource guaranteed by each position;

[0021] Obtaining a normalization factor based on the attention potential field, mathematically described as wherein, is the normalization factor, and M·N is the total number of positions, is the sharpness coefficient, is the attention potential value of the position, is the attention mean value, is the attention standard deviation;

[0022] Obtaining the resource allocation matrix based on the total computing resource, the minimum computing resource and the normalization factor, and performing computing power allocation, mathematically described as wherein, is the computing resource allocated to the position, is the minimum computing resource, is the total computing resource, is the attention potential value of the position.

[0023] Preferably, the attention dynamics prediction in the step S4 is specifically:

[0024] Learning the attention transfer rule through a recurrent neural network (RNN), mathematically described as wherein, is the predicted attention potential field at time t+1, is the RNN hidden state at time t+1, is the attention feature extraction function, is the RNN hidden state at time t, is the RNN parameter, is the attention potential field at time t.

[0025] Preferably, the prospective visual control in the step S5 is specifically:

[0026] Obtaining the current position coordinates of the robot;

[0027] Based on the attention potential field, the current world coordinates of the attention centroid are obtained, mathematically described as follows: ,in, As the current world coordinates of the center of mass of attention, for The attentional potential energy value of a location, where M·N is the total number of locations. for The world coordinates corresponding to the location;

[0028] Based on the predicted attention potential field, the predicted attention centroid world coordinates are obtained, mathematically described as follows: ,in, To predict the world coordinates of the attention centroid, for Predicted attention potential value for location;

[0029] Based on the robot's current position coordinates, the current attention centroid world coordinates, and the predicted attention centroid world coordinates, the robot's control commands are output, mathematically described as follows:

[0030] ,in, For control commands, , and These are proportional gain, differential gain, and integral gain, respectively. The coordinates of the robot's current position. It is the integral variable.

[0031] A robot vision control system based on spatiotemporal context awareness, used to execute the robot vision control method based on spatiotemporal context awareness described above, includes a spatiotemporal context awareness module, an attention potential field output module, a computing power allocation module, a prediction module, and a vision control module.

[0032] The spatiotemporal context awareness module is used to acquire the original image sequence and perform spatiotemporal context feature encoding to output a spatiotemporal context feature map.

[0033] The attention potential output module is used to map the spatiotemporal context feature map into an initial attention map through a lightweight convolutional network, and to introduce an attention diffusion process to output an attention potential field.

[0034] The computing power allocation module is used to obtain a resource allocation matrix based on the attention potential field and to allocate computing power.

[0035] The prediction module is used to obtain the historical attention potential field based on the attention potential field, and to perform attention dynamics prediction based on the historical attention potential field to obtain the predicted attention potential field.

[0036] The visual control module is configured to implement prospective visual control based on the attention potential field and the predicted attention potential field.

[0037] The beneficial effects of the present application are:

[0038] (1) By spatiotemporal context gating fusion, dynamically balancing static and dynamic features, reducing false positives and false negatives.

[0039] (2) By a nonlinear resource allocation strategy, high-attention areas are prioritized, ensuring basic monitoring while optimizing computing power usage.

[0040] (3) RNN-based attention dynamics prediction allows the robot to adjust its posture in advance, improving tracking performance. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to facilitate the understanding of those skilled in the art, the present application will be further described below with reference to the accompanying drawings.

[0042] Figure 1 A step flow chart of a robot visual control method based on spatiotemporal context perception according to the present application. DETAILED DESCRIPTION

[0043] In order to better understand the present application, various aspects of the present application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely descriptions of exemplary embodiments of the present application, and do not limit the scope of the present application in any way. Throughout the specification, the expression "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the words "approximately", "about", and similar expressions are used as terms of approximation and not as terms of degree, and are intended to account for the inherent deviations in a measured or calculated value that would be recognized by those of ordinary skill in the art. In addition, in the present application, the order of the processing described in each step does not necessarily represent the order in which the processes appear in actual operation, unless there is an explicit other limitation or can be derived from the context.

[0044] It should also be understood that expressions such as "include", "including", "have", "has", "contain" and / or "containing" and the like are open-ended terms that mean the presence of the stated feature, element and / or component, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when expressions such as "at least one of" appear after a list of listed features, they modify the entire list of features, not just the individual elements of the list. In addition, when describing embodiments of the present application, "may" means "one or more embodiments of the present application". And the word "exemplary" is intended to mean an example or illustration.

[0045] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0046] It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0047] Example 1:

[0048] Please refer to Figure 1 A robot vision control method based on spatio-temporal context perception, comprising:

[0049] Step S1: obtaining an original image sequence and performing spatio-temporal context feature encoding, outputting a spatio-temporal context feature map;

[0050] Step S2: mapping the spatio-temporal context feature map to an initial attention map of a single channel through a lightweight convolutional network, introducing an attention diffusion process for smoothing noise and enhancing spatial consistency, outputting an attention potential field;

[0051] Step S3: obtaining a resource allocation matrix based on the attention potential field, and performing computing power allocation;

[0052] Step S4: obtaining a historical attention potential field based on the attention potential field, performing attention dynamics prediction according to the historical attention potential field, and obtaining a predicted attention potential field;

[0053] Step S5: realizing prospective visual control based on the attention potential field and the predicted attention potential field.

[0054] In this embodiment, the spatio-temporal context feature encoding is specifically:

[0055] S101: obtaining the original image sequence by a robot;

[0056] S102: performing feature extraction on the original image sequence through two parallel deep neural network encoders, obtaining a spatial feature map and a temporal feature map, the deep neural network encoders including a spatial encoder and a temporal encoder, the spatial encoder being used for processing a current frame (i.e. extracting multi-scale spatial features through a convolutional neural network, the feature responses of which cover static clues such as color, texture and contrast; the temporal encoder being used for processing a continuous frame stack (i.e. , , …), motion features are extracted by 3D convolution or optical flow network, and the feature responses show the motion regions in the scene;

[0057] S103: The spatial feature map and the temporal feature map are fused by a spatiotemporal context gating fusion formula to obtain the spatiotemporal context feature map, and the mathematical description of the spatiotemporal context gating fusion formula is , wherein, is the spatiotemporal context feature map at time t, is a Sigmoid activation function, is a gating weight matrix, is a spatial feature map, is a temporal feature map, represents splicing along the feature channel dimension, is a gating bias, and is a transformation weight matrix, and is a transformation bias, is a hyperbolic tangent function, is a gating weight map, dynamically learns whether the spatial feature or the temporal feature should be trusted more at each spatial position, for example, on a static but high-contrast object, tends to 1, and depends on the spatial feature, on a low-texture but moving object, tends to 0, and depends on the temporal feature, is an element-wise multiplication.

[0058] In the embodiment, the attention diffusion process is specifically:

[0059] S201: The spatiotemporal context feature map is mapped into the initial attention map, and the mathematical description is wherein, is the initial attention map at time t, is an attention weight, is an attention bias, is the spatiotemporal context feature map at time t;

[0060] S202: The initial attention map is converted into a probability distribution map by spatial Softmax normalization, and the sum is 1, so that the attention between different images is comparable;

[0061] S203: The attention potential field is obtained based on the initial attention map and the probability distribution map, and the mathematical description is wherein, Let t be the attention potential field (the high potential energy region corresponds to the attention focus). This is a probability distribution diagram. For spatial Softmax normalization, The diffusion coefficient is used to control the smoothness. For the Pilates operator, The step size for spatial discretization.

[0062] In this embodiment, step S3 can be implemented through the following steps:

[0063] S301: Obtain the total computing power resources and preset the minimum computing power resources guaranteed for each location (i.e., the minimum computing power resources that can be obtained by each area to maintain basic scene monitoring).

[0064] S302: Obtain the normalization factor based on the attention potential field, mathematically described as follows: ,in, M is the normalization factor, and M·N is the total number of positions. This is the sharpness coefficient. The larger the area, the more resources are allocated to the high-attention area. for The attentional potential value of a location. The mean of attention ( ), The standard deviation of attention ( );

[0065] S303: The resource allocation matrix is ​​obtained based on the total computing power resources, the minimum computing power resources, and the normalization factor, mathematically described as follows: ,in, for Location-allocated computing resources For minimum computing power resources, For total computing power resources, for The attention potential value of a location. Therefore, for regions with significantly higher than average attention, the resources they acquire can increase exponentially, ultimately achieving nonlinear adaptive allocation of resources.

[0066] In this embodiment, the attention dynamics prediction specifically refers to:

[0067] The attention transfer pattern is learned through a recurrent neural network (RNN), mathematically described as follows: ,in, For the predicted attention potential field at time t+1, This represents the hidden state of the RNN at time t+1. For attention feature extraction function, The hidden state of the RNN at time t encodes the historical information of the attention evolution. These are the parameters of the RNN.

[0068] In this embodiment, the forward-looking visual control specifically refers to:

[0069] S501: Obtain the current position coordinates of the robot;

[0070] S502: Based on the attention potential field, obtain the current world coordinates of the attention centroid (i.e., the system will use a virtual attention point as the tracking target, and the position of this attention point is the weighted center of the attention potential field), mathematically described as follows: ,in, As the current world coordinates of the center of mass of attention, for The attentional potential energy value of a location, where M·N is the total number of locations. for The world coordinates corresponding to the location;

[0071] S503: Based on the predicted attention potential field, obtain the predicted attention centroid world coordinates, mathematically described as follows: ,in, To predict the world coordinates of the attention centroid, for Predicted attention potential value for location;

[0072] S504: Output control commands for the robot based on the robot's current position coordinates, the current attention centroid world coordinates, and the predicted attention centroid world coordinates. Mathematically, this can be described as follows:

[0073] ,in, For control commands, , and These are proportional gain, differential gain, and integral gain, respectively. The coordinates of the robot's current position. It is the integral variable.

[0074] Example 2:

[0075] A robot vision control system based on spatiotemporal context awareness includes a spatiotemporal context awareness module, an attention potential field output module, a computing power allocation module, a prediction module, and a vision control module.

[0076] The spatiotemporal context awareness module is used to acquire the original image sequence and perform spatiotemporal context feature encoding to output a spatiotemporal context feature map.

[0077] The attention potential field output module is configured to map the spatio-temporal context feature map into an initial attention map in a single channel through a lightweight convolutional network, introduce an attention diffusion process for smoothing noise and enhancing spatial consistency, and output an attention potential field;

[0078] The computing power allocation module is configured to obtain a resource allocation matrix based on the attention potential field and perform computing power allocation;

[0079] The prediction module is configured to obtain a historical attention potential field based on the attention potential field, perform attention dynamics prediction according to the historical attention potential field, and obtain a predicted attention potential field;

[0080] The visual control module is configured to realize prospective visual control based on the attention potential field and the predicted attention potential field.

[0081] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some minor changes or modifications to the above disclosed technical content without departing from the scope of the technical solution of the present application, and any simple modification, equivalent change and modification of the above embodiment according to the technical essence of the present application still belongs to the scope of the technical solution of the present application.

Claims

1. A robot vision control method based on spatiotemporal context awareness, characterized in that, Includes the following steps: Step S1: Obtain the original image sequence and perform spatiotemporal context feature encoding to output a spatiotemporal context feature map; the spatiotemporal context feature encoding specifically involves: obtaining the original image sequence; extracting features from the original image sequence using a deep neural network encoder to obtain a spatial feature map and a temporal feature map, wherein the deep neural network encoder includes a spatial encoder and a temporal encoder; The spatial feature map and the temporal feature map are fused using a spatiotemporal context-gated fusion formula to obtain the spatiotemporal context feature map. The mathematical description of the spatiotemporal context-gated fusion formula is as follows: , ,in, This is the spatiotemporal context feature map at time t. It is the Sigmoid activation function. For the gated weight matrix, For spatial feature maps, For time feature maps, This indicates splicing along the feature channel dimension. For gated bias. and To transform the weight matrix, and To change the bias, It is the hyperbolic tangent function. For the gating weight graph, This is element-wise multiplication; Step S2: Map the spatiotemporal context feature map to an initial attention map using a lightweight convolutional network, and introduce an attention diffusion process to output an attention potential field; Step S3: Obtain the resource allocation matrix based on the attention potential field and allocate computing power accordingly; Step S4: Obtain the historical attention potential field based on the historical attention potential field, and perform attention dynamics prediction based on the historical attention potential field to obtain the predicted attention potential field; Step S5: Implement forward-looking visual control based on the attention potential field and the predicted attention potential field.

2. The robot vision control method based on spatiotemporal context awareness according to claim 1, characterized in that, The attention diffusion process in step S2 is specifically as follows: Mapping the spatiotemporal context feature map to the initial attention map can be mathematically described as follows: ,in, Here is the initial attention map at time t. For attention weights, For attention bias, This is the spatiotemporal context feature map at time t; The initial attention map is converted into a probability distribution map by spatial Softmax normalization; The attention potential field is obtained based on the initial attention map and the probability distribution map, mathematically described as follows: ,in, Let be the attention potential field at time t. This is a probability distribution diagram. For spatial Softmax normalization, The diffusion coefficient is... For the Pilates operator, The step size for spatial discretization.

3. The robot vision control method based on spatiotemporal context awareness according to claim 1, characterized in that, Step S3 specifically includes: Obtain the total computing power resources and preset the minimum computing power resources guaranteed for each location; The normalization factor is obtained based on the attention potential field, mathematically described as follows: ,in, M is the normalization factor, and M·N is the total number of positions. This is the sharpness coefficient. for The attentional potential value of a location. The mean of attention. The standard deviation of attention; The resource allocation matrix is ​​obtained based on the total computing power, the minimum computing power, and the normalization factor, and computing power is allocated accordingly. Mathematically, this is described as follows: ,in, for Location-allocated computing resources For minimum computing power resources, For total computing power resources, for The attention potential value of the location.

4. The robot vision control method based on spatiotemporal context awareness according to claim 1, characterized in that, The attention dynamics prediction in step S4 specifically refers to: The attention transfer pattern is learned through a recurrent neural network (RNN), mathematically described as follows: ,in, For the predicted attention potential field at time t+1, This represents the hidden state of the RNN at time t+1. For attention feature extraction function, Let t be the hidden state of the RNN at time t. For RNN parameters, Let be the attention potential field at time t.

5. The robot vision control method based on spatiotemporal context awareness according to claim 1, characterized in that, The forward-looking visual control mentioned in step S5 specifically refers to: Obtain the robot's current position coordinates; Based on the attention potential field, the current world coordinates of the attention centroid are obtained, mathematically described as follows: ,in, As the current world coordinates of the center of mass of attention, for The attentional potential energy value of a location, where M·N is the total number of locations. for The world coordinates corresponding to the location; Based on the predicted attention potential field, the predicted attention centroid world coordinates are obtained, mathematically described as follows: ,in, To predict the world coordinates of the attention centroid, for Predicted attention potential value for location; Based on the robot's current position coordinates, the current attention centroid world coordinates, and the predicted attention centroid world coordinates, the robot's control commands are output, mathematically described as follows: ,in, For control commands, , and These are proportional gain, differential gain, and integral gain, respectively. The coordinates of the robot's current position. It is the integral variable.

6. A robot vision control system based on spatiotemporal context awareness, characterized in that, The system is applied to the robot vision control method based on spatiotemporal context awareness as described in any one of claims 1-5, including a spatiotemporal context awareness module, an attention potential field output module, a computing power allocation module, a prediction module, and a vision control module; The spatiotemporal context awareness module is used to acquire the original image sequence and perform spatiotemporal context feature encoding to output a spatiotemporal context feature map. The attention potential output module is used to map the spatiotemporal context feature map into an initial attention map through a lightweight convolutional network, and to introduce an attention diffusion process to output an attention potential field. The computing power allocation module is used to obtain a resource allocation matrix based on the attention potential field and to allocate computing power. The prediction module is used to obtain the historical attention potential field based on the attention potential field, and to perform attention dynamics prediction based on the historical attention potential field to obtain the predicted attention potential field. The visual control module is used to implement forward-looking visual control based on the attention potential field and the predicted attention potential field.

Images