Image-based simplified robot visual servo control method

Abstract

The invention relates to the technical field of robots, and provides an image-based simplified robot visual servo control method, in the aspects of image acquisition and illumination compensation, adaptive multi-scale histogram equalization is combined with illumination simulation compensation based on a generative adversarial network (GAN), so that the problems of illumination diversity and complexity are effectively solved, image distortion is avoided, and the visual servo control precision of a robot is improved. The image feature identification degree is greatly improved in a complex illumination environment, a robot can obtain clear visual information, a multi-modal feature fusion optimization part and attention mechanism fusion automatically adjust different modal feature weights according to a scene, a cross-modal feature interaction network breaks the independence of each modal feature, and the multi-modal feature fusion optimization part and the attention mechanism fusion optimization part are integrated. The target recognition accuracy and robustness are enhanced through cooperation of the two methods, the feature fusion effect is improved, and for the shielding problem, the robot can optimize the shielding processing strategy according to the environment state and the decision result through the shielding decision based on reinforcement learning, and the adaptability is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of robotics, and specifically to a simplified robot visual servo control method based on images. Background Art

[0002] In the field of robot visual servo control, traditional methods have many technical problems in complex environments. The lighting conditions are complex and variable, ranging from strong direct sunlight to dim light, and traditional image acquisition and processing methods are difficult to cope with. When the lighting changes, the brightness and contrast of the image will change drastically, resulting in the failure of target recognition algorithms based on fixed thresholds or simple feature extraction. The robot cannot accurately identify the target, seriously affecting task execution.

[0003] The occlusion problem also plagues the robot vision system. In actual scenarios, the target object may be partially or completely occluded at any time. Traditional methods are difficult to accurately predict the target position during occlusion and cannot quickly resume tracking after occlusion is removed, resulting in the robot losing the target and the task being interrupted. In addition, single-modal features are difficult to fully describe the target in complex backgrounds, and traditional feature fusion methods cannot fully utilize the advantages of different modal features, resulting in insufficient accuracy and robustness of target recognition.

[0004] Therefore, a simplified robot visual servo control method based on images is proposed. By using adaptive multi-scale histogram equalization and lighting simulation compensation based on a generative adversarial network to improve the lighting effect, and using an attention mechanism fusion and a cross-modal feature interaction network to optimize feature fusion, the visual servo control performance of the robot in complex environments is improved. Summary of the Invention

[0005] Technical Problems to be Solved

[0006] Aiming at the deficiencies of the prior art, the present invention provides a simplified robot visual servo control method based on images.

[0007] Technical Solutions

[0008] To achieve the above-mentioned solution objectives, the present invention provides the following technical solutions: A simplified robot visual servo control method based on images, comprising the following steps:

[0009] S1: Image acquisition and lighting compensation optimization

[0010] S1.1: Adaptive multi-scale histogram equalization. First, the image is divided into N sub-regions of different sizes, and different processing strategies are implemented according to the different characteristics of small regions with rich details and large-area background regions in the image.

[0011] S1.2: Light simulation compensation based on generative adversarial network. Pre-train a light simulation generator G, whose input is the current light intensity data I light and the collected image I ori , and generate a simulated image During the training process, the generator G and the discriminator D play against each other. The generator tries to generate simulated images that are closer to the real ideal light images to deceive the discriminator, while the discriminator tries to accurately distinguish real images from generated images.

[0012] S2: Multimodal feature fusion optimization

[0013] S2.1: Attention mechanism fusion, automatically adjust the weights according to the contribution degree of each feature to target recognition in different scenarios.

[0014] S2.2: Cross-modal feature interaction network. Build a cross-modal feature interaction network to achieve deep interaction between different modal features. Let the color feature matrix be the texture feature matrix be the semantic feature matrix be where m is the number of feature dimensions, d c , d t , d s respectively represent the dimensions of color, texture, and semantic features. Since the dimensions of different modal features are different, first map them to the same dimensional space through linear transformation. The specific transformation formula is as follows:

[0015] C′ = W c C + b c

[0016] T′ = W t T + b t

[0017] S′ = W s S + b s

[0018] where, is the weight matrix, is the bias vector, d is the unified feature dimension. Subsequently, cross-modal interaction is carried out. Taking the interaction between color feature and texture feature as an example, the calculation formula is:

[0019]

[0020] where, is the interaction weight matrix, σ is the activation function, e represents element-wise multiplication. Through this interaction process, the color feature and the texture feature influence each other to generate a new feature representation C T , similarly, the interaction result T between texture and semantic features can be obtainedS and the interaction result S of semantics and color features C , finally, fuse the features after interaction to obtain the final fused feature vector F final :

[0021] F final = Concat(C T + C′, T S + T′, S C + S′).

[0022] S3: Optimization of target recognition and occlusion processing

[0023] S3.1: Occlusion decision-making based on reinforcement learning, introducing a reinforcement learning algorithm in the process of occlusion detection and processing.

[0024] S3.2: Target tracking and occlusion reasoning based on Transformer, modeling the historical trajectory of the target and the current image features, and obtaining a richer feature representation by stacking multiple Transformer layers.

[0025] Preferably, in the adaptive multi-scale histogram equalization, for the i-th sub-region (i = 1, 2,..., N), its gray-level histogram is represented by H i (k), where k represents the gray level, and the cumulative distribution function CDF i (k) is calculated by the formula: In the formula, L is the number of gray levels of the image. This formula is used to statistically calculate the proportion of pixel points with gray values less than or equal to k in the total number of pixel points in the i-th sub-region. The gray value I eq,i (x, y) of the sub-region image after equalization processing is determined by the following formula:

[0026] I eq,i (x, y) = round((L - 1) × CDF i (I i (x, y)))

[0027] where I i (x, y) represents the pixel gray value at coordinates (x, y) in the i-th sub-region of the original image, and I eq,i (x, y) is the gray value of the corresponding pixel after equalization.

[0028] Preferably, in the illumination simulation compensation based on the generative adversarial network, meanwhile, a discriminator D is introduced to judge the generated image and the difference between the real ideal illumination image I real . Their objective functions are defined as:

[0029] Preferably, in the adversarial training process where the generator G and the discriminator D play against each other, the quality of the generated images is continuously improved, approaching the real ideal illumination image, and then used for more accurate illumination compensation of the original images.

[0030] Preferably, in the fusion of the attention mechanism, let the color feature vector be C, the texture feature vector be T, and the semantic feature vector be S. The attention weight α of the color feature C is calculated by the following formula:

[0031]

[0032] where f C 、f T 、f S are functions for calculating the attention scores. Similarly, the attention weight α of the texture feature T and the attention weight α of the semantic feature S are calculated. The fused feature vector F is determined by the following formula:

[0033] F = α C C + α T T + α S S.

[0034] Preferably, for the occlusion decision based on reinforcement learning, the state space S is defined, which contains the information currently observed by the robot and reflects the environmental state of the robot. The action space A contains different occlusion handling strategies. When performing the action a t in the state s t , the robot will obtain the corresponding reward r t and transfer to the next state s t+1 .

[0035] Preferably, when the target is successfully relocked through prediction, a positive reward is given; when the decision-making fails and the target is lost, a negative reward is given. By continuously iteratively optimizing the policy π, the cumulative reward R is maximized. The calculation formula: where γ is the discount factor, with a value range between [0, 1], and T is the time step.

[0036] Preferably, for the target tracking and occlusion reasoning of the Transformer, for the sequence X = [x1, x2,..., x n composed of the historical trajectory of the target and the current image features, the Transformer calculates the representation of each position through the self-attention mechanism. The formula is:

[0037]

[0038] Among them, Q, K, and V are the query, key, and value matrices respectively, which are obtained by performing a linear transformation on the input sequence X, and d k is the dimension of the key matrix K. The self-attention mechanism dynamically assigns weights according to the correlation between different positions in the input sequence.

[0039] Beneficial effects

[0040] Compared with the prior art, the present invention provides a simplified robot vision servo control method based on images, having the following beneficial effects:

[0041] 1. In terms of image acquisition and light compensation, the adaptive multi-scale histogram equalization combined with the light simulation compensation based on the generative adversarial network (GAN) effectively solves the problems of light diversity and complexity, avoids image distortion, greatly improves the recognition rate of image features in a complex light environment, and enables the robot to obtain clear visual information. In the multi-modal feature fusion optimization part, the attention mechanism fusion automatically adjusts the weights of different modal features according to the scene, and the cross-modal feature interaction network breaks the independence of each modal feature. The two cooperate to enhance the accuracy and robustness of target recognition and improve the feature fusion effect.

[0042] 2. For the occlusion problem, the occlusion decision based on reinforcement learning enables the robot to optimize the occlusion processing strategy according to the environmental state and decision results, enhancing adaptability; the target tracking and occlusion reasoning based on Transformer uses its powerful sequence modeling ability to accurately predict the motion state of the target during occlusion, improving the accuracy of occlusion reasoning. Brief description of the drawings

[0043] Figure 1 is a schematic flow chart of the steps of the present invention. Specific implementation manners

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] Please refer to Figure 1 , the present invention proposes a simplified robot vision servo control method based on images, including the following content:

[0046] S1: Optimization of image acquisition and light compensation

[0047] S1.1: Adaptive multi-scale histogram equalization

[0048] In the robot vision system, the diversity and complexity of lighting conditions have a significant impact on image quality and subsequent processing results. To effectively cope with different lighting environments, an adaptive multi-scale histogram equalization method is adopted. This method first divides the image into N sub-regions of different sizes, aiming to implement different processing strategies for the small regions with rich details and the large-area background regions in the image according to their different characteristics.

[0049] For the i-th sub-region (i = 1, 2, …, N), its gray-level histogram is denoted by H i (k), where k represents the gray level. The cumulative distribution function CDF i (k) is calculated by the following formula:

[0050]

[0051] In the formula, L is the number of gray levels of the image. This formula is used to calculate the proportion of the number of pixels with gray values less than or equal to k in the total number of pixels in the i-th sub-region.

[0052] The gray value I eq,i (x, y) of the sub-region image after equalization processing is determined by the following formula:

[0053] I eq,i (x, y) = round((L - 1) × CDF i (I i (x, y)))

[0054] where I i (x, y) represents the gray value of the pixel with coordinates (x, y) in the i-th sub-region of the original image, and I eq,i (x, y) is the gray value of the corresponding pixel after equalization. In this way, under different lighting conditions, it can not only enhance the local contrast but also adjust the overall brightness, effectively avoiding the phenomenon of image distortion. Compared with the traditional histogram equalization method, it can retain more effective information in complex lighting environments and significantly improve the recognition rate of image features.

[0055] S1.2: Lighting simulation compensation based on generative adversarial network (GAN)

[0056] To further improve the ability of the robot to obtain clear visual information under extreme lighting conditions, a generative adversarial network (GAN) is introduced for lighting simulation compensation.

[0057] Pre-train a lighting simulation generator G, whose input is the current lighting intensity data I light and the collected image I ori , and generate a simulated image Specifically expressed as:

[0058]

[0059] Meanwhile, a discriminator D is introduced, whose function is to judge the generated image and the difference from the real ideal illumination image I real The objective functions of the two are defined as follows:

[0060]

[0061] During the training process, the generator G and the discriminator D play against each other. The generator is committed to generating simulated images closer to the real ideal illumination image to deceive the discriminator; the discriminator tries to accurately distinguish between real images and generated images. Through this adversarial training process, the quality of the generated images is continuously improved, making them closer to the real ideal illumination image, and then used for more accurate illumination compensation of the original images. For example, in a low-light environment, the generator can generate high-brightness images that conform to the scene characteristics, providing a compensation reference for the original images and helping the robot obtain clear visual information under extreme illumination conditions.

[0062] S2: Multi-modal feature fusion optimization

[0063] S2.1: Attention mechanism fusion

[0064] In the robot vision system, features of different modalities such as color, texture, and semantics have different importance in the target recognition process, and their importance levels change with the scene. To achieve more efficient feature fusion, an attention mechanism is introduced.

[0065] Let the color feature vector be C, the texture feature vector be T, and the semantic feature vector be S. The attention weight α of the color feature C is calculated by the following formula:

[0066]

[0067] where f C 、f T 、f S are functions for calculating attention scores. For example, they can be in the form of a simple linear transformation combined with an activation function. These functions evaluate the contribution of each feature to target recognition in the current scene by operating on different feature vectors. Similarly, the attention weight α T of the texture feature and the attention weight α S of the semantic feature can be calculated.

[0068] The fused feature vector F is determined by the following formula:

[0069] F = α C C + αT T + α S S

[0070] Through this dynamic adaptive fusion method, the weights can be automatically adjusted according to the contribution degree of each feature to target recognition in different scenarios, effectively improving the effectiveness of feature fusion. For example, in a scenario with complex texture but not obvious color features, this method can automatically increase the weight of texture features and reduce the weight of color features, thereby improving the accuracy of target recognition.

[0071] S2.2: Cross-modal Feature Interaction Network

[0072] To break the limitation that each modal feature is independent in traditional feature fusion, a cross-modal feature interaction network is constructed to achieve deep interaction between different modal features.

[0073] Let the color feature matrix be Let the texture feature matrix be Let the semantic feature matrix be where m is the number of feature dimensions, and d c , d t , d s represent the dimensions of color, texture, and semantic features respectively. Since the dimensions of different modal features are different, they need to be first mapped to the same dimensional space through linear transformation. The specific transformation formula is as follows:

[0074] C′ = W c C + b c

[0075] T′ = W t T + b t

[0076] S′ = W s S + b s

[0077] where, is the weight matrix, is the bias vector, and d is the unified feature dimension.

[0078] Subsequently, cross-modal interaction is carried out. Taking the interaction between color features and texture features as an example, the calculation formula is:

[0079]

[0080] where, is the interaction weight matrix, σ is the activation function (such as ReLU), and e represents element-wise multiplication. Through this interaction process, color features and texture features influence each other to generate a new feature representation C T .

[0081] Similarly, the interaction result T of texture and semantic features can be obtained S and the interaction result S of semantics and color features C .

[0082] Finally, the features after interaction are fused to obtain the final fused feature vector F final :

[0083] F final = Concat(C T + C′, T S + T′, S C + S′)

[0084] Through the above series of operations, the deep interaction between different modality features is achieved, breaking the limitations of traditional feature fusion, making the fused features more representative, and significantly enhancing the accuracy and robustness of target recognition. This cross-modal feature interaction network is innovative in the field of multi-modal feature fusion and provides a more powerful feature fusion ability for the robot vision system.

[0085] S3: Optimization of target recognition and occlusion handling

[0086] S3.1: Occlusion decision-making based on reinforcement learning

[0087] During the robot vision servo control process, the occlusion problem seriously affects the accuracy of target recognition and tracking. To effectively solve this problem, a reinforcement learning algorithm is introduced in the occlusion detection and processing process.

[0088] Define the state space S, which includes the image features currently observed by the robot, target position information, occlusion status, etc. These information comprehensively reflect the environmental state of the robot. The action space A includes different occlusion handling strategies, such as continuing to track the predicted position, switching to other alternative target detection methods, etc.

[0089] When executing the action a t in the state s t , the robot will obtain the corresponding reward r t and transfer to the next state s t+1 . If the target is successfully relocked through prediction, a positive reward is given; if the decision-making error leads to the loss of the target, a negative reward is given. By continuously iteratively optimizing the policy π, the cumulative reward R is maximized, and the calculation formula is as follows:[[]]

[0090]

[0091] Among them, γ is the discount factor, and its value range is usually between [0, 1], which is used to measure the importance of future rewards. The closer γ is to 1, the higher the degree of emphasis on future rewards; the closer γ is to 0, the more it focuses on the current immediate rewards. T is the time step, which limits the time range for calculating the cumulative reward. The policy π can be learned and updated through reinforcement learning algorithms (such as Q-learning, deep Q-network, etc.). The robot continuously tries different decision-making strategies, and based on the reward feedback obtained from the decision results, optimizes the occlusion processing strategy, thereby improving the adaptability to different occlusion situations.

[0092] S3.2: Transformer-based Object Tracking and Occlusion Reasoning

[0093] Transformer has powerful sequence modeling capabilities and can effectively capture the long-range dependencies of objects in the time series. Utilizing this feature, the historical trajectory of the object and the current image features are modeled.

[0094] For the sequence X = [x1, x2, …, x n , which is composed of the historical trajectory of the object and the current image features, Transformer calculates the representation of each position through the self-attention mechanism, and its formula is:

[0095]

[0096] Among them, Q, K, and V are the query, key, and value matrices respectively, which are obtained by linearly transforming the input sequence X, and d k is the dimension of the key matrix K. The self-attention mechanism can dynamically allocate weights according to the correlations between different positions in the input sequence, thereby extracting features more efficiently.

[0097] By stacking multiple Transformer layers, richer feature representations can be obtained. These feature representations not only fuse the current state information of the object but also contain historical trajectory information, which helps to more accurately predict the motion state of the object during occlusion. At the same time, the self-attention mechanism of Transformer can conduct a detailed analysis of the occluded and non-occluded regions in the image, improving the accuracy of occlusion reasoning and providing strong support for the robot's object tracking in occlusion situations.

[0098] To fully demonstrate the advantages of the present invention, embodiments (using the method of the present invention) and comparative examples (using traditional methods) will be set up and compared under three typical scenarios: light change, occlusion, and complex background. The algorithm performance will be evaluated from three key parameters: target recognition accuracy, tracking stability, and processing time, as follows:

[0099] 1. Illumination change scenario: In an indoor environment, simulate the process of the light changing from bright to dark and then to bright again. There are multiple objects with similar shapes in the scenario, and the target object is a red cube. Set the camera resolution to 1920×1080 and the frame rate to 30fps.

[0100] 2. Occlusion scenario: In a laboratory scenario, let the robot track a moving blue cylinder, and periodically occlude the target object with a black baffle during the process. Similarly, set the camera resolution to 1920×1080 and the frame rate to 30fps.

[0101] 3. Complex background scenario: In a warehouse scenario, it is filled with various goods, and the target object is a green cuboid. The camera resolution is 1920×1080 and the frame rate is 30fps.

[0102]

[0103]

[0104]

[0105] From the comparison of the above table data, it can be seen that in the illumination change scenario, the adaptive multi-scale histogram equalization and GAN-based illumination simulation compensation of the embodiments play a role and can stably identify the target, while the traditional method is greatly affected by illumination. In the occlusion scenario, the reinforcement learning and Transformer technology of the embodiments make the target recognition and tracking more stable, and the traditional method is prone to losing the target. In the complex background scenario, the multi-modal feature fusion of the embodiments has obvious advantages and can accurately identify the target, while the traditional method is easily interfered by the background. Overall, the method of the present invention has better performance in complex scenarios and is more suitable for robot visual servo control.

[0106] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A simplified robot visual servo control method based on image, characterized in that: The following steps are involved: S1: Image acquisition and illumination compensation optimization S1.1: Adaptive multi-scale histogram equalization, first divide the image into N sub-regions of different sizes, and implement differentiated processing strategies based on the different characteristics of small areas with rich details and large background areas in the image; S1.2: Based on the illumination simulation compensation of the generative adversarial network, a illumination simulation generator G is pre-trained, whose input is the current illumination intensity data I light And the collected image I ori , generate a simulated image During the training process, the generator G and the discriminator D compete with each other. The generator strives to generate simulated images that are closer to the real ideal lighting images to deceive the discriminator, while the discriminator strives to accurately distinguish between real images and generated images. S2: Multimodal feature fusion optimization S2.1: Attention mechanism fusion, automatically adjusting the weight according to the contribution of each feature to target recognition in different scenarios; S2.2: Cross-modal feature interaction network, build a cross-modal feature interaction network to achieve deep interaction between different modal features. Suppose the color feature matrix is The texture feature matrix is The semantic feature matrix is Where m is the number of feature dimensions, d c d t d s They represent the dimensions of color, texture, and semantic features respectively. Since the dimensions of different modal features are different, they are first mapped to the same dimensional space through linear transformation. The specific transformation formula is as follows: C′=W c C+b c T′=W t T+b t S′=W s S+b s in, is the weight matrix, is the bias vector, d is the unified feature dimension, and then cross-modal interaction is performed. Taking the interaction between color features and texture features as an example, the calculation formula is: in, is the interaction weight matrix, σ is the activation function, and e represents element-by-element multiplication. Through this interaction process, color features and texture features influence each other to generate a new feature representation C T Similarly, we can get the interaction result of texture and semantic features T S And the interaction result of semantic and color features S C Finally, the interactive features are fused to obtain the final fused feature vector F final : F final =Concat(C T +C′,T S +T′,S C +S′); S3: Optimization of target recognition and occlusion processing S3.1: Occlusion decision based on reinforcement learning, reinforcing learning algorithm is introduced in the process of occlusion detection and processing; S3.2: Transformer-based object tracking and occlusion reasoning, modeling the historical trajectory of the object and the current image features, and obtaining richer feature representation by stacking multiple Transformer layers.

2. The image-based simplified robot visual servo control method according to claim 1, characterized in that: In the adaptive multi-scale histogram equalization, for the i-th sub-region (i=1, 2, ..., N), its grayscale histogram is H i (k) represents, where k represents the gray level, the cumulative distribution function CDF i The calculation formula of (k) is: Where L is the grayscale level of the image. This formula is used to count the proportion of pixels with grayscale values ​​less than or equal to k in the i-th sub-region in the total number of pixels in the sub-region. The grayscale value of the sub-region image after equalization is I eq,i (x,y) is determined by the following formula: I eq,i (x,y)=round((L-1)×CDF i (I i (x,y))) Among them, I i (x, y) represents the grayscale value of the pixel with coordinates (x, y) in the i-th sub-region of the original image. eq,i (x, y) is the grayscale value of the corresponding pixel after equalization.

3. The image-based simplified robot visual servo control method according to claim 1, characterized in that: In the illumination simulation compensation based on the generative adversarial network, At the same time, the discriminator D is introduced to judge the generated image Compared with the real ideal lighting image I real The difference between them, the objective function of the two is defined as:

4. The image-based simplified robot visual servo control method according to claim 3, characterized in that: The adversarial training process between the generator G and the discriminator D continuously improves the quality of the generated images, which are close to the real ideal lighting images, and are then used to perform more accurate lighting compensation on the original images.

5. The image-based simplified robot visual servo control method according to claim 1, characterized in that: In the attention mechanism fusion, let the color feature vector be C, the texture feature vector be T, the semantic feature vector be S, and the attention weight of the color feature be α C Calculated by the following formula: Among them, f C 、f T 、f S is the function for calculating the attention score. Similarly, the texture feature attention weight α is calculated. T and semantic feature attention weight α S , the fused feature vector F is determined by the following formula: F=a C C+a T T+a S S.

6. The image-based simplified robot visual servo control method according to claim 1, characterized in that: The occlusion decision based on reinforcement learning defines the state space S, which contains the information currently observed by the robot and reflects the state of the environment in which the robot is located. The action space A contains different occlusion processing strategies. t Next, perform action a t , the robot will receive the corresponding reward r t , and transfer to the next state s t+1 .

7. The image-based simplified robot visual servo control method according to claim 6, characterized in that: The above method gives positive rewards to those who successfully re-lock the target through prediction, and negative rewards to those who make wrong decisions and lose the target. The cumulative reward R is maximized by continuously iterating and optimizing the strategy π. The calculation formula is: Among them, γ is the discount factor, the value range is between [0,1], and T is the time step.

8. The image-based simplified robot visual servo control method according to claim 1, characterized in that: The Transformer object tracking and occlusion reasoning, for a sequence X = [x1, x2, ..., x n ], Transformer calculates the representation of each position through the self-attention mechanism, and its formula is: Among them, Q, K, and V are query, key, and value matrices, respectively, which are obtained by linearly transforming the input sequence X. k is the dimension of the key matrix K, and the self-attention mechanism dynamically assigns weights according to the correlation between different positions in the input sequence.

Images