Robot motion control system and method based on computer vision

Abstract

The invention relates to the field of robot motion control, and particularly discloses a robot motion control system and method based on computer vision, and the method comprises the steps: firstly obtaining a table tennis serving panoramic image collected by a vision sensor and historical table tennis training data of a user, and then carrying out the deep learning technology; and feature extraction and correlation analysis are carried out on the two types of motion trails, and finally, the motion trails of the table tennis partner training robot are generated through the generator, so that a personalized training mode is provided according to habits, strength and weakness of the user, training is more efficient and targeted, and the flexibility and convenience of training are improved.

Description

Technical Field

[0001] The present application relates to the field of robot motion control, and more specifically, to a robot motion control system and method based on computer vision. Background Art

[0002] Table tennis is a popular competitive sport with the characteristics of quick response and high skills, which attracts a large number of enthusiasts. However, in traditional table tennis training, many table tennis enthusiasts find it difficult to find suitable players for effective training due to their busy daily lives, which limits their skill improvement. In this context, table tennis training robots came into being. Table tennis training robots can provide training opportunities at any time to help table tennis enthusiasts practice.

[0003] However, although there are many table tennis training robots on the market at this stage, most of these table tennis training robots can only perform a single ball delivery and cannot achieve more precise serve control.

[0004] Therefore, a robot motion control system and method based on computer vision is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a robot motion control system and method based on computer vision, which first obtains a panoramic image of table tennis serve and a user's table tennis historical practice data collected by a visual sensor, then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally generates a motion trajectory of a table tennis training robot through a generator, thereby providing a personalized training method according to the user's habits, strengths and weaknesses, making the training more efficient and targeted, and improving the flexibility and convenience of training.

[0006] According to one aspect of the present application, there is provided a robot motion control system based on computer vision, comprising:

[0007] A user table tennis data acquisition module, used to acquire a panoramic image of table tennis serve and user table tennis historical practice data collected by a visual sensor;

[0008] A user table tennis data extraction module, used for extracting a table tennis serve panoramic attention feature vector and a table tennis history practice global feature vector from the table tennis serve panoramic image collected by the visual sensor and the user's table tennis history practice data;

[0009] The table tennis training robot motion trajectory generation module is used to generate the table tennis training robot motion trajectory based on the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector.

[0010] According to another aspect of the present application, a robot motion control method based on computer vision is provided, comprising:

[0011] Acquire a panoramic image of table tennis serve and historical table tennis practice data of the user collected by a visual sensor;

[0012] Extracting a table tennis serve panoramic attention feature vector and a table tennis history practice global feature vector from the table tennis serve panoramic image collected by the visual sensor and the user's table tennis history practice data;

[0013] Based on the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector, a motion trajectory of the table tennis training robot is generated.

[0014] Compared with the prior art, the present application provides a computer vision-based robot motion control system and method, which first obtains a panoramic image of table tennis serve and a user's historical table tennis practice data collected by a visual sensor, then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally uses a generator to generate the motion trajectory of the table tennis training robot, thereby providing personalized training methods based on the user's habits, strengths and weaknesses, making the training more efficient and targeted, and improving the flexibility and convenience of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 4 is a block diagram of a robot motion control system based on computer vision according to an embodiment of the present application.

[0017] Figure 2 4 is a block diagram of a user table tennis data extraction module in a computer vision-based robot motion control system according to an embodiment of the present application.

[0018] Figure 3 The present invention is a block diagram of a table tennis serve panoramic image feature extraction unit in a computer vision-based robot motion control system according to an embodiment of the present application.

[0019] Figure 4 The present invention is a block diagram of a table tennis training robot motion trajectory generation module in a computer vision-based robot motion control system according to an embodiment of the present application.

[0020] Figure 5 Flowchart of a robot motion control method based on computer vision according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0022] Figure 1 : is a block diagram of a robot motion control system based on computer vision according to an embodiment of the present application. Figure 1 As shown, according to the robot motion control system 100 based on computer vision according to the embodiment of the present application, it includes: a user table tennis data acquisition module 110, which is used to acquire a table tennis serve panoramic image collected by a visual sensor and a user's table tennis historical practice data; a user table tennis data extraction module 120, which is used to extract a table tennis serve panoramic attention feature vector and a table tennis historical practice global feature vector from the table tennis serve panoramic image collected by the visual sensor and the user's table tennis historical practice data; a table tennis training robot motion trajectory generation module 130, which is used to generate a table tennis training robot motion trajectory based on the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector.

[0023] In the above-mentioned robot motion control system 100 based on computer vision, the user table tennis data acquisition module 110 is used to obtain the table tennis serve panoramic image and the user table tennis historical practice data collected by the visual sensor. It should be understood that table tennis is a very popular competitive sport, known for its quick response and high skills, which attracts many enthusiasts to participate. However, in traditional training, many table tennis enthusiasts have difficulty finding suitable players for effective practice due to their busy lives, which limits the improvement of their skills. In this case, table tennis training robots came into being, which can provide training opportunities for enthusiasts at any time to help them practice. Although there are many table tennis training robots on the market, most robots can only perform a single ball delivery function and lack precise serve control capabilities.

[0024] Based on this, in the technical solution of the present application, a robot motion control solution based on computer vision is provided by acquiring a panoramic image of table tennis serve collected by a visual sensor and the user's historical table tennis practice data. Among them, the panoramic image can provide real-time serve information, including features such as the trajectory, speed and rotation of the table tennis ball, so that the robot can analyze and understand the current training scene. In addition, the user's historical practice data contains past performance records, practice frequency and skill improvement, which can provide valuable background information for the robot to help it consider the user's progress and challenges when designing training programs. Here, the panoramic image data can use high-resolution cameras and other visual devices to capture the complete serve process. The user's historical practice data can be collected through a database, and these applications can record the user's training times, scores and feedback, and aggregate and store the data in the cloud. In this way, the table tennis training robot can effectively obtain and utilize visual information and historical data, thereby providing a more accurate and personalized training experience, greatly improving the user's skills and training effects. Therefore, in the technical solution of the present application, the motion trajectory of the table tennis training robot is generated by acquiring the panoramic image of table tennis serve collected by the visual sensor and the user's table tennis historical practice data, and combining deep learning technology. Thus, a personalized training program is provided according to the user's habits, strengths and weaknesses. This not only makes the training more efficient and targeted, but also improves the flexibility and convenience of training.

[0025] In the above-mentioned robot motion control system 100 based on computer vision, the user table tennis data extraction module 120 is used to extract the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector from the table tennis serve panoramic image collected by the visual sensor and the user table tennis historical practice data. It should be understood that by comprehensively using the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector, the training robot can achieve more accurate serve control and personalized feedback, help users effectively improve their skills, and achieve better training results.

[0026] Figure 2 FIG. 1 is a block diagram of a user table tennis data extraction module in a robot motion control system based on computer vision according to an embodiment of the present application. Figure 2 As shown, in a specific embodiment of the present application, the user table tennis data extraction module 120 includes: a table tennis serve panoramic image feature extraction unit 121, used to perform feature extraction on the table tennis serve panoramic image collected by the visual sensor to obtain the table tennis serve panoramic attention feature vector; a user table tennis history practice data feature extraction unit 122, used to perform feature extraction on the user table tennis history practice data to obtain the table tennis history practice global feature vector.

[0027] It should be understandable that the process of feature extraction of the panoramic image of table tennis serve collected by the visual sensor involves the use of deep learning technology, especially convolutional neural network (CNN) to analyze and identify key elements in the image, and further apply it to the robot decision-making model, so that it can make more accurate responses and adjustments during training, thereby improving the user's training experience and skill level.

[0028] Furthermore, historical practice data usually includes information such as the user's batting frequency, hit rate, performance of different technical actions, duration of training, etc. These data reflect the user's overall skill level and development trend. Through feature extraction, complex raw data can be converted into a vector form that is easy to analyze and process. In a specific embodiment of the present application, feature extraction is performed using statistical analysis methods and machine learning techniques. First, the training data is cleaned to remove outliers and noise to ensure the accuracy of the data. Then, a preliminary feature set is formed by calculating statistical features such as average batting speed, batting success rate, and frequency of various batting methods. In addition, cluster analysis and dimensionality reduction techniques can also be used to identify the commonalities and individuality shown by users in training and extract more representative global features. For example, principal component analysis (PCA) is used to reduce high-dimensional data to low dimensions and extract the features that best reflect the user's skill level. By combining these historical features, the robot can adjust the training strategy in real time and improve the user's training effect, thereby achieving more efficient skill improvement.

[0029] Figure 3 FIG. 1 is a block diagram of a table tennis serve panoramic image feature extraction unit in a robot motion control system based on computer vision according to an embodiment of the present application. Figure 3 As shown, in a specific embodiment of the present application, the table tennis serve panoramic image feature extraction unit 121 includes: a table tennis serve panoramic feature encoding subunit 1211, used to perform feature encoding on the table tennis serve panoramic image collected by the visual sensor to obtain a table tennis serve panoramic attention feature map; a panoramic attention feature maximum pooling subunit 1212, used to perform maximum pooling on the table tennis serve panoramic attention feature map to obtain the table tennis serve panoramic attention feature vector.

[0030] It should be understandable that the process of feature encoding of the panoramic image of table tennis serve collected by the visual sensor usually relies on the convolutional neural network (CNN) in deep learning. Through this process, the complex information in the original image is converted into a more expressive feature map, thereby providing a rich visual information foundation for the table tennis training robot, promoting it to achieve more efficient training feedback and personalized guidance, thereby helping users improve their skills.

[0031] Furthermore, performing maximum pooling on the panoramic attention feature map of table tennis serve is an important step in converting high-dimensional feature information into a low-dimensional, more concise and effective representation. Specifically, the maximum pooling process can effectively retain the most significant features while removing unnecessary noise and redundant information by extracting the maximum value in each local area of ​​the feature map. In the specific implementation, the feature map will be divided into multiple small areas (such as 2x2 or 3x3 windows), and the maximum value will be calculated in each window. This operation not only reduces the feature dimension, but also enhances the translation invariance of the feature, making the model more robust to slight position changes of objects in the image. In this way, the final panoramic attention feature vector of table tennis serve will contain the most important visual information in the image and can concisely describe the key dynamic features of the serve. In addition, maximum pooling can help improve computational efficiency, making subsequent processing and analysis more efficient.

[0032] In a specific embodiment of the present application, the table tennis serve panoramic feature encoding subunit 1211 includes: passing the table tennis serve panoramic image collected by the visual sensor through a table tennis serve panoramic image pixel enhancement module to obtain a table tennis serve panoramic pixel enhanced image; passing the table tennis serve panoramic pixel enhanced image through a table tennis serve panoramic pixel enhanced attention encoder to obtain the table tennis serve panoramic attention feature map.

[0033] It should be understood that the process of processing the panoramic image of table tennis serve collected by the visual sensor through the pixel enhancement module of the panoramic image of table tennis serve usually involves image enhancement techniques, including contrast adjustment, brightness enhancement, noise removal and sharpening operations. Specifically, first, the pixel enhancement module analyzes the input original image and identifies low contrast and blurred areas in the image caused by insufficient light or environmental factors. Then, the brightness and contrast of the image are adjusted using methods such as histogram equalization, so that the table tennis and other key elements are more obvious in the image, thereby improving the clarity of the visual information. Next, the module may apply a denoising algorithm, such as median filtering or Gaussian filtering, to remove noise from the image to ensure that subsequent analysis is based on cleaner data. In addition, the edge features of the image are enhanced by an edge detection algorithm (such as Canny edge detection) to make the outline of the table tennis ball more obvious, thereby facilitating the subsequent feature extraction and recognition process. This enhancement process improves the visibility of the image and greatly enriches the visual information.

[0034] Furthermore, the panoramic pixel-enhanced image of table tennis serve is passed through the panoramic pixel-enhanced attention encoder of table tennis serve. Among them, the panoramic pixel-enhanced attention encoder of table tennis serve is based on the deep learning framework and aims to extract more representative features from the enhanced image. Specifically, first, the pixel-enhanced image is processed by multiple layers of convolution, and the network uses convolution kernels to automatically extract spatial features at different levels, including the shape, speed, and user's hitting action of the table tennis ball. Through multi-layer feature learning, the encoder can identify and focus on important areas in the image, such as the trajectory of the ball and the user's swing posture, which are crucial for understanding the serving action. Then, the attention mechanism is introduced to strengthen the model's attention to key features. This mechanism enhances the features related to the serving action by assigning different weights, while suppressing background noise and irrelevant information, thereby generating a more accurate attention feature map. Finally, the encoded feature map can not only capture the important dynamics of the serving process, but also effectively extract feature information related to the training target. Specifically, in the process of passing the table tennis serve panoramic pixel enhanced image through a table tennis serve panoramic pixel enhanced attention encoder to obtain the table tennis serve panoramic attention feature map, it includes: using the convolution encoding part of the table tennis serve panoramic pixel enhanced attention encoder to perform deep convolution encoding on the table tennis serve panoramic pixel enhanced image to obtain an initial convolution feature map; inputting the initial convolution feature map into the spatial attention part of the table tennis serve panoramic pixel enhanced attention encoder to obtain a spatial attention map; passing the spatial attention map through a Softmax activation function to obtain a spatial attention feature map; calculating the position point multiplication of the spatial attention feature map and the initial convolution feature map to obtain the table tennis serve panoramic attention feature map. More specifically, the initial convolutional feature map is input into the spatial attention part of the table tennis serve panoramic pixel enhanced attention encoder to obtain a spatial attention map, including: performing average pooling and maximum pooling on the initial convolutional feature map along the channel dimension to obtain an average feature matrix and a maximum feature matrix; cascading and channel-adjusting the average feature matrix and the maximum feature matrix to obtain a channel feature matrix; and using the convolution layer of the spatial attention feature map to convolutionally encode the channel feature matrix to obtain a spatial attention map.

[0035] In a specific embodiment of the present application, the panoramic attention feature maximum pooling subunit includes: using Spring Initializr to create a Spring Boot project, selecting Spring Web and Spring Data dependencies; creating a data model class to represent a table tennis serve panoramic attention feature map and a table tennis serve panoramic attention feature vector; implementing the maximum pooling logic in the service layer, processing the input table tennis serve panoramic attention feature map, and obtaining the table tennis serve panoramic attention feature vector; creating a REST controller, defining an endpoint to receive the table tennis serve panoramic attention feature map, and calling the service layer method for processing, returning the table tennis serve panoramic attention feature vector; configuring the Spring Boot application startup class to ensure that all components are loaded correctly, and running the Spring Boot application to ensure that the service starts normally and listens to the specified port.

[0036] The deployment code for defining the data model is shown below.

[0037]

[0038] The deployment code for implementing the maximum pooling logic is shown below.

[0039]

[0040] The deployment code for creating the REST controller is as follows.

[0041]

[0042]

[0043] The deployment code for configuring and running the Spring Boot application is shown below.

[0044]

[0045] In this way, a service that implements maximum pooling is deployed using the Spring Web mechanism. The service receives the panoramic attention feature map of table tennis serve and returns the panoramic attention feature vector of table tennis serve after maximum pooling. Among them, a new Spring Boot project is created using Spring Initializr, and the necessary dependencies of Spring Web and Spring Data are added. This step ensures the infrastructure of the project and provides a solid foundation for subsequent development and deployment. Then, by creating two data model classes, AttentionFeatureMap and AttentionFeatureVector, to represent the panoramic attention feature map and feature vector of table tennis serve respectively. These classes define the data structure and provide a clear interface for subsequent data processing. Next, the MaxPoolingService class is created in the service layer to implement the maximum pooling logic. This class contains the maxPooling method, which receives the input of the AttentionFeatureMap type, extracts the maximum value of each row by traversing the input two-dimensional list, and stores these maximum values ​​in the object of the AttentionFeatureVector type and returns them. This step ensures efficient processing of the feature map and extracts key features. Next, by creating the MaxPoolingController class, a POST request endpoint named processFeatureMap is defined. This endpoint receives a request body of the AttentionFeatureMap type, calls the maxPooling method of the MaxPoolingService class to process it, and returns

[0046] Response of type AttentionFeatureVector. This step implements the interface of the Web service, so that external systems can call the maximum pooling function through HTTP requests. Further, configure the startup class MaxPoolingApplication of the SpringBoot application, use the @SpringBootApplication annotation mark, and call the SpringApplication.run method in the main method to start the application. This step ensures the correct startup and operation of the application, listens to the specified port, and waits for external requests. In this way, a maximum pooling processing service based on the Spring Web mechanism is implemented. The service can efficiently process the panoramic attention feature map of table tennis serve, extract key features, and generate corresponding feature vectors. Through the RESTful API interface, external systems can easily call the service to realize feature extraction and processing. The entire system has a clear architecture, modular design, easy expansion and maintenance, and provides reliable technical support for the generation of motion trajectories for table tennis training robots.

[0047] In a specific embodiment of the present application, the user table tennis history practice data feature extraction unit 122 includes: passing the user table tennis history practice data through a table tennis history practice text encoder based on a context encoder to obtain multiple table tennis history practice feature vectors; arranging the multiple table tennis history practice feature vectors in two dimensions into a table tennis history practice text feature matrix; passing the table tennis history practice text feature matrix through a convolutional neural network model as a table tennis history practice text feature encoder to obtain the table tennis history practice global feature vector.

[0048] It should be understood that the user's table tennis historical practice data is processed by a table tennis historical practice text encoder based on a context encoder, and context information can be used to extract valuable features from the historical practice data in order to better understand the user's training background and needs. Specifically, first of all, considering that the historical practice data usually contains multi-dimensional information such as the user's practice frequency, duration, practice content, performance score, etc., therefore, in the technical solution of the present application, these data are processed by the encoder and converted into structured text features. In this process, the encoder analyzes the user's practice mode, identifies key behaviors and trends, and combines context information, such as the environment, goals and feedback during practice, to generate a feature vector suitable for text analysis. Next, the context-based encoding technology can make full use of the user's historical information and help the encoder focus on the most important factors for improving skills. For example, if the user's serve performance is poor within a certain period of time, the encoder will identify and emphasize this part of the data, generate corresponding text descriptions, and provide effective training feedback and suggestions. In this way, the user's historical practice data is not only converted into multiple text data with deep analysis value, but also can form a dynamic knowledge base to continuously provide users with personalized training guidance. Ultimately, these text data will provide valuable references for training robots and coaches, helping them to develop targeted training plans, thereby more effectively improving the user's table tennis skills and optimizing the training effect. Specifically, in the process of passing the user's table tennis history practice data through a table tennis history practice text encoder based on a context encoder to obtain multiple table tennis history practice feature vectors, it includes: performing word segmentation processing on the user's table tennis history practice data to obtain a history practice word sequence; using the embedding layer of the table tennis history practice text encoder based on the context encoder to map each history practice word in the history practice word sequence to a word embedding vector to obtain a sequence of history practice word embedding vectors; using the converter-based Bert model of the table tennis history practice text encoder based on the context encoder to perform global context semantic encoding on the sequence of history practice word embedding vectors to obtain multiple table tennis history practice feature vectors.

[0049] Furthermore, arranging multiple table tennis history practice feature vectors in two dimensions into a table tennis history practice text feature matrix helps to effectively integrate multi-dimensional training information, thereby improving the training efficiency and accuracy of the model. Specifically, multiple table tennis history practice feature vectors are arranged in rows or columns to form a two-dimensional matrix structure. In this process, each row represents a set of practice data, and each column corresponds to a specific feature. Such an arrangement can clearly show the differences and similarities between different exercises. In this structured way, the feature matrix is ​​not only easy to process by the machine learning algorithm, but also can intuitively reflect the user's training trajectory and evolution.

[0050] Furthermore, considering that convolutional neural networks have superior performance in image and text processing and are widely used in feature extraction, in the technical solution of the present application, the table tennis history practice text feature matrix is ​​processed by a convolutional neural network (CNN) model as a table tennis history practice text feature encoder to obtain a table tennis history practice global feature vector, extract deep features and capture complex patterns. Specifically, first, the table tennis history practice text feature matrix is ​​arranged in two dimensions, presenting rich structural information, and the CNN model can use its convolution layer to extract local features through sliding convolution kernels. This process will generate an activation map for each feature local area, thereby identifying key patterns related to table tennis training. Then, through activation functions such as ReLU, nonlinear factors are further introduced to enhance the model's learning ability for complex features. As the data is processed through multiple convolutional layers and pooling layers, the model gradually extracts more abstract high-level features, thereby obtaining a global understanding of the user's historical training behavior. Finally, after being processed by the fully connected layer, the final generated table tennis history practice global feature vector can effectively aggregate information from different training stages, reflecting the user's overall training level and potential improvement direction. Specifically, in passing the table tennis history practice text feature matrix through a convolutional neural network model serving as a table tennis history practice text feature encoder to obtain the table tennis history practice global feature vector, it includes: using each layer of the convolutional neural network model serving as a table tennis history practice text feature encoder to perform convolution processing, mean pooling processing based on the local feature matrix, and nonlinear activation processing on the input data in the forward pass of the layer so that the last layer of the convolutional neural network model serving as the table tennis history practice text feature encoder outputs the table tennis history practice global feature vector, wherein the input of the convolutional neural network model serving as the table tennis history practice text feature encoder is the table tennis history practice text feature matrix.

[0051] In the above-mentioned robot motion control system 100 based on computer vision, the table tennis training robot motion trajectory generation module 130 is used to generate the table tennis training robot motion trajectory based on the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector. It should be understood that the table tennis serve panoramic attention feature vector provides a detailed analysis of the serve moment, including the trajectory, speed and rotation direction of the table tennis ball, etc. This information enables the robot to judge the characteristics of the serve in real time and predict the landing point of the table tennis ball. On the other hand, the table tennis historical practice global feature vector contains the user's performance data in multiple trainings, such as hit rate, batting method and habits, etc., which reflect the user's overall skill level and progress. By combining these two feature vectors, the training robot can construct a dynamic motion trajectory model and adjust its motion path in real time to cope with different serve situations. In addition, through the feedback mechanism, the robot can continuously adjust the motion strategy according to the user's performance to achieve adaptive training. This dynamic motion trajectory generation based on feature vectors not only improves the responsiveness and training effect of the training robot, but also enhances the user's training fun and sense of participation, making each training more in line with the user's actual needs.

[0052] Figure 4 FIG. 1 is a block diagram of a motion trajectory generation module of a table tennis training robot in a robot motion control system based on computer vision according to an embodiment of the present application. Figure 4 As shown, in a specific embodiment of the present application, the table tennis training robot motion trajectory generation module 130 includes: a training robot data feature association unit 131, which is used to perform perspective co-projection space mapping on the table tennis serve panoramic attention feature vector and the table tennis history practice global feature vector based on the continuous high-dimensional regression space attribute to obtain the training robot motion trajectory generation feature vector; a motion trajectory generation unit 132, which is used to pass the training robot motion trajectory generation feature vector through a generator to generate the table tennis training robot motion trajectory.

[0053] It should be understood that by associating the panoramic attention feature vector of table tennis serve and the global feature vector of table tennis historical practice, the sparring robot can more comprehensively understand the user's current state and historical background. In specific implementation, feature splicing or weighted fusion can be used. Feature splicing is to directly connect the two vectors to form a new feature vector, so that it contains the instant features of serving and the global information of historical practice. This method is simple and direct, and can retain the independence of the two features to a certain extent. In addition, it is possible to consider introducing weighted fusion, by assigning different weights to the two feature vectors, combining the importance of current practice with the relevance of historical training, to generate a more balanced motion trajectory generation feature vector. For example, when the user's serving performance is significantly better than the historical level, the weight of the serving feature can be appropriately increased, and vice versa, the influence of the historical feature can be increased. After such an association process, the final generated sparring robot motion trajectory generation feature vector can not only reflect the user's serving status in real time, but also combine its historical training situation to provide a more accurate guidance basis for the generation of motion trajectory. In this way, the training robot can analyze the user's serving dynamics and training history in real time, thereby generating an optimized motion trajectory to help users improve training results and reach a higher skill level.

[0054] In particular, in the technical solution of the present application, the panoramic attention feature vector of table tennis serve is obtained by processing the images collected by the visual sensor, which emphasizes the key visual information in the process of serving, such as the position of the ball, the trajectory of motion and the posture of the server. These features are usually temporal and spatial, and can reflect the dynamic state of motion. The global feature vector of table tennis historical practice is obtained by analyzing the historical practice data of the user, and mainly contains the summary of the user's behavior patterns and technical movements in past practice. These data are text-based and may involve information such as time, frequency and results. They focus on the user's training rules and preferences, reflecting a more stable behavior pattern. When performing feature fusion, the heterogeneity of the panoramic attention feature vector of table tennis serve and the global feature vector of table tennis historical practice may lead to the mismatch between the feature manifold and the feature space. Among them, the smoothness of the feature manifold refers to the continuity and consistency between points in the feature space. If there is a mismatch when the dynamic features of table tennis serve are fused with the static features of historical practice, it may lead to the formation of irregular shapes in the feature space, thereby affecting the smoothness and predictability of the motion trajectory of the generated sparring robot. In this way, the complexity introduced by feature heterogeneity in the feature fusion process will affect the motion trajectory generation of the sparring robot, making the generated feature vector lack of certainty, thereby reducing the practicality and effect of the robot in table tennis sparring. Based on this, in the technical solution of the present application, the panoramic attention feature vector of table tennis serve and the global feature vector of table tennis historical practice are mapped to the perspective co-projection space based on the attributes of continuous high-dimensional regression space.

[0055] The present invention relates to a method for generating a motion trajectory feature vector of a sparring robot by performing a perspective co-projection spatial mapping on a panoramic table tennis serve attention feature vector and a global table tennis history practice feature vector based on the attribute of a continuous high-dimensional regression space to obtain a motion trajectory feature vector of the sparring robot, including: constructing a table tennis panoramic-historical practice significant distance topological matrix between the panoramic table tennis serve attention feature vector and the global table tennis history practice feature vector; constructing a table tennis panoramic-historical practice significant eigenvalue granularity association matrix between the panoramic table tennis serve attention feature vector and the global table tennis history practice feature vector; and mapping the panoramic table tennis serve attention feature vector to the global table tennis history practice feature vector based on the panoramic table tennis-historical practice significant distance topological matrix. The practice significant eigenvalue granularity correlation matrix is ​​topologically modulated to obtain a table tennis panoramic-historical practice significant topological modulation correlation matrix; the table tennis panoramic-historical practice significant topological modulation correlation matrix is ​​used as a perspective co-projection space, and the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector are projected into the perspective co-projection space to obtain a perspective-modulated table tennis serve panoramic attention feature vector and a perspective-modulated table tennis historical practice global feature vector; the perspective-modulated table tennis serve panoramic attention feature vector and the perspective-modulated table tennis historical practice global feature vector are fused to obtain the training robot motion trajectory generation feature vector.

[0056] The following formula is used to perform perspective co-projection spatial mapping based on the continuous high-dimensional regression space attribute on the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector to obtain the training robot motion trajectory generation feature vector; wherein, the formula is:

[0057] M t =D(V1,V2)

[0058]

[0059] V c =α·V1′+β·V2′

[0060] Wherein, V1 represents the panoramic attention feature vector of table tennis serve, V2 represents the global feature vector of table tennis historical practice, and D(V1, V2) is the distance topological matrix between the panoramic attention feature vector of table tennis serve and the global feature vector of table tennis historical practice, that is, d i,j =d(v 1i ,v 2j ), d i,j is the eigenvalue of the position (i, j) in the distance topology matrix, d(v 1i ,v 2j) represents the distance between the eigenvalue of the ith position of the table tennis serve panoramic attention feature vector and the eigenvalue of the jth position of the table tennis history practice global feature vector, and V1 and V2 are both column vectors, M t represents the table tennis panorama-historical practice significant distance topology matrix, T represents the transpose of the vector, Represents matrix multiplication, M v represents the table tennis panorama-historical practice significant eigenvalue granularity correlation matrix, M c represents the table tennis panorama-historical practice significant topological modulation association matrix, V1′ represents the panoramic attention feature vector of table tennis serve modulated by perspective, V2′ represents the global feature vector of table tennis historical practice modulated by perspective, α and β represent weighted hyperparameters, V c Represents the fused feature vector.

[0061] Based on this, in the technical solution of the present application, the panoramic attention feature vector of table tennis serve and the global feature vector of table tennis historical practice are subjected to perspective co-projection space mapping based on the attributes of continuous high-dimensional regression space, which first constructs the table tennis panoramic-historical practice significant distance topological matrix between the panoramic attention feature vector of table tennis serve and the global feature vector of table tennis historical practice. This step utilizes the idea of ​​graph theory, takes the eigenvalues ​​of each position in the panoramic attention feature vector of table tennis serve and the global feature vector of table tennis historical practice as nodes, and uses the distance metric function to determine the low-dimensional embedding expression of the edges between nodes, so as to construct a feature expression that can quantify the topological correlation relationship between the panoramic attention feature vector of table tennis serve and the global feature vector of table tennis historical practice. This step is crucial to understanding the spatial distribution of data points and their mutual connection, and also provides basic structural information for subsequent steps.

[0062] Next, a table tennis panoramic-historical practice significant feature value granularity association matrix between the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector is constructed. This step focuses on exploring the correlation between the internal attributes of the feature vector. In a specific example, the product between the table tennis serve panoramic attention feature vector and the transposed vector of the table tennis historical practice global feature vector is calculated to obtain the table tennis panoramic-historical practice significant feature value granularity association matrix. Furthermore, the table tennis panoramic-historical practice significant feature value granularity association matrix is ​​topologically modulated based on the table tennis panoramic-historical practice significant distance topological matrix to obtain the table tennis panoramic-historical practice significant topological modulation association matrix. That is, the feature value granularity association information of the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector is driven to wander on the distance topological map to construct a low-dimensional modulation perspective co-projection space domain for mapping the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector to a continuous high-dimensional regression space attribute.

[0063] Next, the table tennis panorama-historical practice significant topological modulation association matrix is ​​used as the perspective co-projection space, and the table tennis serve panorama attention feature vector and the table tennis historical practice global feature vector are projected into the perspective co-projection space to obtain the perspective modulated table tennis serve panorama attention feature vector and the perspective modulated table tennis historical practice global feature vector. That is, the key to this technical solution is to find an implicit third space with modulation capability, in which as much information of the original high-dimensional data as possible is retained, so that the converted new feature vector not only reduces the dimension, but also maintains the relative position relationship between the original features.

[0064] Finally, the perspective-modulated table tennis serve panoramic attention feature vector and the perspective-modulated table tennis historical practice global feature vector are fused to obtain the training robot motion trajectory generation feature vector. Fusion can be achieved in a variety of forms, ranging from simple arithmetic average to complex ensemble learning algorithms, depending on the application scenario and the goals pursued. The final training robot motion trajectory generation feature vector combines the main features of the two sets of input data and provides a more comprehensive data representation, which is not only conducive to improving the performance of machine learning models, but also promotes the ability of cross-modal or multi-source data analysis, thereby significantly enhancing the understanding and utilization efficiency of complex data structures.

[0065] Furthermore, the motion trajectory generation feature vector integrates the user's current serve characteristics and historical practice background, providing a rich information foundation. The generator plays a key role in this process, responsible for converting these features into specific motion trajectories. First, the generator can use deep learning models, such as generative adversarial networks (GANs), to capture dynamic changes and complex patterns in time series. These models can learn the underlying laws of motion trajectories based on the input feature vectors, thereby generating motion paths that meet user needs. Specifically, the generator first receives the training robot motion trajectory generation feature vector as input. These features include not only the instantaneous state when serving, but also the user's performance in historical training and potential improvement points. After being processed by a multi-layer neural network, the generator will feature map these features and convert abstract information into specific motion instructions, such as moving direction, speed, and acceleration. This process can be continuously optimized through the back-propagation algorithm to ensure that the generated motion trajectory is both in line with physical laws and can maximize the user's training goals. The generated motion trajectory of the table tennis training robot not only takes into account the instantaneous feedback of serving, but is also closely related to the user's historical performance, enabling the robot to adjust its motion strategy in time during training. Ultimately, this feature vector-based generation method ensures that the training robot can adapt to the user's training needs in real time and continuously optimize the training effect, thereby helping users improve their table tennis skills and gain a better training experience. Through this intelligent motion trajectory generation mechanism, the training robot achieves efficient interaction and precise guidance with the user.

[0066] In summary, the embodiment of the present application first obtains a panoramic image of a table tennis serve and the user's historical table tennis practice data collected by a visual sensor, and then uses deep learning technology to perform feature extraction and correlation analysis on the two. Finally, a generator is used to generate the motion trajectory of the table tennis training robot, thereby providing personalized training methods based on the user's habits, strengths and weaknesses, making the training more efficient and targeted, and improving the flexibility and convenience of training.

[0067] As described above, the robot motion control system 100 based on computer vision according to the embodiment of the present application can be implemented in various terminal devices. In one example, the robot motion control system 100 based on computer vision can be integrated into the terminal device as a software module and / or a hardware module. For example, the robot motion control system 100 based on computer vision can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the robot motion control system 100 based on computer vision can also be one of the many hardware modules of the terminal device.

[0068] Alternatively, in another example, the computer vision-based robot motion control system 100 and the terminal device may also be separate devices, and the computer vision-based robot motion control system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0069] Figure 5 FIG. 1 is a flow chart of a robot motion control method based on computer vision according to an embodiment of the present application. Figure 5 As shown, according to the robot motion control method based on computer vision according to the embodiment of the present application, it includes: S110, acquiring a panoramic image of table tennis serve collected by a visual sensor and a user's table tennis historical practice data; S120, extracting a table tennis serve panoramic attention feature vector and a table tennis historical practice global feature vector from the table tennis serve panoramic image collected by the visual sensor and the user's table tennis historical practice data; S130, generating a motion trajectory of a table tennis training robot based on the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector.

[0070] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned robot motion control method based on computer vision have been described in the above reference. Figures 1 to 4 The description of the computer vision-based robot motion control system has been introduced in detail, and therefore, its repeated description will be omitted.

Claims

1. A robot motion control system based on computer vision, characterized in that: include: A user table tennis data acquisition module, used to acquire a panoramic image of table tennis serve and user table tennis historical practice data collected by a visual sensor; A user table tennis data extraction module, used for extracting a table tennis serve panoramic attention feature vector and a table tennis history practice global feature vector from the table tennis serve panoramic image collected by the visual sensor and the user's table tennis history practice data; The table tennis training robot motion trajectory generation module is used to generate the table tennis training robot motion trajectory based on the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector.

2. The robot motion control system based on computer vision according to claim 1, characterized in that: The user table tennis data extraction module comprises: A table tennis serve panoramic image feature extraction unit, used for extracting features from the table tennis serve panoramic image collected by the visual sensor to obtain the table tennis serve panoramic attention feature vector; The user table tennis historical practice data feature extraction unit is used to extract features from the user table tennis historical practice data to obtain the table tennis historical practice global feature vector.

3. The robot motion control system based on computer vision according to claim 2, characterized in that: The table tennis serve panoramic image feature extraction unit comprises: A table tennis serve panoramic feature encoding subunit, used for performing feature encoding on the table tennis serve panoramic image collected by the visual sensor to obtain a table tennis serve panoramic attention feature map; The panoramic attention feature maximum pooling subunit is used to perform maximum pooling on the table tennis serve panoramic attention feature map to obtain the table tennis serve panoramic attention feature vector.

4. The robot motion control system based on computer vision according to claim 3, characterized in that: The table tennis serve panoramic feature encoding subunit comprises: Passing the table tennis serve panoramic image collected by the visual sensor through a table tennis serve panoramic image pixel enhancement module to obtain a table tennis serve panoramic pixel enhanced image; The table tennis serve panoramic pixel enhanced image is passed through a table tennis serve panoramic pixel enhanced attention encoder to obtain the table tennis serve panoramic attention feature map.

5. The robot motion control system based on computer vision according to claim 4, characterized in that: The panoramic attention feature maximum pooling subunit includes: Use Spring Initializr to create a Spring Boot project, selecting Spring Web and Spring Data dependencies; Create a data model class to represent the ping pong serve panoramic attention feature map and ping pong serve panoramic attention feature vector; Implement the maximum pooling logic in the service layer to process the input ping pong serve panoramic attention feature map to obtain the ping pong serve panoramic attention feature vector; Create a REST controller, define an endpoint to receive the panoramic attention feature map of table tennis serve, call the service layer method to process it, and return the panoramic attention feature vector of table tennis serve; Configure the startup class of the Spring Boot application, ensure that all components are loaded correctly, and run the Spring Boot application to ensure that the service starts normally and listens to the specified port.

6. The robot motion control system based on computer vision according to claim 5, characterized in that: The user table tennis historical practice data feature extraction unit comprises: Passing the user's table tennis practice history data through a table tennis practice history text encoder based on a context encoder to obtain a plurality of table tennis practice history feature vectors; Arranging the plurality of table tennis history practice feature vectors in two dimensions into a table tennis history practice text feature matrix; The table tennis history practice text feature matrix is ​​passed through a convolutional neural network model as a table tennis history practice text feature encoder to obtain the table tennis history practice global feature vector.

7. The robot motion control system based on computer vision according to claim 6, characterized in that: The table tennis training robot motion trajectory generation module includes: The training robot data feature association unit is used to perform perspective co-projection space mapping on the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector based on the continuous high-dimensional regression space attribute to obtain the training robot motion trajectory generation feature vector; The motion trajectory generating unit is used to generate a feature vector of the motion trajectory of the training robot through a generator to generate a motion trajectory of the table tennis training robot.

8. The robot motion control system based on computer vision according to claim 7, characterized in that: The training robot data feature association unit includes: Constructing a table tennis panorama-historical practice significant distance topological matrix between the table tennis serve panorama attention feature vector and the table tennis historical practice global feature vector; Constructing a table tennis panorama-historical practice significant eigenvalue granularity association matrix between the table tennis serve panorama attention feature vector and the table tennis historical practice global feature vector; Based on the table tennis panorama-historical practice significant distance topological matrix, the table tennis panorama-historical practice significant eigenvalue granularity association matrix is ​​topologically modulated to obtain a table tennis panorama-historical practice significant topological modulation association matrix; Taking the table tennis panorama-historical practice significant topological modulation association matrix as the perspective co-projection space, projecting the table tennis serve panorama attention feature vector and the table tennis historical practice global feature vector into the perspective co-projection space to obtain the perspective-modulated table tennis serve panorama attention feature vector and the perspective-modulated table tennis historical practice global feature vector; The perspective-modulated table tennis serve panoramic attention feature vector and the perspective-modulated table tennis historical practice global feature vector are integrated to obtain the motion trajectory generation feature vector of the training robot.

9. A robot motion control method based on computer vision, characterized in that: include: Acquire a panoramic image of table tennis serve and historical table tennis practice data of the user collected by a visual sensor; Extracting a table tennis serve panoramic attention feature vector and a table tennis history practice global feature vector from the table tennis serve panoramic image collected by the visual sensor and the user's table tennis history practice data; Based on the table tennis serve panoramic attention feature vector and the table tennis historical practice global feature vector, a motion trajectory of the table tennis training robot is generated.

10. The robot motion control method based on computer vision according to claim 9, characterized in that: Extracting a table tennis serve panoramic attention feature vector and a table tennis history practice global feature vector from the table tennis serve panoramic image collected by the visual sensor and the user's table tennis history practice data, including: Performing feature extraction on the table tennis serve panoramic image collected by the visual sensor to obtain the table tennis serve panoramic attention feature vector; Feature extraction is performed on the user's table tennis historical practice data to obtain the table tennis historical practice global feature vector.

Images