An event camera and vision camera cooperative-based motion training evaluation system and method

Abstract

The application discloses a motion training evaluation system and method based on cooperation of an event camera and a visual camera, and is applied to the field of computer vision, and aims at the problems that an existing training evaluation system based on an optical camera cannot capture fast motion trajectories and limb shapes, cannot effectively evaluate action quality, and cannot provide effective training guidance; the application provides continuous human motion events in the form of event streams, can achieve millisecond-level motion response, is not affected by motion blur effects of high-speed moving objects, and can provide a higher dynamic range, and can provide more effective motion training evaluation in scenes with strong light, backlight, and sharp changes in brightness. Through cooperation of the visual camera and the event camera, not only the accuracy of fast actions can be evaluated, but also standard action teaching and training in the form of visual scene interaction can be performed.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and specifically relates to a technology for recognizing and assessing the quality of fast movements using an event camera and a depth camera. Background Technology

[0002] Optical sensor-based visual cameras accumulate pixel brightness values ​​over a period of time. However, their frame rate is limited by the photosensitivity and data readout rate. Furthermore, they suffer from drawbacks such as susceptibility to ambient lighting, high hardware performance requirements, high power consumption, and significant latency. They also exhibit motion blur and long response times for fast-moving objects, making them unsatisfactory for action recognition and evaluation. Event cameras, on the other hand, are special cameras sensitive only to changes in pixel brightness. They are a novel type of neuromorphic visual sensor. Their pulsed neuromorphic system features asynchronous operation, low power consumption, low computational throughput, high dynamic range, high temporal resolution, and dynamic relationship with the scene environment. They can provide microsecond-level response signals, and their reaction speed and dynamic range are significantly superior to existing conventional optical cameras, making them well-suited for high-dynamic target detection applications.

[0003] Movement training and quality assessment are currently hot topics in competitive sports research. Examples include the standardization of badminton striking motions, including the correctness of shoulder, elbow, and wrist trajectories, which affects power generation and shot effectiveness; table tennis serving and smashing motions, trajectory analysis, and footwork analysis; and the standardization analysis of gymnastic movements. Because competitive sports involve fast and complex movements (requiring coordinated participation from various body parts), optical camera-based training and assessment systems cannot capture rapid movement trajectories and limb forms, thus failing to effectively evaluate movement quality and provide effective training guidance. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a motion training and evaluation system and method based on the collaboration of an event camera and a visual camera. By collaborating with the event camera, not only can the accuracy of rapid movements be evaluated, but standard movements can also be taught and trained in a visually interactive manner.

[0005] One of the technical solutions adopted in this invention is: a motion training and evaluation system based on the collaboration of an event camera and a visual camera, comprising: an event acquisition terminal, a host, and an interactive terminal;

[0006] The event acquisition terminal includes an event camera and an event stream acquisition module, and the event stream acquisition module is connected to the event camera via USB.

[0007] The host includes an event trajectory extraction module, an action recognition module, and an action quality evaluation module; the event trajectory extraction module extracts event trajectories based on the output of the event acquisition terminal, the action recognition module performs action recognition based on the extracted event trajectories, and the action quality evaluation module evaluates the recognized actions;

[0008] The interactive terminal is used to display the action recognition results;

[0009] The event acquisition terminal also includes an optical camera, which is used to transmit the acquired video stream to the interactive terminal for action playback.

[0010] The event camera is used to capture motion events in the target area in real time.

[0011] Each event is represented as a quadruple, denoted as [x,y,t,p], where x and y are pixel coordinates, t is the timestamp, and p is the polarity, which is used to indicate an increase or decrease in brightness.

[0012] The optical camera is an RGB camera.

[0013] The second technical solution adopted in this invention is: a motion training evaluation method based on the collaboration of an event camera and a visual camera, comprising:

[0014] S1. Represent the motion events of the target area captured in real time by the event camera as follows:

[0015] e t =(x,y,t,p)

[0016] Where x and y are pixel coordinates, t is the timestamp, and p is the polarity, which is used to indicate an increase or decrease in brightness;

[0017] S2. Convert the captured motion events into an event stream image;

[0018] S3. For each event stream image, extract the event trajectory;

[0019] S4. Extract features from the time trajectory;

[0020] S5. Input the extracted features into the Long Short-Term Memory network and output the action category and action similarity.

[0021] Step S3 is as follows:

[0022] A1, in event stream image E i In the middle, for each event point P i (x,y,v i If the number of times the event occurs is v i If the value is less than the threshold ε, then update v at that time point. i=0, eliminating insignificant points in motion; ε = α·average(v i ), v i >0, 1≥α≥0, where α is the filtration coefficient;

[0023] A2. The event stream image E processed in step A1 i Perform morphological dilation followed by erosion and closing operations.

[0024] A3. The event stream image E processed in step A2 i For each event point, eliminate outlier noise points.

[0025] Step A3 specifically includes the following sub-steps:

[0026] A31. Search the neighborhood of each event point in the current event stream image step by step according to the neighborhood expansion method, and calculate the Euclidean distance between each event point and its nearest point in the neighborhood; for the first point found, calculate its Euclidean distance;

[0027] A32. Based on the nearest Euclidean distances of all event points in the current event flow image, obtain the nearest Euclidean distance table corresponding to the current event flow image;

[0028] A33. Based on the nearest Euclidean distance table corresponding to the current event stream image, calculate the mean and standard deviation of the Euclidean distance distribution;

[0029] A34. If the nearest Euclidean distance to an event point is greater than the sum of the mean and the standard deviation, then that event point is removed as an outlier, and the remaining event image is denoted as ET. i ;

[0030] A35. Using the Canny edge detection method from ET i Extract its edge information and save it as edge graph E i Each event flow graph corresponds to an edge graph.

[0031] The beneficial effects of this invention are as follows: This invention provides continuous human motion events in the form of an event stream, achieving millisecond-level motion response and is unaffected by the motion blur effect of high-speed moving objects. It also provides a higher dynamic range and can provide more effective motion training and evaluation in scenarios with strong light, backlight, and drastic changes in brightness. The method of this invention reconstructs the trajectory of the action based on the position of each micro-stage of the action and obtains trajectory features through deep learning methods, thereby realizing action recognition and quality assessment. Attached Figure Description

[0032] Figure 1 This is the system structure of the present invention;

[0033] Figure 2 This is a schematic diagram of event stream image generation provided in an embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram illustrating the action event space description provided in an embodiment of the present invention. Detailed Implementation

[0035] To facilitate understanding of the technical content of this invention by those skilled in the art, the following description, in conjunction with the accompanying drawings, further illustrates the invention.

[0036] like Figure 1 The system shown consists of three parts: a data acquisition terminal, a host computer, and an interactive large screen.

[0037] The functions of each part are briefly described below:

[0038] Event acquisition terminal: Includes an event camera and an event stream acquisition module. The event stream acquisition module is connected to the event camera via USB and captures motion events in the target area in real time, outputting high dynamic range event stream data. Each event is represented by a tuple containing four elements: pixel coordinates x, y, timestamp t (time stamp of the event), and polarity p, i.e., [x, y, t, p], where polarity indicates an increase or decrease in brightness.

[0039] Optical camera: RGB camera, capturing motion video for visual interaction; the optical camera connects to the host via a USB 3.0 interface and a USB cable.

[0040] The host unit, primarily a high-configuration computer system, is mainly responsible for analyzing event stream data, determining the type of actions, evaluating the accuracy of actions, and outputting the error between the actions and standard actions. It includes three main functional modules: event trajectory extraction, action recognition, and action quality evaluation, based on a deep learning-based action recognition model and a standard action pattern library. The host unit connects to the interactive screen via an HDMI interface, and can store, encode, decode, and output video. The host unit and the event acquisition terminal are connected via a USB 3.0 interface and a USB cable.

[0041] Interactive large screen: Provides display outputs such as action playback, action error display, real-time action video playback, and action teaching. Enables user-friendly human-computer interaction.

[0042] The method of the present invention includes the following steps:

[0043] S1: Event Stream Acquisition and Representation

[0044] Step S1 mainly includes event representation and converting the events captured by the camera into event stream images, and preprocessing the event stream images.

[0045] (1) Event representation

[0046] e t =(x,y,t,p)

[0047] An event is represented by a quadruple (x, y, t, p), where x and y are the two-dimensional coordinates of the pixel where the event occurred, t is the timestamp of the event in milliseconds, and p is the polarity of the event signal: -1 for a decrease in brightness exceeding a threshold, +1 for an increase in brightness exceeding a threshold, and no event is generated if the brightness change does not exceed the threshold. In this embodiment, the threshold is set to 0.2, indicating that the brightness change does not exceed 20%. The threshold is a settable system parameter that needs to be adjusted under different lighting conditions. Generally, this parameter is set through calibration after system startup. If the acquired event stream image has a lot of noise, the threshold needs to be increased, and vice versa.

[0048] (2) Event Stream Image

[0049] Because the event camera is high dynamic range, it outputs an event stream. Saving the output at a certain moment (event point) as an image is called an event stream image.

[0050] An event stream image is an integral image of streaming event data along the event dimension, represented by a three-dimensional array of events, denoted as E. i (x, y, V), where x and y are the two-dimensional coordinates of a pixel in the event image, i is the sequence number of the event stream image, and V is the cumulative polarity value of the corresponding point in the image. That is...

[0051]

[0052] Where, ΔT=t i -t i-1 The cumulative interval for the i-th event is generally set to less than 10ms for high-speed human movements, but can be flexibly set according to the specific movement. i and t i-1 These represent the end and start times of the interval, respectively. ΔT is a fixed length, and its value is determined as follows: Figure 2 The sliding window pattern shown has a sliding window length of w = ΔT / 2.

[0053] S2: Event Tracking Acquisition

[0054] For each event stream image, it is necessary to extract the event trajectory. The human body's motion in a very short time can be regarded as rigid body motion, so its edges can be regarded as the contour of the body parts. The motion trajectory can be obtained through the contour edges. The main steps are as follows:

[0055] A1, in event stream image E i In the middle, for each event point P i (x,y,v i ),v i <ε, i.e., the number of times the event occurs v iIf it is less than the threshold ε, then v i =0, eliminating insignificant points in the motion and reducing subsequent calculations. Where ε = α·average(v i ),v i >0, 1≥α≥0, where α is the filtering coefficient.

[0056] A2. After eliminating insignificant motion points, the filtered event stream image E... i Perform morphological dilation followed by erosion and closing operations.

[0057] A3, regarding E i For each event point, outlier noise points are eliminated, which specifically includes the following steps:

[0058] A31. Search its neighborhood step by step according to the neighborhood expansion method. For the first point found, that is, the closest event point, calculate its Euclidean distance.

[0059] A32. Traverse the nearest Euclidean distance table for the entire event point and calculate the mean and standard deviation of the Euclidean distance distribution;

[0060] A33. Data points whose Euclidean distance to the nearest point is greater than the sum of the mean and standard deviation are removed as outliers; the remaining event image ET is then selected. i Enter the next calculation step.

[0061] A34. Using the Canny edge detection method (proposed in 1987 by John Canny et al. in the paper "A computational approach to edge detection" published in Readings in Computer Vision, pp. 184-203), from ET... i Extract its edge information and save it as edge graph E i Each event flow graph corresponds to an edge graph.

[0062] S3: Action Event Space Representation

[0063] Action event space: Represents the spatial information changes corresponding to an action from start to finish. For example, the set of all actions in a table tennis smash from start to finish. The action space is used to extract the trajectory characteristics of the action set, thereby judging the accuracy and quality of the action.

[0064] An action has a certain duration, such as 1 second or 500 milliseconds; a micro-stage refers to a short period of time in the action process, such as 8 ms. That is to say, an action process can be regarded as composed of multiple micro-actions. This invention reconstructs the trajectory of the action based on the position of each micro-stage and obtains trajectory features through deep learning methods, thereby realizing action recognition and quality assessment.

[0065] Spatial description of action events, such as Figure 3 As shown, action events are composed of time-series event side graphs. To effectively represent the action trajectories within the events, a 3D Convolutional Neural Network (3DCNN) deep learning model is used to extract features from the action event trajectories. The model extracts features from every N adjacent side graphs E. i As a set of input cubes i Two adjacent groups of N / 4 images are slid together, and feature representations (x1, x2, ..., x) are extracted using a 3D CNN. M );

[0066] S4: Action Recognition

[0067] like Figure 3 As shown, the extracted event sequence action features (x1, x2, ..., x...) M Action recognition is performed using a Long Short-Term Memory Network (LSTM), and the model outputs action categories and similarity scores. This enables the recognition and similarity measurement of highly dynamic actions.

[0068] S5: Visual Interaction Assessment

[0069] Visual interactive evaluation overlays the training movement trajectory, the standard movement trajectory, and the video stream from the visual camera, providing trainees with a visual display of movement errors. This facilitates movement improvement, reduces errors, and enhances training efficiency. The main steps are as follows:

[0070] Based on the transformation matrix parameters obtained during the registration of the event camera and the visual camera in the interactive training scene, the original RGB image is transformed into a registered image that is adapted to the imaging size of the event camera.

[0071] (1) Obtain the standard action trajectory stored in the standard library according to the recognized action type, and determine the action matching start time point by matching the scale-invariant feature transform (SIFT) feature of the trajectory image;

[0072] (2) Starting from the starting point, the similarity between the standard motion trajectory and the training motion trajectory is calculated using a shape-based method to achieve motion quality assessment;

[0073] (3) Based on the time synchronization characteristics of event stream and video stream, determine the registered image in the visual camera corresponding to the time information;

[0074] (4) Overlay the training motion trajectory and the standard motion trajectory onto the registration image to obtain the visual interactive evaluation results;

[0075] (5) Trainees can review the similarity of movements and visual errors through videos to improve their movements and enhance their quality.

[0076] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. An event camera and vision camera collaborative based motion training evaluation system, characterized in that, The application relates to a motion training evaluation system and a method thereof. The system comprises an event acquisition terminal, a host computer and an interactive terminal. The event acquisition terminal comprises an event camera and an event stream acquisition module, the event stream acquisition module being connected to the event camera through a USB. The host computer comprises an event trajectory extraction module, a motion recognition module and a motion quality evaluation module; the event trajectory extraction module extracts an event trajectory according to the output of the event acquisition terminal; the motion recognition module performs motion recognition according to the extracted event trajectory; and the motion quality evaluation module evaluates the recognized motion. The interactive terminal is used for displaying the motion recognition result. The event acquisition terminal further comprises a visual camera, which is used for transmitting the collected video stream to the interactive terminal for motion playback. The method comprises the following steps: S1, representing the motion event of a target region captured by the event camera in real time as et=(x, y, t, p), wherein x and y are pixel coordinates, t is a time stamp, and p is polarity, which is used for representing the increase or decrease of brightness; S2, converting the captured motion event into an event stream image; S3, extracting an event trajectory in each event stream image; S4, extracting a feature in the time trajectory; S5, inputting the extracted feature into a long short-term memory network to output a motion category and a motion similarity; The step S1 specifically comprises the following steps: In the step S1, the event representation and the conversion of the event captured by the camera into an event stream image are mainly included, and the event stream image is preprocessed. (1) event representation et=(x, y, t, p) An event is represented by a four-tuple (x, y, t, p), wherein x and y are two-dimensional coordinates of a pixel point where the event occurs, t is a time stamp of the event, and the unit is millisecond; p is the polarity of the event signal, the brightness decrease exceeding the threshold value is -1, the brightness increase exceeding the threshold value is +1, and the brightness change not exceeding the threshold value does not generate an event; the threshold value is set to 0.2, which represents that the brightness change does not exceed 20%; the threshold value is a settable system parameter, which needs to be adjusted under different light environments, and the parameter is set through calibration after the system starts; if the noise of the collected event stream image is relatively large, the threshold value needs to be increased, and vice versa. (2) event stream image Since the event camera is high dynamic, the output is an event stream, and the output of a certain moment, i.e. an event point, is saved as an image, which is an event stream image. The step S3 specifically comprises the following steps: The step A3 specifically comprises the following steps: The event stream image is an integral image of the streaming event data in the event dimension, represented by an event three-dimensional array, denoted as E i (x, y, V), wherein x and y are two-dimensional coordinates of a pixel point of the event image, i is a serial number of the event stream image, and V is a polarity accumulation value of a corresponding point of the image; that is: ； Wherein, ΔT=t i -t i-1 The accumulated interval time of the i-th event, for high-speed human motion, less than 10 ms, flexible setting according to specific motion, t i And t i-1 Respectively represent the interval end and start time; ΔT fixed length, the value is adopted sliding window mode, the sliding window length is w=ΔT / 2; A31, the neighborhood of each event point in the current event stream image is searched step by step in the neighborhood expansion mode, and the Euclidean distance between the nearest point in the neighborhood of each event point and the event point is calculated; the Euclidean distance of the first found point is calculated; A1. In the event flow image E i , for each event point P i (x, y, v i ), if the event occurrence number v i is less than a threshold value ε, then update the time point v i = 0, eliminate the motion insignificant point; ε = a · average(v i ), v i > 0, 1 ≥ a ≥ 0, a is a filtering coefficient; A2, to the event stream image E processed in step A1 i A morphological closing by dilation followed by erosion is performed; A3, the event stream image E processed in step A2 i eliminate outlier noise points at each event point; A32, the nearest Euclidean distance table corresponding to the current event stream image is obtained according to the nearest Euclidean distance of all event points in the current event stream image; A33, the average value and the standard deviation of the Euclidean distance distribution are calculated according to the nearest Euclidean distance table corresponding to the current event stream image. The step S5 specifically comprises the following steps: ​ A34、If the nearest Euclidean distance of a certain event point is greater than the sum of the mean and standard deviation, it is removed as an outlier noise point, and the remaining event image is ET i ; A35. Extracting the edge information from ETi using Canny edge detection method and save it as edge map E i Each event stream graph corresponds to an edge map; ​ The visual interactive evaluation superimposes the training motion trajectory, the standard motion trajectory and the video stream of the visual camera to output a visual display of the motion error for the trainer to improve the motion, reduce the error and improve the training efficiency. The main steps are as follows: Based on the transformation matrix parameters obtained by registering the event camera and the visual camera in the interactive training scene, the original acquired RGB image is converted into a registered image with an imaging size suitable for the event camera; (1) According to the identified motion type, the standard motion trajectory stored in the standard library is acquired, and the motion matching starting time point is determined through the scale-invariant feature transformation feature matching of the trajectory image; (2) Starting from the starting point, the shape-based method is used to calculate the similarity of the standard motion trajectory and the training motion trajectory to realize the motion quality evaluation; (3) According to the time synchronization characteristics of the event stream and the video stream, the registered image corresponding to the time information in the visual camera is determined; (4) The training motion trajectory and the standard motion trajectory are superimposed on the registered image to obtain the visual interactive evaluation result; (5) The trainer improves the motion and improves the quality by viewing the motion similarity and the visual error through the video.

Images