Motion capture method and system based on spatial differencing technique

Abstract

The application provides a motion capture method and system based on a space difference technique, comprising: acquiring visual information to be processed of a scene and a target object, the visual information to be processed comprising space difference visual information; performing motion capture key point extraction according to the space difference visual information to obtain a key point extraction result; and performing modeling of the target object according to the key point extraction result to obtain a motion capture result. The application uses the visual information to be processed collected by the space difference technique to realize sparse, efficient and complete visual perception and representation of a motion capture scene. Since the space difference visual information retains complete edge information of the target object, it is not necessary to perform time accumulation like a dynamic visual sensor, and key points in visual information at the same time can be directly detected, so that the application can greatly improve an actual sampling frame rate, thereby realizing motion capture with super-high positioning accuracy and sampling speed, and minimum resources and cost.

Description

Technical Field

[0001] This invention relates to the field of motion capture technology, and in particular to a motion capture method and system based on spatial difference technology. Background Technology

[0002] Motion capture (MoCap or mocap) is the process of measuring the motion trajectory or posture of a target object in real 3D space and then remodeling it in virtual space.

[0003] Currently, commonly used motion capture solutions include optically labeled motion capture and optically labeled motion capture. Optically labeled motion capture uses real-world markers on the target object and extracts key information from the optical features of these markers. This achieves the highest achievable positioning accuracy (sub-millimeter level) and sampling rate (up to 300-400 fps, which can be increased to over 1000 fps by narrowing the field of view and reducing resolution). Optically labeled motion capture, on the other hand, uses computer vision technology to calculate and model the target object's movements based on the inherent visual features of the target and scene. It offers greater flexibility and lower system complexity, but its accuracy and speed are inferior to optically labeled motion capture.

[0004] However, both optical marker-based motion capture and markerless motion capture technologies face the bandwidth wall problem caused by their frame-based sampling principle. Optical marker-based motion capture technology needs to transmit image frames to subsequent processing modules such as FPGAs to locate marker positions, while also retaining the original image frames (which may be at a lower frame rate) for marker calibration in post-processing. Markerless motion capture technology, on the other hand, requires high-fidelity original frame-based video data as data for key point recognition and tracking algorithms. Motion capture scenes themselves are sparse, and frame-based sampling introduces data redundancy, creating a contradiction that exacerbates the redundancy problem. This leads to bandwidth and power consumption wall issues, limiting further increases in the frame rate of motion capture technology.

[0005] Inspired by the temporal difference (TD) concept, event cameras such as Dynamic Vision Sensors (DVS) offer high-speed and sparse data representation. Recent work has successfully applied DVS cameras to label-free motion capture techniques. While TVS significantly reduces data bandwidth, it suffers from the limitation that sensors can only detect the moving parts of an object, failing to respond to stationary or slowly moving portions. This means the acquired TVS visual information cannot contain complete information about the object at any given moment, impacting recognition accuracy. Current solutions involve superimposing and accumulating TVS visual information over time intervals. However, this trade-off of temporal resolution for accuracy does not fundamentally address the inherent problems of TVS.

[0006] In summary, existing technologies cannot simultaneously achieve both recognition accuracy and performance. Summary of the Invention

[0007] This invention provides a motion capture method and system based on spatial difference technology to solve the shortcomings of existing technologies that cannot simultaneously achieve both recognition accuracy and performance, and to achieve motion capture with high performance and high recognition accuracy.

[0008] This invention provides a motion capture method based on spatial difference technology, comprising: Acquire visual information to be processed about the scene and the target object, wherein the visual information to be processed includes spatial difference visual information; Based on the spatial difference visual information, motion capture key points are extracted to obtain key point extraction results; wherein, the motion capture key point extraction includes key point recognition and localization processing. The target object is modeled based on the key point extraction results to obtain motion capture results.

[0009] According to the present invention, a motion capture method based on spatial difference technology is provided, wherein the spatial difference visual information is the result of weighted difference between the intensity signal of the current pixel position at the current time and the intensity signal of at least one arbitrary pixel position at the current time.

[0010] According to a motion capture method based on spatial difference technology provided by the present invention, the extraction of motion capture key points further includes preprocessing. Accordingly, the preprocessing includes: To reduce or eliminate noise and / or other interfering information in the spatial difference visual information, thereby improving the prominence of key point information in the spatial difference visual information; and / or, The data type of the spatial difference visual information is converted.

[0011] According to the motion capture method based on spatial difference technology provided by the present invention, when the motion capture is optical marker motion capture, the key point recognition and localization processing includes optical marker recognition and localization processing, wherein the optical markers are active luminous points or passive reflective points pre-set on the scene and / or the target object; the key point recognition and localization processing is performed based on the spatial difference visual information to obtain key point extraction results, specifically including: Based on the optical markers in the spatial difference visual information, a pre-selected recognition and positioning algorithm is used to identify and locate them, and the spatial coordinates of the center of the optical markers are obtained as the key point extraction result.

[0012] According to a motion capture method based on spatial difference technology provided by the present invention, when the motion capture is label-free optical motion capture, the key point identification and localization processing includes the extraction of feature nodes or surface modeling meshes; the key point identification and localization processing is performed according to the spatial difference visual information to obtain key point extraction results, specifically including: inputting the spatial difference visual information into the target label-free motion capture algorithm to extract feature nodes or surface modeling meshes to obtain key point extraction results; The target unlabeled motion capture algorithm is a pre-constructed unlabeled motion capture algorithm adapted to the data type of the input spatial difference visual information.

[0013] According to the motion capture method based on spatial difference technology provided by the present invention, the extraction of motion capture key points further includes post-processing; Accordingly, the post-processing includes: The results of the key point identification and positioning process are adjusted to improve the accuracy and correctness of the results.

[0014] According to the motion capture method based on spatial difference technology provided by the present invention, the step of acquiring the visual information to be processed of the scene and the target object specifically includes: Visual information of the scene and the target object is acquired using a visual sensor with a spatial difference path to obtain visual information to be processed, including spatial difference visual information; or visual information of the scene and the target object is acquired using a frame-based image sensor to obtain intensity visual information, and spatial difference is calculated on the intensity visual information to obtain visual information to be processed, including spatial difference visual information; or Visual information of the scene and the target object is generated based on a preset simulation algorithm to obtain visual information to be processed, including spatial difference visual information.

[0015] The present invention also provides a motion capture system based on spatial difference technology, comprising: A visual information acquisition unit is used to acquire visual information to be processed about a scene and a target object, wherein the visual information to be processed includes spatial difference visual information. The first processing unit is used to extract motion capture key points based on the spatial difference visual information to obtain key point extraction results; wherein, the motion capture key point extraction includes key point recognition and localization processing. The second processing unit is used to model the target object based on the key point extraction results to obtain motion capture results.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the motion capture method based on spatial difference technology as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the motion capture method based on spatial difference technology as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the motion capture method based on spatial difference technology as described above.

[0019] The motion capture method and system based on spatial difference technology provided by this invention acquires visual information to be processed from a scene and a target object, including spatial difference visual information; extracts key points for motion capture based on the spatial difference visual information to obtain key point extraction results; wherein, the key point extraction for motion capture includes key point recognition and localization processing; and models the target object based on the key point extraction results to obtain motion capture results. This invention uses the visual information acquired through spatial difference technology to achieve sparse, efficient, and complete visual perception and representation of the motion capture scene. Because spatial difference visual information preserves the complete edge information of the target object, it can directly detect key points in the visual information at the same moment without the need for time accumulation like dynamic visual sensors. This invention can significantly improve the actual sampling frame rate, thereby achieving ultra-high positioning accuracy and sampling rate with minimal resources and costs for motion capture. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts of the motion capture method based on spatial difference technology provided by the present invention.

[0022] Figure 2 This is the second flowchart of the motion capture method based on spatial difference technology provided by the present invention.

[0023] Figure 3 This is a schematic diagram of the motion capture system based on spatial difference technology provided by the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] Spatial difference and temporal difference are both special data types distinct from traditional frame-based sampling. The idea behind spatial difference comes from the receptive field of the retina in the human visual system, or the central enhancement and peripheral antagonism model. The retina consists of central excitatory cells and peripheral inhibitory cells. These cells respond to specific properties of light stimuli through light signals within their visual receptive field. In this model, central excitatory cells respond with enhanced light signals incident on the center of the receptive field, while peripheral inhibitory cells respond with suppressed light signals incident on the periphery of the receptive field. This central enhancement and peripheral inhibition relationship allows us to perceive local contrast. Simply put, when the central excitatory cells of the retina are stimulated by light, they generate enhanced electrical signals. Simultaneously, peripheral inhibitory cells, stimulated by weak light around the central excitatory cells, generate suppressed electrical signals. This central enhancement and peripheral inhibition pattern plays a crucial role in our visual system. By enhancing the central light signal and suppressing the peripheral light signal, visual features such as edges, contrast, and spatial location can be perceived. This model is significant for explaining phenomena such as edge detection, spatial filtering, and visual contrast modulation in visual perception.

[0027] The most common application of spatial difference in computer vision is edge detection algorithms. Its goal is to identify and extract edge regions from the visual information of a scene and a target object. An edge is the boundary between different regions in a scene and a target object, typically represented by a sudden change in pixel value intensity or a large spatial gradient. Edge detection algorithms locate these edges by analyzing changes in pixel values ​​within the visual information of the scene and the target object. Edge detection algorithms use operations such as differencing, convolution, or gradient gradation to find regions with significant changes in pixel values, thereby extracting edge information. Commonly used edge detection algorithms include Sobel, Prewitt, Canny, and Laplace. The choice of a suitable algorithm depends on the application requirements and the characteristics of the visual information of the scene and the target object.

[0028] Similar to temporal difference, spatial difference or edge detection processing, compared to the original data, extracts features, making the data more sparse, which reduces the pressure on transmission bandwidth. Compared to temporal difference sensors, spatial difference is not affected by static or dynamic objects in the scene, but records all edges in the scene completely, avoiding the problem of incomplete single-frame data in temporal difference sensors.

[0029] Based on this, the present invention applies spatial difference technology to the field of motion capture, and proposes a motion capture method and system based on spatial difference technology.

[0030] The following is combined Figures 1-2 The present invention describes a motion capture method based on spatial difference technology. Figure 1 This is one of the flowcharts illustrating the motion capture method based on spatial difference technology provided by the present invention, such as... Figure 1 As shown, the method includes steps 110, 120 and 130.

[0031] Step 110: Obtain visual information to be processed for the scene and the target object, wherein the visual information to be processed includes spatial difference visual information; The acquired visual information to be processed mainly includes spatial difference visual information. The visual information acquired by spatial difference technology retains the complete edge information of the target object. While achieving high-speed and sparse scene representation and data transmission, it can directly detect marker points in the visual information at the same moment without the need for time accumulation like dynamic visual sensors. This significantly improves the actual sampling frame rate of the motion capture system, enabling ultra-high positioning accuracy and sampling rate with minimal resources and costs. In some embodiments, the spatial difference visual information is the result of weighted difference between the intensity signal of the current pixel position at the current moment and the intensity signal of at least one arbitrary pixel position at the current moment.

[0032] It is understood that motion capture technology can be mainly divided into two categories based on its principles: optical motion capture and inertial motion capture. This invention primarily focuses on optical motion capture. Optical motion capture uses one or more image sensors to capture images of a target object and extracts key points based on visual features. Visual feature extraction can be achieved by setting actively emitting or passively reflecting markers on the target object to introduce additional visual features, or by using markerless motion capture algorithms such as neural networks to obtain motion capture results from inherent visual information within the scene.

[0033] In the specific implementation process, such as Figure 2 As shown, visual information about the target object and the scene is sent to the image sensor via the optical lens in the optical module. After signal processing, the final result can be output.

[0034] Figure 2 The optical module mainly refers to the optical lens, used to converge light and image it onto the pixel array surface of the image sensor. The incident light includes light of all wavelengths, and filters are generally introduced to retain only the light of the working wavelength to avoid unnecessary interference. Regarding the working wavelength, 850nm is preferred as the peak wavelength of the supplementary light source, but the protection range should be further expanded to all wavelengths of light. The next best options are 940nm near-infrared light, 460nm, and 480nm / 550nm mixed blue-green light. The next best options are near-infrared light and visible light of various other wavelengths, and supplementary lighting is also optional. It is understood that in some embodiments, passive tagging and motion capture based on the near-infrared band is used. For sensors with temporal or spatial differential functions, an infrared filter module is not necessary because as long as the infrared light source is strong enough, the spatiotemporal differential path can eliminate interference from other wavelengths of light through its own threshold. If it is necessary to add one, it is to cover the surface with an infrared bandpass filter.

[0035] Furthermore, after selecting the appropriate light source, the filtering band of the filter must also be adjusted accordingly.

[0036] When choosing an image sensor, a vision sensor with spatial difference can be used, or spatial difference can be implemented in subsequent data.

[0037] It should be noted that an image sensor is a sensor used to acquire visual information in a scene, and is also called a visual sensor. This is because a series of image frames are output by frame-based sampling technology. For image sensors with temporal difference paths or spatial difference paths, the visual information returned may not necessarily be represented as image frames. To avoid ambiguity, the embodiments of the present invention uniformly use the term "visual sensor" to describe it.

[0038] In some embodiments, acquiring the visual information to be processed regarding the scene and the target object specifically includes: Visual information of the scene and the target object is acquired using a visual sensor with a spatial difference path to obtain visual information to be processed, including spatial difference visual information; or visual information of the scene and the target object is acquired using a frame-based image sensor to obtain intensity visual information, and spatial difference is calculated on the intensity visual information to obtain visual information to be processed, including spatial difference visual information; or Visual information of the scene and the target object is generated based on a preset simulation algorithm to obtain visual information to be processed, including spatial difference visual information.

[0039] Specifically, in some embodiments, a spatial differential visual information sensor can be directly used to acquire visual information of the scene and the target object, or a three-channel visual sensor (i.e., intensity + TD + SD three-channel visual sensor) can be used but only its spatial differential channel is used to acquire visual information of the scene and the target object. At the same time, a simulation algorithm can be used to generate visual information to be processed, including spatial differential visual information.

[0040] Furthermore, this invention does not limit the specific simulation algorithm; the algorithm can be selected according to the actual situation.

[0041] In other embodiments, where spatial difference paths cannot be directly obtained due to hardware limitations, the spatial difference processing can be deferred to the FPGA or host computer, i.e., only the high-speed intensity path is used to simulate spatial difference visual information. In specific implementations, a high-speed camera (frame-based) acquires high-speed intensity path information, resulting in a series of intensity images. Spatial difference calculations are performed on these intensity images at the beginning of the signal processing module within the camera or the main control computer's processing program to obtain the visual information to be processed. This processing method reduces system performance to some extent, but compared to existing technologies, the data transmission and processing after spatial difference processing still exhibit sparse characteristics.

[0042] In addition, the camera's in-camera signal processing module integrates the image sensor's output and performs "necessary processing." This necessary processing refers to two aspects: firstly, encoding and outputting the information output by the pixel array according to the interface protocol; and secondly, performing processing and calculations on the signal as described in steps 120-130. These processing and calculation tasks include, but are not limited to, keypoint recognition and extraction of the two-dimensional coordinates of the marked center point.

[0043] In some embodiments, processing and computation tasks can be performed within the camera's in-camera signal processing module. This has the advantage of reducing the amount of data transmitted to the main control computer, thus lowering bandwidth requirements. However, the camera's internal computing resources are limited, leading to increased power consumption and data transmission latency. In other embodiments, processing and computation tasks can also be performed within the main control computer. This allows the main control computer to utilize high-performance computing chips, at the cost of increased data transmission. In still other embodiments, processing and computation tasks can be performed partly within the camera and partly within the main control computer. The hardware implementation of the in-camera signal processing module can be achieved through various methods, such as using FPGA (Field-Programmable Gate Array) chips, CPU (Central Processing Unit) chips, GPU (Graphics Processing Unit) chips, MCU (Microcontroller Unit) chips, ASIC (Application-Specific Integrated Circuit) chips, neuromorphic processors, neuromorphic chips, or directly embedded as a sub-module within the image sensor in the form of a SoC (System-on-a-Chip).

[0044] Furthermore, the output interface comes in various types, with the Ethernet interface being the preferred option. Its advantages include long cable transmission distances, convenient control of multiple devices via network protocols, and support for simultaneous power supply and data transmission over a single cable (PoE). Alternatively, the switch can also use USB interfaces or other interfaces (such as PCIe, FMC, etc.) to achieve similar functionality. The main control computer analyzes the camera information to ultimately obtain the three-dimensional spatial coordinates of the key points.

[0045] Step 120: Extract motion capture key points based on the spatial difference visual information to obtain key point extraction results; wherein, the motion capture key point extraction includes key point recognition and localization processing. It should be noted that optical motion capture includes marked motion capture and unmarked motion capture. The specific extraction method for key points depends on the type of optical motion capture.

[0046] When performing optical marker motion capture, the implementation scenarios of this invention are as follows: In one embodiment, passive infrared reflective spheres are placed on the target object as key points, and infrared light source illumination is added. These spheres reflect near-infrared light emitted by the camera's infrared light source, and the motion capture camera captures the infrared light reflected from the key points. A switch connects all motion capture cameras, used to control and synchronously trigger all cameras, and simultaneously aggregates the camera output information and uploads it to a computer.

[0047] Furthermore, to ensure that the wavelength of the near-infrared light reflected by the ball matches the operating wavelength of the camera, instead of using a separate infrared light source for illumination, a fill light plate is added. In one specific embodiment, the fill light plate is placed in front of the image sensor, emitting infrared light to illuminate the scene. The fill light plate typically needs to be parallel to the camera's focal plane and integrated into the camera module, rather than being a separate fill light, because this ensures that the light emitted by the fill light can be received by the camera after reflection. The wavelength of the fill light plate matches the operating wavelength of the camera. The fill light plate can be constantly lit, or it can receive signals emitted by the image sensor or the signal processing module within the camera, operating only during the exposure phase of the image sensor, thus reducing system power consumption; alternatively, it can receive signals and flash at a certain frequency, allowing the signal processing process to reduce interference from other light signals in the scene through time filtering.

[0048] In another embodiment, active infrared emitting spheres are placed on the target object as key points. It's important to note that these spheres actively emit infrared light and may carry optical codes that can be recognized by the camera. Motion capture cameras capture the infrared light emitted by the key points. A switch connects all motion capture cameras, controlling and synchronously triggering them, and simultaneously summarizing the camera output information and uploading it to a computer. It should be noted that in motion capture scenarios with active optical markers, not all key points may actually emit infrared light at any given moment; that is, there are indeed situations where a single frame image cannot detect all key points. In practical implementation, the infrared camera can be used for multiple exposures, with only one key point illuminating in each exposure. This avoids interference between markers, requiring n exposures for n key points. Alternatively, for greater efficiency, binary encoding can be used instead of a single-bit hot code, such as for 2... n Each key point requires n exposures; alternatively, all key points can flash continuously at a fixed frequency, and the image captured by the infrared camera can be filtered to obtain signals near this frequency band, which can effectively eliminate other interference signals. This invention does not limit its specific implementation.

[0049] Furthermore, the motion capture key point extraction includes one or more of preprocessing, key point extraction, and postprocessing.

[0050] Accordingly, to improve the efficiency and accuracy of key point extraction, spatial difference visual information is preprocessed before key point extraction. It should be emphasized that preprocessing is not a mandatory step; it can be performed or not. Correspondingly, the preprocessing task can be merged into the aforementioned processing and computation tasks. Naturally, the hardware setup and execution timing are the same as for the processing and computation tasks, and will not be elaborated upon in this embodiment.

[0051] In some embodiments, the motion capture key point extraction further includes preprocessing; Accordingly, the preprocessing includes: To reduce or eliminate noise and / or other interfering information in the spatial difference visual information, thereby improving the prominence of key point information in the spatial difference visual information; and / or, The data type of the spatial difference visual information is converted. Specifically, reducing or eliminating noise or other interference information in the spatial difference visual information includes: noise reduction, thresholding, erosion and dilation, masking, etc.

[0052] The main purpose of converting the data type of the spatial difference visual information is to adapt to the input requirements of other key point extraction algorithms in subsequent processing steps. The data types referred to include, but are not limited to, intensity information, temporal difference information, spatial difference information with varying difference directions, higher-order spatial difference information, and depth information.

[0053] In other words, preprocessing includes, but is not limited to, the following types: noise reduction, thresholding, quadratic difference processing, erosion and dilation, reconstruction, masking, and image magnification. It should be emphasized that in actual operation, one or more of these methods can be selected for execution, and the order of execution for multiple methods is not fixed and can be adjusted according to the actual situation. This invention does not impose any restrictions on this.

[0054] Furthermore, this embodiment will describe each preprocessing type in turn.

[0055] 1) Noise reduction processing: It is understandable that spatial difference visual information includes pixels with no output and pixels with output. Pixels with output can be further divided into pixels caused by changes in the spatial dimensions of actual objects in the scene and pixels caused by noise. Among these, pixels caused by noise are isolated because noise is independent in both time and space, and this type of preprocessing mainly targets these pixels.

[0056] In one specific embodiment, the noise reduction methods that can be considered include: averaging or accumulating several adjacent frames, and if the pixel is a noise point, the value will be significantly reduced; for each pixel with output, observe its neighborhood, and if the number of output pixels in its neighborhood within a certain range is less than a certain value, it is considered an isolated noise point; perform traditional noise reduction operations such as Gaussian filtering on the pixels; or use a pre-trained neural network model for noise reduction.

[0057] It should be noted that the present invention does not limit the specific method of noise reduction. The above examples are only optional embodiments. In actual operation, the specific method of noise reduction can be selected according to the actual situation.

[0058] 2) Threshold filtering process: It is understood that threshold filtering refers to setting the output of pixels with absolute values ​​below a threshold to 0. The threshold can be a fixed threshold, an adjustable threshold, or an adaptive threshold. The threshold can be set according to the characteristics of the optical markers, and this invention does not impose any limitations on this.

[0059] 3) Quadratic difference processing: It is understood that secondary difference processing includes further difference of spatial difference visual information, or modification of the difference angle based on single difference visual information. This invention does not impose specific limitations on the specific method of secondary difference; the specific method of secondary difference can be selected according to the actual situation during operation.

[0060] 4) Corrosion and expansion treatment: It is understood that erosion and dilation processing involves expanding or contracting the extracted information based on the existing output pixels. This invention does not impose specific limitations on the specific expansion or contraction methods; the choice of expansion or search method can be made according to the actual situation during operation.

[0061] 5) Reconstruction process: It is understood that the reconstruction process includes image reconstruction based on spatial difference visual information. This invention does not impose specific limitations on the particular image reconstruction method; the specific method of image reconstruction can be selected according to the actual situation during operation.

[0062] 6) Masking treatment: Understandably, masking involves forcibly setting all pixel signals within a preset range in the scene to 0. This is because, in addition to the bright optical marker ball, other interfering objects in the scene may reflect or emit infrared light. For these parts, pre-calibration is generally required, meaning the image is taken when there are no optical markers in the scene. For the detected pixels or bright spots that have output, a mask is generated, and then, throughout the entire process, signals within the mask are not processed.

[0063] 7) Image magnification: Understandably, image magnification includes using zero-order interpolation, linear interpolation, bilinear interpolation, and other interpolation methods to increase image resolution, making the visual features of key points more obvious and easier to detect.

[0064] Furthermore, preprocessing also includes other possible digital signal processing operations or preprocessing and representation reconstruction methods for event camera / brain-like vision sensor signals, which will not be elaborated upon in this invention.

[0065] In some embodiments, when the motion capture is optical marker motion capture, the key point recognition and localization process includes optical marker recognition and localization processing, wherein the optical markers are active luminous points or passive reflective points pre-set on the scene and / or the target object; key point recognition and localization processing is performed based on the spatial difference visual information to obtain key point extraction results, specifically including: Based on the optical markers in the spatial difference visual information, a pre-selected recognition and positioning algorithm is used to identify and locate them, and the spatial coordinates of the center of the optical markers are obtained as the key point extraction result.

[0066] Specifically, spatial difference visual information, including the spatial difference visual information of the bright spots of optical markers, is used to perform recognition and localization tasks, obtain the coordinates of the optical markers, and output them as key point extraction results.

[0067] The identification and localization task refers to locating optical markers in a scene and determining the coordinates of their centers using localization methods. The identification and localization algorithms employed include contour detection algorithms, clustering algorithms, Hough circle detection algorithms, fast detection algorithms, and object detection and recognition algorithms.

[0068] It should be noted that the contour detection algorithm includes: based on the thresholded image, selecting the binarized image, identifying all contours, and filtering the contours. The clustering algorithm includes: dividing pixels in the spatial difference visual information into several categories or clusters to distinguish optical marker points. For example, K-means clustering minimizes the number of pixels whose values ​​reach several centroids, while hierarchical clustering groups similar output pixels into larger clusters to form a cluster tree. The Hough circle detection algorithm includes: matching circles with different center coordinates and radii based on parameters to detect all circular patterns in the spatial difference visual information. The fast detection algorithm targets a certain range of output pixels, averaging all points to calculate the equivalent centroid; averaging the coordinates of the top, bottom, left, and rightmost points in the spatial difference visual information; calculating the diameter based on the longest vertical and horizontal position and using the midpoint of the diameter as the center point; and fitting ellipse parameters to all points to determine the center. Object detection and recognition algorithms include: training and detecting optical markers using object detection neural networks such as YOLO, and using the center of the bounding box as the center of the optical marker.

[0069] Furthermore, other high-speed circle center detection methods can be used for key point extraction. Specifically, after threshold filtering, the highest, lowest, leftmost, and rightmost points of the bright spots in the spatial difference visual information are directly extracted, and the average of these points is taken as the center point. The coordinates of this center point are then used as the key point coordinates.

[0070] Furthermore, other high-speed circle center detection methods include recording the number of luminous pixels row by row after threshold filtering, selecting the row with the most pixels as the diameter, and taking the intersection of the horizontal and vertical diameters as the center point. Alternatively, edge detection after threshold filtering can be used to calculate the centroid as the circle center point by weighted averaging of the coordinates of all points (using numerical values ​​as precision).

[0071] Furthermore, in some embodiments, the key point coordinates and spatial difference visual information of the bright spot including optical markers can be output simultaneously as the key point extraction result.

[0072] It is understood that the above steps describe the steps for extracting optical markers from single-frame two-dimensional spatial difference visual information. If there is multi-frame spatial difference visual information, the above steps can be used to process it sequentially.

[0073] When performing markerless motion capture, the implementation scenarios of this invention are as follows: Tagless motion capture does not introduce additional optical markers or devices into the scene. It uses only one or more cameras to record the scene as in normal shooting. Then, computer vision algorithms are used to process the acquired visual information data to identify the key points in the target object for tracking.

[0074] When spatial difference technology is applied to label-free motion capture scenarios, it extracts not the contours of optical markers, but rather the edge information of the entire scene. In this case, keypoint recognition and tracking algorithms are still required to achieve label-free motion capture. This includes two methods: edge detection and image reconstruction.

[0075] In some embodiments, when the motion capture is markerless optical motion capture, the key point recognition and localization process includes the extraction of feature nodes or surface modeling meshes; and the key point recognition and localization process is performed based on the spatial difference visual information to obtain key point extraction results, specifically including: The spatial difference visual information is input into the target unmarked motion capture algorithm to extract feature nodes or surface modeling meshes, and the key point extraction results are obtained. The target unlabeled motion capture algorithm is a pre-constructed unlabeled motion capture algorithm adapted to the data type of the input spatial difference visual information.

[0076] Specifically, for label-free motion capture, the extracted key points include feature points or human body modeling meshes. Feature points refer to skeleton models, while mesh modeling refers to dense models such as SMPL, representing two different data types for label-free motion capture.

[0077] The label-free motion capture model based on visual neural networks uses a pre-built edge detection model to extract feature points or human body modeling meshes. The edge detection model can be obtained by fine-tuning an existing edge detection-based CNN algorithm. In this case, since the first few layers of existing image neural networks with edge detection layers often use edge detection operators, and the spatial difference path of this invention has already implemented the relevant functions, while offering better data quality and robustness than edge detection achieved through CNNs, the first few layers of the network containing edge detection operators are removed during the edge detection model construction process. Then, spatial difference data is input for training, and the network structure is fine-tuned.

[0078] Edge detection models can also be obtained by training spatial difference data using label-free motion capture algorithms for intensity paths, followed by distillation or pruning of the trained model. The specific training steps are as follows: Step 210: Obtain massive spatial difference data samples; Step 220: Input the spatial difference data samples into the pre-selected intensity path label-free motion capture model for training until the preset termination condition is met, and output the first training result; Step 230: Distill or prune the first training result according to preset conditions to obtain the second training result; Step 240: Fine-tune the second training result, delete modules that are more sensitive to color or fine texture and strengthen modules that are sensitive to edge information to obtain the edge detection model.

[0079] Specifically, after acquiring massive spatial difference data samples, the label-free motion capture algorithm for intensity pathway images is trained using the spatial difference data. The trained model is then distilled or pruned to obtain a label-free motion capture algorithm suitable for spatial difference. Furthermore, the network structure needs fine-tuning; modules sensitive to color or fine texture can be pre-deleted while modules sensitive to edge information are strengthened.

[0080] Furthermore, the label-free motion capture model based on graphics or model segmentation performs label-free motion capture on visual information based solely on graphics or model segmentation, which will not be elaborated upon in this invention.

[0081] It is understood that the above steps describe the steps for extracting feature points or human body modeling meshes from single-frame two-dimensional spatial difference visual information. If there is multi-frame spatial difference visual information, the above steps can be used to process it sequentially.

[0082] It should be noted that both labeled and unlabeled motion capture may involve multiple spatial difference paths. Since spatial difference processes two-dimensional visual information, it can exist in multiple dimensions. When multiple spatial difference paths exist, the information from these paths needs to be integrated to obtain the final result. Typical implementation schemes include pre-fusion, that is, fusing visual information before keypoint extraction. For example, SD1 and SD2 are spatial difference visual information results from two orthogonal directions; adding their squares and taking the square root yields a unique visual information. Many similar fusion methods exist, such as addition and division by two, all aiming to integrate visual information. This invention does not limit the specific pre-fusion method of visual information.

[0083] Alternatively, post-fusion can be chosen, which means that each spatial difference visual information is processed separately, and the processed results are then integrated by averaging or weighted averaging, etc. This invention does not limit the specific post-fusion method of visual information.

[0084] Step 130: Model the target object based on the key point extraction results to obtain motion capture results.

[0085] It should be noted that step 120 yields the results of keypoint extraction, including the two-dimensional localization estimation results. A keypoint can be located in three dimensions using triangulation if it is captured simultaneously by at least two cameras. Naturally, in real-world scenarios, to improve accuracy, simultaneous capture by three or more cameras may be required for localization. Information from all infrared cameras is aggregated to the main control computer via a switch, and post-processing algorithms determine the three-dimensional spatial coordinates of the keypoints at each moment.

[0086] After obtaining the coordinates of the key points, the target object can be modeled using the key points. The modeling results can effectively represent the motion process and posture of the target object, thus obtaining the motion capture results.

[0087] Furthermore, to improve the accuracy of motion capture results or eliminate incorrectly identified key point data, post-processing can be performed on the extracted key points. It is important to emphasize that post-processing is not a mandatory step; it can be performed or not. Correspondingly, the post-processing task can be merged into the aforementioned processing and calculation tasks. Naturally, the hardware setup and execution timing are the same as for the processing and calculation tasks, and will not be elaborated upon in this embodiment.

[0088] In some embodiments, the motion capture key point extraction further includes post-processing; Accordingly, the post-processing includes: The results of the key point identification and localization processing are adjusted to improve their accuracy and correctness. Specifically, post-processing includes, but is not limited to, the following types: key point tracking processing, trajectory smoothing processing, spatial matching processing, spatial calibration processing, sub-pixel resolution processing, inspection calibration processing, and roundness detection processing. It should be emphasized that in actual operation, one or more of these processes can be selected for execution, and the order of execution is not fixed and can be adjusted according to the actual situation; this invention does not impose any limitations on this.

[0089] 1) Key point tracking and processing: Understandably, keypoint tracking involves using a target tracking algorithm to trace the coordinate trajectories of keypoints across consecutive visual information. For example, if there are two keypoints 1 and 2 in the visual information, and the next visual moment also contains two keypoints 1' and 2', it's necessary to determine whether 1 and 1' are the same marker through tracking. In practice, one embodiment uses a nearest neighbor approach for identification and tracking, assuming that among all keypoints detected in the next moment, the keypoint closest to keypoint 1 from the previous moment is keypoint 1 in this moment.

[0090] 2) Trajectory smoothing: Understandably, trajectory smoothing involves smoothing the trajectory based on the tracked data to improve positioning accuracy.

[0091] 3) Spatial matching processing: It's understandable that tracking can be interpreted as matching visual information over time, while the matching here refers to spatial matching, that is, matching visual information provided by sensors at different locations. For example, visual information 1 detects keypoints 1 and 2, while visual information 2 from another camera at the same time detects keypoints 1' and 2'. Whether 1 and 1' are the same marker needs to be determined through matching.

[0092] 4) Spatial calibration processing: Understandably, spatial calibration refers to comparing the detection results of cameras at different spatial locations on the same marker at the same time based on the matching results, and further calibrating to improve positioning accuracy.

[0093] 5) Subpixel resolution processing: Understandably, subpixel resolution processing refers to selecting the neighborhood around the center of a key point after it has been identified and located, fitting a high-order surface using all pixels, and using the extreme point as the new key point center. This operation is equivalent to fine-tuning the aforementioned positioning results, thereby improving positioning accuracy.

[0094] 6) Check calibration process: It is understood that the calibration process refers to using other methods to identify the original visual information and comparing it with the aforementioned identification and localization results to determine feasibility. In one specific embodiment, manual identification can be used to compare with the aforementioned identification and localization results.

[0095] 7) Roundness inspection and processing: Understandably, roundness detection processing refers to fitting elliptical parameters to all pixel output points within a certain range. If the calculation result deviates too much from a perfect circle, this range is considered not to belong to a key point and needs to be removed.

[0096] This invention utilizes spatial difference as a pathway to achieve sparse, efficient, and complete visual perception and representation of motion capture scenes, thereby increasing the motion capture sampling frame rate, reducing the computational load of the motion capture post-processing algorithm, and improving computational efficiency. Overall, it achieves high-performance, low-cost motion capture.

[0097] Furthermore, in another embodiment, a more accurate Kalman filter algorithm, such as DeepSort or ByteTrack, is used for identification and tracking. Inputting the spatiotemporal differential path data into the tracking algorithm requires corresponding adjustments to the algorithm's input structure and parameters. This invention does not impose specific limitations on the specific identification and tracking methods.

[0098] Furthermore, multi-scale techniques can be used to expand the receptive field, thereby improving positioning accuracy. Specifically, by selecting appropriate receptive fields and spatial difference functions, the sensor becomes more sensitive to key points or key visual features in the scene, which helps improve the positioning accuracy of the motion capture system.

[0099] The motion capture method based on spatial difference technology provided by this invention acquires visual information to be processed from a scene and a target object, including spatial difference visual information; extracts key points for motion capture based on the spatial difference visual information to obtain key point extraction results; wherein, the key point extraction for motion capture includes key point recognition and localization processing; and models the target object based on the key point extraction results to obtain motion capture results. This invention uses the visual information acquired through spatial difference technology to achieve sparse, efficient, and complete visual perception and representation of the motion capture scene. Because spatial difference visual information preserves the complete edge information of the target object, it can directly detect key points in the visual information at the same moment without the need for time accumulation like dynamic visual sensors. This invention can significantly improve the actual sampling frame rate, thereby achieving ultra-high positioning accuracy and sampling rate with minimal resources and costs for motion capture.

[0100] The motion capture system based on spatial difference technology provided by the present invention will be described below. The motion capture system based on spatial difference technology described below can be referred to in correspondence with the motion capture method based on spatial difference technology described above. Figure 3 This is a schematic diagram of the motion capture system based on spatial difference technology provided by the present invention, as shown below. Figure 3 As shown, the system includes: The visual information acquisition unit 310 is used to acquire visual information to be processed about the scene and the target object, wherein the visual information to be processed includes spatial difference visual information. The first processing unit 320 is used to extract motion capture key points based on the spatial difference visual information to obtain key point extraction results; wherein, the motion capture key point extraction includes key point recognition and localization processing. The second processing unit 330 is used to model the target object based on the key point extraction results to obtain motion capture results.

[0101] According to the present invention, a motion capture system based on spatial difference technology is provided, wherein the spatial difference visual information is the result of weighted difference between the intensity signal of the current pixel position at the current time and the intensity signal of at least one arbitrary pixel position at the current time.

[0102] According to the motion capture system based on spatial difference technology provided by the present invention, the extraction of motion capture key points further includes preprocessing; Accordingly, the preprocessing includes: To reduce or eliminate noise and / or other interfering information in the spatial difference visual information, thereby improving the prominence of key point information in the spatial difference visual information; and / or, The data type of the spatial difference visual information is converted.

[0103] According to the present invention, a motion capture system based on spatial difference technology, in the case of optical marker motion capture, the key point recognition and localization processing includes optical marker recognition and localization processing, wherein the optical markers are active luminous points or passive reflective points pre-set on the scene and / or the target object; the key point recognition and localization processing is performed based on the spatial difference visual information to obtain key point extraction results, specifically including: Based on the optical markers in the spatial difference visual information, a pre-selected recognition and positioning algorithm is used to identify and locate them, and the spatial coordinates of the center of the optical markers are obtained as the key point extraction result.

[0104] According to the present invention, a motion capture system based on spatial difference technology is provided. In the case of markerless optical motion capture, the key point identification and localization process includes the extraction of feature nodes or surface modeling meshes. The key point identification and localization process is performed based on the spatial difference visual information to obtain the key point extraction result. Specifically, the process includes: inputting the spatial difference visual information into the target markerless motion capture algorithm to extract feature nodes or surface modeling meshes to obtain the key point extraction result. The target unlabeled motion capture algorithm is a pre-constructed unlabeled motion capture algorithm adapted to the data type of the input spatial difference visual information.

[0105] According to the motion capture system based on spatial difference technology provided by the present invention, the extraction of motion capture key points further includes post-processing; Accordingly, the post-processing includes: The results of the key point identification and positioning process are adjusted to improve the accuracy and correctness of the results.

[0106] According to the motion capture system based on spatial difference technology provided by the present invention, the acquisition of visual information to be processed of the scene and the target object specifically includes: Visual information of the scene and the target object is acquired using a visual sensor with a spatial difference path to obtain visual information to be processed, including spatial difference visual information; or visual information of the scene and the target object is acquired using a frame-based image sensor to obtain intensity visual information, and spatial difference is calculated on the intensity visual information to obtain visual information to be processed, including spatial difference visual information; or Visual information of the scene and the target object is generated based on a preset simulation algorithm to obtain visual information to be processed, including spatial difference visual information.

[0107] The motion capture system based on spatial difference technology provided by this invention acquires visual information of a scene and a target object, including spatial difference visual information; extracts key points for motion capture based on the spatial difference visual information to obtain key point extraction results; wherein, the key point extraction includes key point recognition and localization processing; and models the target object based on the key point extraction results to obtain motion capture results. This invention uses the visual information acquired through spatial difference technology to achieve sparse, efficient, and complete visual perception and representation of the motion capture scene. Because spatial difference visual information preserves the complete edge information of the target object, it can directly detect key points in the visual information at the same moment without the need for time accumulation like dynamic visual sensors. This invention can significantly improve the actual sampling frame rate, thereby achieving ultra-high positioning accuracy and sampling rate with minimal resources and costs for motion capture. Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a motion capture method based on spatial difference technology. This method includes: acquiring visual information to be processed about a scene and a target object, the visual information including spatial difference visual information; extracting motion capture key points based on the spatial difference visual information to obtain key point extraction results; wherein the motion capture key point extraction includes key point recognition and localization processing; and modeling the target object based on the key point extraction results to obtain motion capture results.

[0108] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the motion capture method based on spatial difference technology provided by the above methods. The method includes: acquiring visual information to be processed of a scene and a target object, the visual information to be processed including spatial difference visual information; extracting motion capture key points based on the spatial difference visual information to obtain key point extraction results; wherein, the motion capture key point extraction includes key point recognition and localization processing; and modeling the target object based on the key point extraction results to obtain motion capture results.

[0110] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the motion capture method based on spatial difference technology provided by the above methods. The method includes: acquiring visual information to be processed of a scene and a target object, the visual information to be processed including spatial difference visual information; extracting motion capture key points based on the spatial difference visual information to obtain key point extraction results; wherein the motion capture key point extraction includes key point recognition and localization processing; and modeling the target object based on the key point extraction results to obtain motion capture results.

[0111] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A motion capture method based on spatial difference technology, characterized in that, include: Acquire visual information to be processed about the scene and the target object, wherein the visual information to be processed includes spatial difference visual information; The spatial difference visual information is the result of weighted difference between the intensity signal of the current pixel position at the current time and the intensity signal of at least one arbitrary pixel position at the current time, and the spatial difference visual information contains complete edge information of the target object in both static and dynamic states. Motion capture key point extraction is performed based on the spatial difference visual information to obtain key point extraction results; wherein, the motion capture key point extraction includes key point recognition and localization processing on the spatial difference visual information; The target object is modeled based on the key point extraction results to obtain motion capture results.

2. The motion capture method based on spatial difference technology according to claim 1, characterized in that, The extraction of motion capture key points also includes preprocessing; Accordingly, the preprocessing includes: To reduce or eliminate noise and / or other interfering information in the spatial difference visual information, thereby improving the prominence of key point information in the spatial difference visual information; and / or, The data type of the spatial difference visual information is converted.

3. The motion capture method based on spatial difference technology according to claim 1, characterized in that, In the case where the motion capture is optical marker motion capture, the key point recognition and localization process includes optical marker recognition and localization process, wherein the optical marker is an active light-emitting point or a passive reflective point that is pre-set on the scene and / or the target object; Based on the spatial difference visual information, key point recognition and localization processing is performed to obtain key point extraction results, specifically including: Based on the optical markers in the spatial difference visual information, a pre-selected recognition and positioning algorithm is used to identify and locate them, and the spatial coordinates of the center of the optical markers are obtained as the key point extraction result.

4. The motion capture method based on spatial difference technology according to claim 1, characterized in that, In the case of markerless optical motion capture, the key point identification and localization process includes the extraction of feature nodes or surface modeling meshes; Based on the spatial difference visual information, key point recognition and localization processing is performed to obtain key point extraction results, specifically including: The spatial difference visual information is input into the target unmarked motion capture algorithm to extract feature nodes or surface modeling meshes, and the key point extraction results are obtained. The target unlabeled motion capture algorithm is a pre-constructed unlabeled motion capture algorithm adapted to the data type of the input spatial difference visual information.

5. The motion capture method based on spatial difference technology according to claim 1, characterized in that, The extraction of motion capture key points also includes post-processing; Accordingly, the post-processing includes: The results of the key point identification and positioning process are adjusted to improve the accuracy and correctness of the results.

6. The motion capture method based on spatial difference technology according to claim 1, characterized in that, The acquisition of the visual information to be processed regarding the scene and the target object specifically includes: Visual information of the scene and the target object is acquired using a visual sensor with a spatial difference path to obtain visual information to be processed, including spatial difference visual information; or Visual information of the scene and the target object is acquired using a frame-based image sensor to obtain an intensity image. Spatial difference is then calculated on the intensity image to obtain visual information to be processed, including spatial difference visual information; or Visual information of the scene and the target object is generated based on a preset simulation algorithm to obtain visual information to be processed, including spatial difference visual information.

7. A motion capture system based on spatial difference technology, characterized in that, include: A visual information acquisition unit is used to acquire visual information to be processed about a scene and a target object, wherein the visual information to be processed includes spatial difference visual information. The spatial difference visual information is the result of weighted difference between the intensity signal of the current pixel position at the current time and the intensity signal of at least one arbitrary pixel position at the current time, and the spatial difference visual information contains complete edge information of the target object in both static and dynamic states. The first processing unit is used to extract motion capture key points based on the spatial difference visual information to obtain key point extraction results; wherein, the motion capture key point extraction includes key point recognition and localization processing on the spatial difference visual information. The second processing unit is used to model the target object based on the key point extraction results to obtain motion capture results.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the motion capture method based on spatial difference technology as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the motion capture method based on spatial difference technology as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the motion capture method based on spatial difference technology as described in any one of claims 1 to 6.

Images