Stiffness-flexibility coupled force potential hybrid sensing method, device, apparatus, medium and product

Abstract

The application provides a rigid-flexible coupling force-position hybrid sensing method, device, equipment, medium and product. The method is applied to a force-position hybrid sensing device. The device comprises a force-receiving platform and a binocular camera. The binocular camera is used for collecting a structure image of a mark structure on the force-receiving platform. The method comprises the following steps: acquiring a structure image in real time. The structure image comprises images of the mark structure of the force-receiving platform in a force-receiving state and a non-force-receiving state. Coordinate extraction is performed on the structure image. Three-dimensional reconstruction is performed based on the extracted two-dimensional coordinates to obtain a three-dimensional point cloud of the mark structure. The end position is determined based on the three-dimensional point cloud in the force-receiving state and the non-force-receiving state. The six-dimensional force vector of the force-receiving platform is determined based on the real-time three-dimensional point cloud of the mark structure. The defects that the current method cannot simultaneously sense the six-dimensional position and the six-axis force / torque information are overcome. The end position and the six-axis force / torque information can be accurately obtained simultaneously through the force-position hybrid sensing device, and the demand of multi-dimensional interactive perception is met.

Description

Technical Field

[0001] This invention relates to the field of six-axis force / torque sensing technology, and more particularly to a rigid-flexible coupled force-position hybrid sensing method, device, equipment, medium, and product. Background Technology

[0002] In the field of robotics, accurately acquiring force / torque information and end-effector pose is crucial for achieving efficient, precise, and intelligent operational tasks. Six-axis force / torque and end-effector pose sensing provides robots with comprehensive environmental interaction and perception capabilities, enabling them to complete tasks such as assembly and grasping under complex and changing working conditions, making it highly valuable for applications.

[0003] Currently, traditional force / torque sensors are mainly based on resistance strain gauges or piezoelectric materials. Their working principle involves measuring the strain generated in a material under stress and converting it into a corresponding force / torque signal. While these sensors offer excellent accuracy and bandwidth, their design principles and structures limit their size and rigidity. With the trend towards lightweight and flexible robots, these characteristics make direct integration into lightweight end effectors difficult.

[0004] Furthermore, with the increasing demand for multi-dimensional interactive perception in robots, existing technologies face new challenges. Currently, most solutions can only achieve force / torque or pose measurement independently. This separate sensing makes it difficult for robots to fully perceive their interaction with the environment in real time during operation, limiting their adaptability to complex tasks and operational accuracy. Summary of the Invention

[0005] This invention provides a rigid-flexible coupled force-position hybrid sensing method, device, equipment, medium, and product to address the shortcomings of existing force / torque sensing and pose sensing technologies, which cannot simultaneously acquire six-dimensional pose and six-axis force / torque information in the same structure, making it difficult to meet the needs of multi-dimensional interactive perception.

[0006] This invention provides a rigid-flexible coupled force-potential hybrid sensing method, applied to a force-potential hybrid sensing device. The force-potential hybrid sensing device includes a force-receiving platform and a binocular camera. A marker structure is disposed on the force-receiving platform, and the binocular camera is used to acquire structural images of the marker structure on the force-receiving platform. The method includes:

[0007] The structural image is acquired in real time, and the structural image includes images of the marked structure of the force-bearing platform under both stressed and unstressed states.

[0008] Based on the structural image, coordinate extraction is performed to obtain the two-dimensional coordinates of the marked structure;

[0009] Based on the two-dimensional coordinates, a three-dimensional reconstruction is performed to obtain the three-dimensional point cloud of the marked structure;

[0010] Based on the three-dimensional point cloud under the stress state and the three-dimensional point cloud under the unstressed state, the end pose of the stress-bearing platform is determined.

[0011] Based on the real-time three-dimensional point cloud of the marked structure, the six-dimensional force vector of the force-bearing platform is determined.

[0012] According to a rigid-flexible coupled force-position hybrid sensing method provided by the present invention, determining the end pose of the force-bearing platform based on the three-dimensional point cloud under the force-bearing state and the three-dimensional point cloud under the unforced state includes:

[0013] Point cloud registration is performed between the three-dimensional point cloud under the stress state and the three-dimensional point cloud under the no-stress state to obtain the rotation matrix and translation vector between the three-dimensional point cloud under the stress state and the three-dimensional point cloud under the no-stress state.

[0014] The end pose of the force-bearing platform is determined based on the rotation matrix and the translation vector.

[0015] According to a rigid-flexible coupled force-position hybrid sensing method provided by the present invention, determining the end pose of the force-bearing platform based on the rotation matrix and the translation vector includes:

[0016] Based on the rotation matrix and the translation vector, construct the rigid body transformation matrix;

[0017] Based on the rigid body transformation matrix, the end pose of the force-bearing platform is determined.

[0018] According to the present invention, a rigid-flexible coupled force-potential hybrid sensing method is provided, wherein the step of extracting coordinates based on the structural image to obtain the two-dimensional coordinates of the marked structure includes:

[0019] The structure image is processed to obtain an enhanced structure image;

[0020] Centroid extraction is performed on the enhanced structure image to obtain the two-dimensional coordinates of the marked structure;

[0021] The data processing includes at least one of Gaussian filtering, image differencing, binarization, and morphological operations.

[0022] According to the present invention, a rigid-flexible coupled force-potential hybrid sensing method is provided, wherein the step of performing three-dimensional reconstruction based on the two-dimensional coordinates to obtain a three-dimensional point cloud of the marked structure includes:

[0023] By performing circular grouping, the two-dimensional coordinates of the marked structure are matched to obtain the matching point pairs in the marked structure;

[0024] By using ray tracing modeling, the matching point pairs in the marker structure are reconstructed in three dimensions to obtain the three-dimensional point cloud of the marker structure.

[0025] According to the rigid-flexible coupling force-potential hybrid sensing method provided by the present invention, the determination of the six-dimensional force vector of the force-bearing platform based on the real-time three-dimensional point cloud of the marked structure includes:

[0026] Based on the real-time 3D point cloud of the marked structure, the displacement sequence of the 3D point cloud of the marked structure is determined;

[0027] The three-dimensional point cloud displacement sequence is input into the six-axis force measurement model to obtain the six-dimensional force vector of the force-bearing platform output by the six-axis force measurement model;

[0028] The six-axis force measurement model is obtained by training the sample three-dimensional point cloud displacement sequence based on the sample marker structure and the sample six-dimensional force vector corresponding to the sample marker structure.

[0029] The present invention also provides a force-position hybrid sensing device, including a force-receiving platform, a binocular camera, an image acquisition unit, an image processing unit, a three-dimensional reconstruction unit, a pose estimation unit, and a force estimation unit; a marker structure is provided on the force-receiving platform, and the binocular camera is used to acquire structural images of the marker structure on the force-receiving platform;

[0030] The image acquisition unit is used to acquire the structure image in real time, and the structure image includes images of the marked structure of the force-bearing platform in both the force-bearing state and the unforced state.

[0031] The image processing unit is used to extract coordinates based on the structural image to obtain the two-dimensional coordinates of the marked structure.

[0032] The three-dimensional reconstruction unit is used to perform three-dimensional reconstruction based on the two-dimensional coordinates to obtain a three-dimensional point cloud of the marked structure.

[0033] The pose estimation unit is used to determine the end pose of the force-bearing platform based on the three-dimensional point cloud under the force-bearing state and the three-dimensional point cloud under the unforced state.

[0034] The force estimation unit is used to determine the six-dimensional force vector of the force-bearing platform based on the real-time three-dimensional point cloud of the marked structure.

[0035] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the rigid-flexible coupling force-position hybrid sensing method as described above.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the rigid-flexible coupling force-position hybrid sensing method as described above.

[0037] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the rigid-flexible coupling force-position hybrid sensing method as described above.

[0038] The present invention provides a rigid-flexible coupled force-position hybrid sensing method, device, equipment, medium, and product. It acquires structural images in real time, including images of the marked structure under both stressed and unstressed states of the force-bearing platform. These structural images are captured by a binocular camera in the force-position hybrid sensing device. Coordinates are extracted from the structural images to obtain the two-dimensional coordinates of the marked structure. Three-dimensional reconstruction is performed based on these coordinates to obtain a three-dimensional point cloud of the marked structure. The end-effector pose of the force-bearing platform is determined based on both the stressed and unstressed point clouds. Finally, the six-dimensional force vector of the force-bearing platform is determined based on the real-time three-dimensional point cloud of the marked structure. This overcomes the limitations of traditional separate sensing methods, which cannot simultaneously sense six-dimensional pose and six-axis force / torque information, thus failing to meet the demands for multi-dimensional interactive perception. The force-position hybrid sensing device can simultaneously and accurately acquire end-effector pose and six-axis force / torque information, meeting the requirements of lightweight, integrated, and multi-dimensional interactive perception in the field of robot operation. This enables robots to handle complex tasks and improves operational accuracy. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the rigid-flexible coupling force-position hybrid sensing method provided by the present invention.

[0041] Figure 2 This is a rendering of the force-bearing platform before and after force application provided by the present invention;

[0042] Figure 3 This is a schematic diagram of the ring matching process provided by the present invention;

[0043] Figure 4 This is a schematic diagram of the point cloud for three-dimensional reconstruction provided by the present invention;

[0044] Figure 5 This is a flowchart of the measurement process for the six-dimensional force vector provided by the present invention;

[0045] Figure 6 This is a test result diagram of the six-dimensional force vector provided by the present invention;

[0046] Figure 7 This is a schematic diagram of the force-position hybrid sensing device provided by the present invention;

[0047] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] In the field of robotics, accurately acquiring force / torque information and end-effector pose information is crucial for achieving efficient, precise, and intelligent operational tasks. Six-axis force / torque and end-effector pose sensing provides robots with comprehensive environmental interaction and perception capabilities, enabling them to complete tasks such as assembly and grasping under complex and changing working conditions, and has immeasurable application value.

[0050] Currently, traditional force / torque sensors are mainly based on resistance strain gauges or piezoelectric materials. Their working principle involves measuring the strain generated in a material under stress and converting it into a corresponding force / torque signal. While these sensors offer excellent accuracy and bandwidth, their design principles and structural characteristics often result in large size and high rigidity. With the increasing trend towards lightweight and flexible robots, these characteristics make direct integration into lightweight end effectors difficult. Excessive size and high rigidity not only increase the overall weight of the robot's end effector, affecting its flexibility and response speed, but may also cause damage or excessive impact on the manipulated object when interacting with complex environments due to a lack of flexibility.

[0051] Meanwhile, with the ever-increasing demand for multi-dimensional interactive perception in robots, existing technologies face another major challenge. That is, current sensing solutions often can only measure force / torque or pose individually, failing to simultaneously acquire six-dimensional pose and six-axis force / torque information within the same structure. This fragmented sensing approach makes it difficult for robots to perceive their interaction with the environment in real-time and comprehensively during operation, limiting their adaptability and operational accuracy for complex tasks, and failing to meet the urgent needs for multi-dimensional interactive perception in modern industrial production, medical surgery, and service robots.

[0052] To address this, the present invention provides a rigid-flexible coupled force-position hybrid sensing method, which aims to overcome the shortcomings of current separate sensing methods. It can simultaneously acquire six-dimensional pose and six-axis force / torque information through the same structure (force-position hybrid sensing device), thereby well adapting to the needs of the robotics field for lightweight, integrated, and multi-dimensional interactive perception. This enables robots to cope with complex tasks and improves operational accuracy.

[0053] Figure 1 This is a flowchart illustrating the rigid-flexible coupling force-position hybrid sensing method provided by the present invention, as shown below. Figure 1 As shown, this method is applied to a force-potential hybrid sensing device, which includes a force platform and a binocular camera. A marker structure is set on the force platform, and the binocular camera is used to acquire structural images of the marker structure on the force platform. The method includes:

[0054] Step 110: Acquire structural images in real time. The structural images include images of the structure marked by the force-bearing platform in both the stressed and unstressed states.

[0055] Step 120: Extract coordinates based on the structural image to obtain the two-dimensional coordinates of the marked structure;

[0056] Step 130: Perform three-dimensional reconstruction based on two-dimensional coordinates to obtain a three-dimensional point cloud of the marked structure;

[0057] Step 140: Determine the end pose of the stressed platform based on the three-dimensional point cloud under stress and the three-dimensional point cloud under no stress.

[0058] Step 150: Based on the real-time three-dimensional point cloud of the marked structure, determine the six-dimensional force vector of the force-bearing platform.

[0059] Specifically, considering that current sensing solutions can often only measure force / torque or pose independently, making it difficult for robots to fully perceive their interaction with the environment in real time during operation and failing to meet the needs of multi-dimensional interactive perception, this invention proposes a device that can simultaneously and accurately acquire end-effector pose and six-axis force / torque information, namely a force-position hybrid sensing device. This device can simultaneously sense end-effector pose and six-axis force / torque information.

[0060] The force-position hybrid sensing device includes a force-bearing platform and a binocular camera. The force-bearing platform is equipped with a marker structure, which is a point structure composed of multiple marker points. The purpose of the marker structure is to facilitate identification by the binocular camera, thereby determining the end effector pose and six-axis force / torque information. The binocular camera can be used to acquire images of the force-bearing platform before and after the force is applied, specifically acquiring structural images of the marker structure on the force-bearing platform.

[0061] In detail, in this embodiment of the invention, when performing force-position hybrid sensing, it is first necessary to acquire structural images of the marked structure captured by the binocular camera. That is, in real time, images of the marked structure captured by the left and right cameras of the binocular camera are acquired respectively, namely, the left-eye structural image and the right-eye structural image, which are collectively referred to as structural images. The structural images include the left-eye structural image captured by the left-eye camera and the right-eye structural image captured by the right-eye camera. It should be noted that, since the marked structure is designed for easy visual recognition, and the binocular camera mainly captures structural images before and after force application, the continuous multi-frame structural images acquired in real time include structural images before and after force application. That is, images of the marked structure under force and without force application on the force-bearing platform.

[0062] After obtaining multiple consecutive frames of structural images through real-time image acquisition, this embodiment of the invention can perform preliminary processing on these structural images to enhance the detectability and stability of the marked structures, thereby obtaining a processed structural image. The preprocessing operations here can include Gaussian filtering, image differencing, binarization, morphological operations, etc., or other operations that can enhance the marked structures in the structural image; this embodiment of the invention does not specifically limit these operations. Further, coordinate extraction can be performed on the processed structural image to extract the planar coordinates of the marked structures, thereby obtaining the two-dimensional coordinates of each marked point in the marked structure. This coordinate extraction process can be implemented using shape matching algorithms, corner detection algorithms, target detection algorithms, centroid extraction algorithms, etc.; the specific implementation method can be selected according to actual needs, and this embodiment of the invention does not specifically limit this implementation.

[0063] After this, the extracted two-dimensional coordinates can be further processed to convert the coordinates of the two-dimensional plane into coordinates in three-dimensional space, thereby obtaining a three-dimensional point cloud of the marked structure. Specifically, this can be achieved by performing three-dimensional reconstruction based on the two-dimensional coordinates of the marked structure, so as to transform the coordinates of each marked point in the two-dimensional plane into three-dimensional space to obtain three-dimensional spatial coordinates, i.e., a three-dimensional point cloud. Here, the three-dimensional reconstruction process can be implemented using triangulation algorithms, structured light related algorithms (such as coded structured light, depth cameras, etc.), deep learning algorithms (such as depth estimation, end-to-end 3D reconstruction, etc.), ray tracing algorithms, etc.; the specific implementation method can be selected according to actual needs, and the embodiments of this invention do not impose specific limitations.

[0064] After obtaining the three-dimensional point cloud of the marked structure, in this embodiment of the invention, the end pose and six-axis force / torque information of the force-bearing platform can be obtained based on this three-dimensional point cloud, thereby realizing the acquisition of pose information and six-axis force / torque information based on the same structure.

[0065] Specifically, pose estimation can be performed based on the three-dimensional point cloud before and after the force is applied to estimate the pose change of the force-applied platform and thus obtain the end-effector pose. That is, pose estimation can be performed based on the three-dimensional point cloud of the marked structure under the force and the three-dimensional point cloud of the marked structure under the unforced state. By analyzing the three-dimensional point cloud before and after the force is applied, the pose change of the force-applied platform before and after the force is applied can be determined, thus obtaining its end-effector pose.

[0066] Simultaneously, based on the three-dimensional point cloud of the marked structure in consecutive frames of structural images, six-axis force / moment measurements can be performed to obtain the six-axis force / moment information of the stressed platform, which can be represented by a six-dimensional force vector. This six-dimensional force vector can include the forces on the stressed platform in three directions, as well as the moments in those three directions.

[0067] The rigid-flexible coupling force-position hybrid sensing method provided by this invention is applied to a force-position hybrid sensing device. It acquires structural images in real time, including images of the marked structure under both stressed and unstressed states of the force-bearing platform. These structural images are captured by a binocular camera within the force-position hybrid sensing device. Coordinates are extracted from the structural images to obtain the two-dimensional coordinates of the marked structure. Three-dimensional reconstruction is performed based on these coordinates to obtain a three-dimensional point cloud of the marked structure. The end-effector pose of the force-bearing platform is determined based on both the stressed and unstressed point clouds. Finally, the six-dimensional force vector of the force-bearing platform is determined based on the real-time three-dimensional point cloud of the marked structure. This method overcomes the limitations of traditional separate sensing methods, which cannot simultaneously sense six-dimensional pose and six-axis force / torque information, thus failing to meet the demands for multi-dimensional interactive perception. The force-position hybrid sensing device can simultaneously and accurately acquire end-effector pose and six-axis force / torque information, meeting the requirements of lightweight, integrated, and multi-dimensional interactive perception in the robotics field. This enables robots to handle complex tasks and improves operational accuracy.

[0068] Based on the above embodiments, step 140 includes:

[0069] Point cloud registration is performed between the 3D point cloud under stress and the 3D point cloud under no stress to obtain the rotation matrix and translation vector between the 3D point cloud under stress and the 3D point cloud under no stress.

[0070] The end pose of the force-bearing platform is determined based on the rotation matrix and translation vector.

[0071] Specifically, the process of pose estimation based on the 3D point cloud before and after the force is applied can include:

[0072] First, calculations can be performed based on the 3D point cloud under stress and the 3D point cloud under no stress to obtain the pose change of the platform before and after the stress. Specifically, this can be done by registering the 3D point cloud under stress and the 3D point cloud under no stress, and obtaining the rotation matrix and translation vector between the two sets of 3D point clouds through Singular Value Decomposition (SVD). This rotation matrix and translation vector can characterize the change of the platform before and after the stress.

[0073] Figure 2 These are effect diagrams of the force-bearing platform before and after force application provided by this invention, such as... Figure 2 As shown, by comparing the force-bearing platform before and after the force is applied, it can be clearly seen that the force-bearing platform has undergone a change in pose after the force is applied, while the rotation matrix and translation vector can reflect the rotation and translation changes of the force-bearing platform in three-dimensional space before and after the force is applied.

[0074] Then, the pose change of the stressed platform can be estimated based on the rotation matrix and translation vector between the 3D point cloud under stress and the 3D point cloud under no stress obtained in the previous step, thus obtaining the end effector pose; that is, pose estimation can be performed based on the rotation matrix and translation vector to analyze the pose change of the stressed platform before and after stress, thereby obtaining the end effector pose of the stressed platform. The end effector pose here can be represented by a homogeneous transformation matrix.

[0075] Based on the above embodiments, the end pose of the force-bearing platform is determined based on the rotation matrix and translation vector, including:

[0076] Construct the rigid body transformation matrix based on the rotation matrix and translation vector;

[0077] The end pose of the force-bearing platform is determined based on the rigid body transformation matrix.

[0078] Specifically, the process of determining the end pose of the force-bearing platform based on the rotation matrix and translation vector actually involves first constructing a rigid body transformation matrix based on the obtained rotation matrix and translation vector. This rigid body transformation matrix can more accurately represent the pose change of the force-bearing platform, and then the end pose can be directly calculated based on this rigid body transformation matrix, thereby realizing the pose estimation of the force-bearing platform.

[0079] Based on the above embodiments, step 120 includes:

[0080] Data processing is performed on the structural image to obtain an enhanced structural image;

[0081] Centroid extraction is performed on the enhanced structure image to obtain the two-dimensional coordinates of the marked structure;

[0082] Data processing includes at least one of Gaussian filtering, image differencing, binarization, and morphological operations.

[0083] Specifically, the process of extracting coordinates from the structural image to obtain the two-dimensional coordinates of the marked structure can include:

[0084] First, the acquired structural images of multiple consecutive frames can be preliminarily processed, that is, Gaussian filtering, image differencing, binarization, morphological operations or any one or more of these data processing methods can be applied to the structural images to enhance the detectability and stability of the marked structures in the structural images, thereby obtaining the enhanced structural images, i.e., the enhanced structural images.

[0085] Next, coordinate extraction can be performed on this enhanced structure image to extract the planar coordinates of the marker structure, thereby obtaining the two-dimensional coordinates of each marker point in the marker structure. Specifically, this can be achieved by using centroid extraction algorithms (such as connected component analysis, contour detection, moment calculation, etc.) to extract coordinates from the enhanced structure image. Alternatively, it can be understood as performing centroid extraction on the enhanced structure image to extract the two-dimensional coordinates of each marker point, thereby obtaining the two-dimensional coordinates of the marker structure.

[0086] Based on the above embodiments, step 130 includes:

[0087] By performing circular grouping, the two-dimensional coordinates of the marked structure are matched to obtain the matching point pairs in the marked structure;

[0088] By using ray tracing modeling, the matching point pairs in the marker structure are reconstructed in 3D to obtain the 3D point cloud of the marker structure.

[0089] Specifically, the process of performing three-dimensional reconstruction based on two-dimensional coordinates to obtain a three-dimensional point cloud of the marked structure includes the following steps:

[0090] First, the two-dimensional coordinates of the marker structure can be grouped in a circular manner and matched in a three-dimensional manner to obtain the matching point pairs in the marker structure. Figure 3 This is a diagram illustrating the effect of the ring matching process provided by the present invention, such as... Figure 3 As shown, based on the two-dimensional coordinates of the extracted marker structure, a circular grouping method is used to achieve marker matching between the left and right structure images corresponding to the left and right cameras in the binocular camera, respectively, thereby obtaining the matching point pairs between the marker structures in the left structure image and the marker structures in the right structure image.

[0091] Subsequently, 3D reconstruction can be performed based on the matching point pairs in the matched marker structure, thus obtaining the 3D point cloud of the marker structure. That is, based on the matching point pairs of the marker structure in the left and right eye images, the 3D coordinates of the marker structure are calculated through ray tracing modeling and a self-calibration method. Specifically, the matching point pairs are modeled using ray tracing, and 3D reconstruction is performed using a self-calibration method based on refraction parameters. Figure 4 This is a schematic diagram of the point cloud for three-dimensional reconstruction provided by the present invention, such as... Figure 4 As shown, a 3D point cloud of the marked structure can be obtained through ray tracing modeling.

[0092] Based on the above embodiments, step 150 includes:

[0093] Based on the real-time 3D point cloud of the marker structure, determine the 3D point cloud displacement sequence of the marker structure;

[0094] The three-dimensional point cloud displacement sequence is input into the six-axis force measurement model to obtain the six-dimensional force vector of the force platform output by the six-axis force measurement model;

[0095] The six-axis force measurement model is trained based on the displacement sequence of the sample three-dimensional point cloud of the sample marker structure and the sample six-dimensional force vector corresponding to the sample marker structure.

[0096] Specifically, the process of determining the six-dimensional force vector of the force-bearing platform based on the real-time three-dimensional point cloud of the marked structure can include:

[0097] First, based on the three-dimensional point cloud of the marker structure in multiple consecutive frames, the point cloud changes of the marker structure can be determined, and its displacement can be calculated, thereby obtaining multiple displacement sequences of the marker structure, that is, its three-dimensional point cloud displacement sequence.

[0098] Subsequently, six-axis force / torque measurements can be performed based on this three-dimensional point cloud displacement sequence, thereby obtaining the six-axis force / torque information of the force-bearing platform, which can be represented by a six-dimensional force vector. This six-dimensional force vector can include the forces on the force-bearing platform in three directions, as well as the torques in those three directions.

[0099] Here, the measurement of six-axis force / torque can be achieved using a model. That is, a three-dimensional point cloud displacement sequence can be input into a six-axis force measurement model, so that the model can predict six-axis force / torque information based on the input three-dimensional point cloud displacement sequence. Figure 5 This is a flowchart of the measurement process for the six-dimensional force vector provided by the present invention, as follows: Figure 5 As shown, the input three-dimensional point cloud displacement sequence can first be processed by a multilayer perceptron (MLP). Then, the temporal features can be extracted by a long short-term memory (LSTM) network, and prediction can be made based on these features. Finally, the estimated values ​​of the six-dimensional force / torque predicted by the model can be obtained, which are the six-dimensional force vectors of the force-bearing platform.

[0100] However, it is worth noting that before applying the six-axis force measurement model for six-axis force / torque measurement, the model can be pre-trained using the sample three-dimensional point cloud displacement sequence of the sample marker structure and the corresponding sample six-dimensional force vector. The training process of the six-axis force measurement model includes: first, collecting a large number of sample marker images and determining the sample three-dimensional point cloud displacement sequence of the sample marker structure and the corresponding six-dimensional force vector, thus obtaining the sample six-dimensional force vector corresponding to the sample marker structure; the process of determining the sample three-dimensional point cloud displacement sequence is basically the same as the process of determining the three-dimensional point cloud displacement sequence in actual applications, and this process has been described in detail above and will not be repeated here; next, this sample three-dimensional point cloud displacement sequence and sample six-dimensional force vector can be used to train the initial model, so that the model learns the mapping relationship between the input sample three-dimensional point cloud displacement sequence and the label (sample six-dimensional force vector). Then, in actual applications, predictions can be made based on this mapping relationship, mapping the input three-dimensional point cloud displacement sequence into a six-dimensional vector containing forces in three directions and moments in three directions.

[0101] Here, the initial model during training can be built based on a deep neural network. After training with sample 3D point cloud displacement sequences and sample 6D force vectors, a well-trained model can be obtained. Furthermore, this model can be tested to assess its performance in six-axis force / torque measurement. Figure 6 This is a test result diagram of the six-dimensional force vector provided by the present invention. See [link / reference]. Figure 6 It can be seen that the model trained by the sample three-dimensional point cloud displacement sequence and sample six-dimensional force vector in the embodiments of the present invention outputs a predicted value that is very close to the true value. This shows that the model has excellent performance in six-axis force / torque measurement, can accurately predict the estimated value of six-dimensional force / torque, and obtain an accurate and reliable six-dimensional force vector.

[0102] The force-position hybrid sensing device provided by the present invention is described below. The force-position hybrid sensing device described below can be referred to in correspondence with the rigid-flexible coupling force-position hybrid sensing method described above.

[0103] Figure 7 This is a schematic diagram of the force-position hybrid sensing device provided by the present invention, as shown below. Figure 7 As shown, the device includes a force-bearing platform and a binocular camera; a marker structure is provided on the force-bearing platform, and the binocular camera is used to acquire structural images of the marker structure on the force-bearing platform; the device also includes an image acquisition unit 710, an image processing unit 720, a three-dimensional reconstruction unit 730, a pose estimation unit 740, and a force estimation unit 750.

[0104] The image acquisition unit 710 is used to acquire the structural image in real time, and the structural image includes images of the marked structure of the force-bearing platform in both the force-bearing state and the unforced state.

[0105] The image processing unit 720 is used to extract coordinates based on the structural image to obtain the two-dimensional coordinates of the marked structure.

[0106] The three-dimensional reconstruction unit 730 is used to perform three-dimensional reconstruction based on the two-dimensional coordinates to obtain a three-dimensional point cloud of the marker structure.

[0107] The pose estimation unit 740 is used to determine the end pose of the force-bearing platform based on the three-dimensional point cloud under the force state and the three-dimensional point cloud under the unforced state.

[0108] The force estimation unit 750 is used to determine the six-dimensional force vector of the force-bearing platform based on the real-time three-dimensional point cloud of the marked structure.

[0109] The force-position hybrid sensing device provided by this invention includes a force-bearing platform, a binocular camera, an image acquisition unit, an image processing unit, a 3D reconstruction unit, a pose estimation unit, and a force estimation unit. A marker structure is disposed on the force-bearing platform, and the binocular camera is used to acquire structural images of the marker structure on the force-bearing platform. The image acquisition unit is used to acquire structural images in real time, including images of the marker structure on the force-bearing platform under both stressed and unstressed conditions. The image processing unit is used to extract coordinates based on the structural images to obtain the two-dimensional coordinates of the marker structure. The 3D reconstruction unit is used to perform 3D reconstruction based on the two-dimensional coordinates to obtain the 3D point cloud of the marker structure. Pose estimation is performed. The unit is used to determine the end-effector pose of the stressed platform based on the 3D point cloud under stress and the 3D point cloud under no stress. The force estimation unit is used to determine the six-dimensional force vector of the stressed platform based on the real-time labeled 3D point cloud. This overcomes the shortcomings of traditional separate sensing methods, which cannot simultaneously sense six-dimensional pose and six-axis force / torque information, making it difficult to meet the needs of multi-dimensional interactive perception. Through the force-position hybrid sensing device, the end-effector pose and six-axis force / torque information can be accurately acquired at the same time. This can meet the needs of the robot operation field for lightweight, integrated and multi-dimensional interactive perception, enabling the robot to cope with complex tasks and improve the operation accuracy.

[0110] Based on the above embodiments, the pose estimation unit 740 is used for:

[0111] Point cloud registration is performed between the three-dimensional point cloud under the stress state and the three-dimensional point cloud under the no-stress state to obtain the rotation matrix and translation vector between the three-dimensional point cloud under the stress state and the three-dimensional point cloud under the no-stress state.

[0112] The end pose of the force-bearing platform is determined based on the rotation matrix and the translation vector.

[0113] Based on the above embodiments, the pose estimation unit 740 is used for:

[0114] Based on the rotation matrix and the translation vector, construct the rigid body transformation matrix;

[0115] Based on the rigid body transformation matrix, the end pose of the force-bearing platform is determined.

[0116] Based on the above embodiments, the image processing unit 720 is used for:

[0117] The structure image is processed to obtain an enhanced structure image;

[0118] Centroid extraction is performed on the enhanced structure image to obtain the two-dimensional coordinates of the marked structure;

[0119] The data processing includes at least one of Gaussian filtering, image differencing, binarization, and morphological operations.

[0120] Based on the above embodiments, the three-dimensional reconstruction unit 730 is used for:

[0121] By performing circular grouping, the two-dimensional coordinates of the marked structure are matched to obtain the matching point pairs in the marked structure;

[0122] By using ray tracing modeling, the matching point pairs in the marker structure are reconstructed in three dimensions to obtain the three-dimensional point cloud of the marker structure.

[0123] Based on the above embodiments, the force estimation unit 750 is used for:

[0124] Based on the real-time 3D point cloud of the marked structure, the displacement sequence of the 3D point cloud of the marked structure is determined;

[0125] The three-dimensional point cloud displacement sequence is input into the six-axis force measurement model to obtain the six-dimensional force vector of the force-bearing platform output by the six-axis force measurement model;

[0126] The six-axis force measurement model is obtained by training based on the displacement sequence of the sample three-dimensional point cloud of the sample marker structure and the sample six-dimensional force vector corresponding to the sample marker structure.

[0127] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logic instructions in the memory 830 to execute a rigid-flexible coupled force-position hybrid sensing method. This method is applied to a force-position hybrid sensing device, which includes a force-bearing platform and a binocular camera. A marker structure is disposed on the force-bearing platform, and the binocular camera is used to acquire structural images of the marker structure on the force-bearing platform. The method includes: acquiring the structural image in real time, the structural image containing images of the marker structure of the force-bearing platform in both a force-bearing state and an unforced state; extracting coordinates based on the structural image to obtain two-dimensional coordinates of the marker structure; performing three-dimensional reconstruction based on the two-dimensional coordinates to obtain a three-dimensional point cloud of the marker structure; determining the end pose of the force-bearing platform based on the three-dimensional point cloud in the force-bearing state and the three-dimensional point cloud in the unforced state; and determining the six-dimensional force vector of the force-bearing platform based on the real-time three-dimensional point cloud of the marker structure.

[0128] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, which, when executed by a computer, enable the computer to execute the rigid-flexible coupling force-position hybrid sensing method provided by the above methods. This method is applied to a force-position hybrid sensing device, the force-position hybrid sensing device comprising a force platform and a binocular camera, wherein a marker structure is disposed on the force platform, and the binocular camera is used to acquire structural images of the marker structure on the force platform. The method comprises: acquiring the structural image in real time, the structural image including images of the marker structure of the force platform under force and without force; extracting coordinates based on the structural image to obtain two-dimensional coordinates of the marker structure; performing three-dimensional reconstruction based on the two-dimensional coordinates to obtain a three-dimensional point cloud of the marker structure; determining the end pose of the force platform based on the three-dimensional point cloud under force and the three-dimensional point cloud under without force; and determining a six-dimensional force vector of the force platform based on the real-time three-dimensional point cloud of the marker structure.

[0130] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the rigid-flexible coupling force-position hybrid sensing method provided by the above methods. This method is applied to a force-position hybrid sensing device, which includes a force-bearing platform and a binocular camera. A marker structure is disposed on the force-bearing platform, and the binocular camera is used to acquire structural images of the marker structure on the force-bearing platform. The method includes: acquiring the structural image in real time, the structural image including images of the marker structure of the force-bearing platform in a force-bearing state and an unforced state; extracting coordinates based on the structural image to obtain two-dimensional coordinates of the marker structure; performing three-dimensional reconstruction based on the two-dimensional coordinates to obtain a three-dimensional point cloud of the marker structure; determining the end pose of the force-bearing platform based on the three-dimensional point cloud in the force-bearing state and the three-dimensional point cloud in the unforced state; and determining the six-dimensional force vector of the force-bearing platform based on the real-time three-dimensional point cloud of the marker structure.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A force-position hybrid sensing method with rigid-flexible coupling, characterized in that, An application is made in a force-potential hybrid sensing device, the force-potential hybrid sensing device comprising a force-receiving platform and a binocular camera, wherein a marker structure is disposed on the force-receiving platform, and the binocular camera is used to acquire structural images of the marker structure on the force-receiving platform, the method comprising: The structural image is acquired in real time, and the structural image includes images of the marked structure of the force-bearing platform under both stressed and unstressed states. Based on the structural image, coordinate extraction is performed to obtain the two-dimensional coordinates of the marked structure; Based on the two-dimensional coordinates, a three-dimensional reconstruction is performed to obtain the three-dimensional point cloud of the marked structure; Based on the three-dimensional point cloud under the stress state and the three-dimensional point cloud under the unstressed state, the end pose of the stress-bearing platform is determined. Based on the real-time three-dimensional point cloud of the marked structure, the six-dimensional force vector of the force-bearing platform is determined; The determination of the six-dimensional force vector of the force-bearing platform based on the real-time three-dimensional point cloud of the marked structure includes: Based on the real-time 3D point cloud of the marked structure, the displacement sequence of the 3D point cloud of the marked structure is determined; The three-dimensional point cloud displacement sequence is input into the six-axis force measurement model to obtain the six-dimensional force vector of the force-bearing platform output by the six-axis force measurement model; The six-axis force measurement model is obtained by training the sample three-dimensional point cloud displacement sequence based on the sample marker structure and the sample six-dimensional force vector corresponding to the sample marker structure.

2. The rigid-flexible coupled force-position hybrid sensing method according to claim 1, characterized in that, Determining the end pose of the stressed platform based on the 3D point cloud under the stressed state and the 3D point cloud under the unstressed state includes: Point cloud registration is performed between the three-dimensional point cloud under the stress state and the three-dimensional point cloud under the no-stress state to obtain the rotation matrix and translation vector between the three-dimensional point cloud under the stress state and the three-dimensional point cloud under the no-stress state. The end pose of the force-bearing platform is determined based on the rotation matrix and the translation vector.

3. The rigid-flexible coupled force-position hybrid sensing method according to claim 2, characterized in that, Determining the end pose of the force-bearing platform based on the rotation matrix and the translation vector includes: Based on the rotation matrix and the translation vector, construct the rigid body transformation matrix; Based on the rigid body transformation matrix, the end pose of the force-bearing platform is determined.

4. The rigid-flexible coupling force-position hybrid sensing method according to any one of claims 1 to 3, characterized in that, The step of extracting coordinates based on the structure image to obtain the two-dimensional coordinates of the marked structure includes: The structure image is processed to obtain an enhanced structure image; Centroid extraction is performed on the enhanced structure image to obtain the two-dimensional coordinates of the marked structure; The data processing includes at least one of Gaussian filtering, image differencing, binarization, and morphological operations.

5. The rigid-flexible coupling force-position hybrid sensing method according to any one of claims 1 to 3, characterized in that, The process of performing three-dimensional reconstruction based on the two-dimensional coordinates to obtain the three-dimensional point cloud of the marked structure includes: By performing circular grouping, the two-dimensional coordinates of the marked structure are matched to obtain the matching point pairs in the marked structure; By using ray tracing modeling, the matching point pairs in the marker structure are reconstructed in three dimensions to obtain the three-dimensional point cloud of the marker structure.

6. A force-position hybrid sensing device, characterized in that, It includes a force-bearing platform, a binocular camera, an image acquisition unit, an image processing unit, a 3D reconstruction unit, a pose estimation unit, and a force estimation unit; the force-bearing platform is provided with a marker structure, and the binocular camera is used to acquire structural images of the marker structure on the force-bearing platform; The image acquisition unit is used to acquire the structure image in real time, and the structure image includes images of the marked structure of the force-bearing platform in both the force-bearing state and the unforced state. The image processing unit is used to extract coordinates based on the structural image to obtain the two-dimensional coordinates of the marked structure. The three-dimensional reconstruction unit is used to perform three-dimensional reconstruction based on the two-dimensional coordinates to obtain a three-dimensional point cloud of the marked structure. The pose estimation unit is used to determine the end pose of the force-bearing platform based on the three-dimensional point cloud under the force-bearing state and the three-dimensional point cloud under the unforced state. The force estimation unit is used to determine the six-dimensional force vector of the force-bearing platform based on the real-time three-dimensional point cloud of the marked structure. The determination of the six-dimensional force vector of the force-bearing platform based on the real-time three-dimensional point cloud of the marked structure includes: Based on the real-time 3D point cloud of the marked structure, the displacement sequence of the 3D point cloud of the marked structure is determined; The three-dimensional point cloud displacement sequence is input into the six-axis force measurement model to obtain the six-dimensional force vector of the force-bearing platform output by the six-axis force measurement model; The six-axis force measurement model is obtained by training the sample three-dimensional point cloud displacement sequence based on the sample marker structure and the sample six-dimensional force vector corresponding to the sample marker structure.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the rigid-flexible coupling force-position hybrid sensing method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the rigid-flexible coupling force-position hybrid sensing method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the rigid-flexible coupling force-position hybrid sensing method as described in any one of claims 1 to 5.

Images