A massage robot trajectory generation method based on non-coplanar coding visual markers

Abstract

The application provides a massage robot trajectory generation method based on a non-coplanar coding visual marker, belongs to the technical field of machine vision, simulates a massage action through a handheld acquisition tool, detects a visual marker frame by frame by a camera, solves a six-degree-of-freedom pose of an action point in a camera coordinate system through a multi-marker joint PnP algorithm, carries out quality screening on a recorded massage trajectory, stores an effective trajectory as a camera coordinate system source trajectory for single storage, determines a locking pose of a reference datum tool relative to the camera coordinate system after triggering a coordinate conversion instruction, carries out trajectory coordinate system conversion based on the source trajectory, the locking pose, and a fixed transformation relationship between the reference datum tool and a mechanical arm base coordinate system, and generates a trajectory file containing position and attitude information for a robot control system to read. Thus, the teaching difficulty and equipment cost are reduced, and the trajectory precision, stability and robot reproduction effect are improved.

Description

Technical Field

[0001] This application belongs to the field of machine vision technology, and more specifically, relates to a method for generating the trajectory of a massage robot based on non-coplanar coded visual markers. Background Technology

[0002] With the development of service robot technology, massage robots are gradually being applied in scenarios such as health and wellness, physiotherapy, and home use. Robot teaching is a core step in achieving personalized massage motion replication. Currently, the industry mainly uses the following methods for acquiring and teaching massage robot trajectories: First, relying on human description, 2D screen drawing, and single-point recording to plan the massage path. This method cannot reproduce the continuous three-dimensional motion trajectory of the curved parts of the human body, nor can it represent the posture and contact direction of the massage tool; Second, using robotic arm drag teaching, directly dragging the robotic arm to record the motion path, which has problems such as high operational difficulty, poor human contact experience, and harsh on-site deployment conditions; Third, using planar visual markers such as single-plane QR codes, or using ordinary visual key points for positioning, estimating the tool's pose by recognizing planar markers, but this method is not suitable for the human shoulder and back areas. In complex curved surface operation scenarios such as the waist, back, buttocks and legs, handheld tools are prone to translation, pitch and roll, and problems such as hand occlusion, viewpoint shift and marking of the field of view are superimposed, which can easily cause recognition interruption and posture jitter. Moreover, there is no rigid body geometry consistency verification mechanism, resulting in poor positioning stability. Fourth, the use of master-slave operation devices, traditional posture identification systems or high-precision optical ball marking systems is problematic. The former is mostly used in surgical robots and industrial manipulator control, and lacks a complete business link for trajectory acquisition-storage-coordinate transformation for human massage scenarios. The latter has high equipment costs and is difficult to integrate with conventional RGB-D cameras, making it difficult to scale up in massage robot scenarios.

[0003] It is evident that existing trajectory acquisition and teaching solutions have significant shortcomings in terms of positioning stability, ease of operation, cost control, trajectory quality, and scene adaptability, and cannot meet the needs of generating high-precision and highly flexible massage trajectories for curved parts of the human body. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method for generating the trajectory of a massage robot based on non-coplanar coded visual markers. By combining two sets of non-coplanar stereo vision tooling with machine vision algorithms, the method enables handheld teaching of massage trajectories, multiple reuses of a single data acquisition, reduces teaching difficulty and equipment cost, and improves trajectory accuracy, stability and robot reproduction effect.

[0005] This application provides a method for generating the trajectory of a massage robot based on non-coplanar coded visual markers, the method comprising the following steps: Configure a handheld acquisition fixture and a reference fixture; the handheld acquisition fixture and the reference fixture are equipped with multiple non-coplanar coded visual markers with unique identifiers and corner features; Massage motions are simulated by handheld acquisition fixtures. The camera detects visual markers frame by frame and distinguishes the fixture type. The six-degree-of-freedom pose of the point of action in the camera coordinate system is solved by multi-marker joint PnP algorithm to complete the acquisition of massage trajectory. The recorded massage trajectories are screened online according to the trajectory processing threshold, and the effective trajectories obtained from the screening are stored as the source trajectories of the camera coordinate system once. After triggering the coordinate transformation command, the reference reference fixture is detected and stability is verified by combining the sliding window and multiple indicators; once the indicators meet the standards, the locked pose of the reference reference fixture relative to the camera coordinate system is determined. Based on the source trajectory, locked pose, and the fixed transformation relationship between the reference reference tooling and the robot arm's base coordinate system, trajectory coordinate system transformation is performed; The six-degree-of-freedom trajectory in the converted robotic arm base coordinate system is encapsulated to generate a trajectory file containing position and attitude information for the robot control system to read.

[0006] In some embodiments, solving the six-DOF pose of the point of action in the camera coordinate system using a multi-marker joint PnP algorithm includes the following steps: After the camera captures a single frame image, it identifies all coded visual markers in the image one by one, extracts the two-dimensional pixel corner point corresponding to each marker, and binds the unique ID of each marker. Retrieve the pre-stored 3D rigid body model of the handheld acquisition tool based on the unique ID, and match the coordinates of the corresponding marked 3D spatial corner points; The matching relationship is established for all visible markers belonging to the handheld acquisition tool and their corresponding two-dimensional and three-dimensional corner points. All matching point pairs are then integrated and substituted into the PnP algorithm to solve the problem, thereby obtaining the six-degree-of-freedom overall pose of the entire handheld acquisition tool relative to the camera coordinate system in one go.

[0007] In some embodiments, the origin of the coordinate system of the handheld acquisition tool is set at the center of the spherical action point at its end, and the pose solution directly characterizes the spatial position and attitude of the action point in the camera coordinate system.

[0008] In some embodiments, the online quality screening of the recorded massage trajectories according to a trajectory processing threshold, and the storage of the selected valid trajectories as source trajectories in a single instance using the camera coordinate system, includes the following steps: After the massage trajectory is collected, online quality screening is performed based on pre-configured multiple trajectory processing thresholds. Among them, the pre-configured trajectory processing thresholds include visual marker imaging threshold, inter-frame motion smoothness threshold, and effective marker recognition threshold. The selected valid trajectories are defined as source trajectories in the camera coordinate system and stored once. The stored source trajectories are then subjected to multi-level post-processing, which includes one or more of outlier removal, moving average filtering, distance downsampling, and equal arc length resampling. At least one of the following is set for the source trajectory: version identifier, time identifier, task unique identifier, overwrite confirmation mechanism, permission verification mechanism, or verification information, in order to manage the repeated writing, overwriting, or modification of the source trajectory.

[0009] In some embodiments, a sliding window mechanism is adopted to determine stability based on one or more of the following: number of visible markers, single-frame reprojection error, average pixel side length of markers, position deviation of consecutive frames, pose deviation of consecutive frames, and number of outliers. When the number of valid frames or the proportion of valid frames that meet the preset conditions in the window reaches a preset threshold, the locked pose of the reference fixture relative to the camera coordinate system is determined according to the average pose, weighted average pose, or robust estimated pose of the valid frames.

[0010] In some embodiments, trajectory coordinate system transformation is performed using the following formula: T_T2_T1=(T_C_T2_locked)^(-1)×T_C_T1 T_B_P=T_B_T2×T_T2_T1×T_T1_P Wherein, T_C_T1 is the six-DOF rigid body transformation matrix of the handheld acquisition fixture relative to the camera coordinate system C, T_C_T2_locked is the rigid body transformation matrix of the reference fixture relative to the camera coordinate system C, T_T2_T1 is the rigid body transformation matrix of the handheld acquisition fixture relative to the reference fixture coordinate system, T_B_T2 is the fixed rigid body transformation matrix of the reference fixture relative to the robotic arm base coordinate system B, T_B_P is the final six-DOF rigid body transformation matrix of the massage application point relative to the robotic arm base coordinate system B, and T_T1_P is the fixed rigid body transformation matrix from the handheld acquisition fixture coordinate system T1 to the massage application point coordinate system P; when the origin of the handheld acquisition fixture coordinate system T1 coincides with the massage application point P, T_T1_P is the unit transformation matrix.

[0011] In some embodiments, the reference reference fixture and the robot arm base are rigidly fixed by means of screw connection, locating pin, rigid clamp, or integrated machining; if there are installation position and / or attitude deviations, the complete six-degree-of-freedom rigid body transformation relationship between the reference reference fixture and the robot arm base coordinate system is determined by external calibration.

[0012] In some embodiments, the method further includes the following steps: Read the historically saved source trajectory file and restore the coordinate system definition, pose expression method, time information and quality information of the source trajectory; after completing the locking of the reference datum fixture and the determination of coordinate relationships, perform coordinate transformation on the historical source trajectory again.

[0013] In some embodiments, the method further includes the following steps: When the camera position changes during the trajectory acquisition and coordinate transformation stages, the rigid body transformation relationship between the original camera coordinate system and the current camera coordinate system is determined by using fixed world reference markers, external positioning devices, environmental feature matching, multiple reference reference fixtures or their combinations; after updating the source trajectory based on the rigid body transformation relationship, reference reference fixture locking and trajectory transformation are then performed.

[0014] In some embodiments, a corresponding robot adaptation parameter set is invoked for different robotic arm devices; the robot adaptation parameter set includes one or more of the following: coordinate axis mapping relationship, length unit conversion relationship, posture representation method, rigid body transformation relationship from reference tooling to robotic arm base coordinate system, trajectory file format rules, and kinematic constraints.

[0015] In some embodiments, the robot control system performs one or more of the following processes based on the trajectory file: unit conversion, coordinate axis mapping, posture expression transformation, time parameter generation, velocity planning, inverse kinematics solution, safety verification, and control command generation.

[0016] In some embodiments, a massage robot trajectory generation device based on non-coplanar coded visual markers is also provided, the device comprising: The configuration module is used to configure the handheld acquisition fixture and the reference fixture; the handheld acquisition fixture and the reference fixture are equipped with multiple non-coplanar coded visual markers with unique identifiers and corner features; The acquisition module is used to simulate massage movements using a handheld acquisition fixture. The camera detects visual markers frame by frame and distinguishes the fixture type. The multi-marker joint PnP algorithm is used to solve the six-degree-of-freedom pose of the point of action in the camera coordinate system to complete the acquisition of the massage trajectory. The storage module is used to perform online quality screening of the recorded massage trajectories according to the trajectory processing threshold, and to store the selected valid trajectories as the source trajectories of the camera coordinate system once. The locking module is used to detect the reference reference fixture after triggering the coordinate transformation command and perform stability verification by combining the sliding window and multiple indicators; after the indicators meet the standards, the locked pose of the reference reference fixture relative to the camera coordinate system is determined. The transformation module is used to perform trajectory coordinate system transformation based on the source trajectory, locked pose, and fixed transformation relationship between the reference reference tooling and the robot arm base coordinate system; The generation module is used to encapsulate the six-degree-of-freedom trajectory in the converted robot arm base coordinate system, generating a trajectory file containing position and attitude information for the robot control system to read.

[0017] In some embodiments, an electronic device is also provided, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the massage robot trajectory generation method based on non-coplanar coded visual markers described above are performed.

[0018] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the massage robot trajectory generation method based on non-coplanar coded visual markers as described above.

[0019] This application describes a method for generating the trajectory of a massage robot based on non-coplanar coded visual markers. It constructs a rigid three-dimensional body using multiple non-coplanar stereo visual markers and solves the six-degree-of-freedom pose using a multi-marker combined with a PnP algorithm, significantly improving the positioning accuracy and stability during handheld teaching. Simultaneously, it decouples trajectory acquisition from the reference coordinate system, eliminating the need for the reference fixture to continuously appear in the camera's field of view during recording, reducing limitations on on-site layout and shooting angles, and adapting to various human body massage teaching scenarios. Furthermore, the camera coordinate system source trajectory is only acquired and saved once, and subsequent reference fixture locking and coordinate transformation can be triggered multiple times. The same teaching trajectory can be repeatedly transformed to adapt to different installation conditions and different robotic arm devices, eliminating the need for repeated handheld teaching and significantly improving the efficiency of robot debugging and motion replication. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of the massage robot trajectory generation method based on non-coplanar coded visual markers according to an embodiment of this application is shown; Figure 2 A schematic diagram of the handheld data acquisition tool according to an embodiment of this application is shown; Figure 3 A schematic diagram of the structure of the reference tooling according to an embodiment of this application is shown; Figure 4This illustration shows a schematic diagram of the multi-label joint PnP algorithm solution in an embodiment of this application; Figure 5 This application illustrates a flowchart of trajectory post-processing and data output according to an embodiment of the present application. Figure 6 A schematic diagram of the multi-coordinate system rigid body transformation relationship in an embodiment of this application is shown; Figure 7 A schematic diagram of the structure of the massage robot trajectory generation device based on non-coplanar coded visual markers according to an embodiment of this application is shown; Figure 8 A schematic diagram of the structure of the electronic device described in an embodiment of this application is shown. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0023] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0024] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0025] In view of the technical problems raised in the background art, this application provides a method for generating massage robot trajectories based on non-coplanar coded visual markers. By combining two sets of non-coplanar stereo vision tooling with machine vision algorithms, the method enables handheld teaching of massage trajectories and multiple reuses of a single data acquisition, thereby reducing the difficulty of teaching and equipment costs, and improving trajectory accuracy, stability and robot reproduction effect.

[0026] See the instruction manual appendix Figure 1 This application provides a method for generating the trajectory of a massage robot based on non-coplanar coded visual markers, the method comprising the following steps: S1. Configure handheld acquisition fixture and reference reference fixture; wherein, the handheld acquisition fixture and reference reference fixture are equipped with multiple non-coplanar coded visual markers with unique identifiers and corner features; S2. The massage motion is simulated by hand-held acquisition tool. The camera detects visual markers frame by frame and distinguishes the tool type. The six-degree-of-freedom pose of the point of action in the camera coordinate system is solved by multi-marker joint PnP algorithm to complete the acquisition of massage trajectory. S3. Perform online quality screening on the recorded massage trajectory according to the trajectory processing threshold, and store the selected valid trajectory as the source trajectory of the camera coordinate system once. S4. After triggering the coordinate transformation command, the reference reference fixture is detected and stability is verified by combining the sliding window and multiple indicators; after the indicators meet the standards, the locked pose of the reference reference fixture relative to the camera coordinate system is determined. S5. Based on the source trajectory, locked pose, and the fixed transformation relationship between the reference reference tooling and the robot arm base coordinate system, perform trajectory coordinate system transformation; S6. Encapsulate the six-degree-of-freedom trajectory in the converted robotic arm base coordinate system to generate a trajectory file containing position and attitude information for the robot control system to read.

[0027] In step S1, the visual markers on the surfaces of the two types of fixtures (handheld acquisition fixture and reference fixture) are dispersed in non-coplanar positions in space. Multiple sets of independent markers form a three-dimensional rigid body constraint, providing a geometric basis for subsequent multi-marker joint PnP high-precision pose calculation.

[0028] In a preferred embodiment, the visual markers use the AprilTag 36h11 dictionary; in other embodiments, the ArUco dictionary or other planar visual markers that can provide a unique ID and four corner points can also be used. To ensure detection reliability, each marker, in addition to the detection code body, also has a reserved outer white border (quiet zone) to isolate the surrounding environmental textures and color blocks from interfering with the marker contour recognition, ensuring that the four corner contours of the marker are extracted completely and clearly during the image recognition stage; the reference size parameter tag_size_mm used in the PnP operation of the pose calculation stage of this application only refers to the single-side length value of the internal identification code body, and does not include the outer white border area or the overall outer contour size of the marker patch.

[0029] See the instruction manual appendix Figure 2The image shows the side and top views of the handheld acquisition fixture, which has five independent visual marker mounting surfaces. Each marker mounting surface has an overall size of 45 mm × 45 mm, and the marker detection code body has a side length of 40 mm. A 2.5 mm white margin is evenly reserved around the code body. The handheld acquisition fixture uses an externally predefined rigid body coordinate system, which follows general visual calibration standards. The unique ID, detection area size, center 3D coordinates, marker plane normal vector, and marker local coordinate direction or four 3D corner coordinates of each coded visual marker are pre-stored in the fixture's rigid body model. The system establishes the matching relationship between the 2D and 3D corner points of the coded visual markers based on the fixture's rigid body model. Specific layout parameters are shown in Table 1 below.

[0030]

[0031] Table 1 See the instruction manual appendix Figure 3 The reference fixture has four independent visual marker mounting surfaces. Each mounting surface is 50 mm × 50 mm in size, and the marker detection code body has a side length of 44 mm. A 3.0 mm white margin is evenly reserved around the code body. The reference fixture also uses an externally defined rigid body coordinate system, which follows general visual calibration standards. The unique ID, detection area size, center 3D coordinates, marker plane normal vector, and marker local coordinate direction or four 3D corner coordinates of each coded visual marker are pre-stored in the fixture's rigid body model. The system establishes the matching relationship between the 2D and 3D corners of the coded visual markers based on the fixture's rigid body model. Specific layout parameters are shown in Table 2 below.

[0032]

[0033] Table 2 It should also be noted that before executing step S2, system initialization is required to complete the global loading and preparation of the hardware and software environment, algorithm parameters, and calibration data. The color camera is started and pre-calibrated camera intrinsic parameters and distortion parameters are loaded; simultaneously, the visual marker IDs, geometric dimensions, 3D corner points, and mounting poses of the handheld acquisition fixture and reference fixture are loaded, along with pose determination, trajectory processing thresholds, and algorithm configuration parameters. After the parameters are loaded, the camera starts a continuous image data stream.

[0034] Step S2 mainly uses machine vision combined with multi-label joint PnP algorithm to identify the spatial pose of the handheld acquisition tool in real time, continuously acquire the original three-dimensional six-degree-of-freedom trajectory corresponding to the massage action on the human body surface, and restore the real massage path, tool posture and contact direction.

[0035] Specifically, the operator holds a handheld acquisition fixture, fitting it to the curved areas of the body such as the shoulders, back, waist, buttocks, and legs for massage, simulating real massage techniques, movement paths, and contact postures for demonstration and instruction. During this process, a color camera continuously outputs a real-time image stream, performing visual marker detection and analysis on each frame: first, it traverses all identified planar visual markers within the image, classifying the fixture based on its unique ID, and accurately selecting all valid markers belonging to the handheld acquisition fixture.

[0036] For the selected visual markers on the handheld acquisition fixture, the corner points of the corresponding 2D images and the pre-stored corner point data of the 3D model are integrated, and a multi-marker joint PnP algorithm is used to jointly solve the six-degree-of-freedom pose. Relying on the constraints of multiple spatially non-coplanar markers, the spatial position and attitude of the handheld acquisition fixture relative to the camera coordinate system are calculated. Since the origin of the coordinate system of the handheld acquisition fixture is preset at the center of the spherical action point at the end, the obtained pose data can directly represent the real-time six-degree-of-freedom information of the massage action point in the camera coordinate system. In this way, the motion trajectory, contact angle and spatial attitude of the action point are recorded frame by frame to complete the continuous trajectory acquisition of the entire human massage path.

[0037] The logic for the joint solution of six-DOF pose using the multi-marker joint PnP algorithm can be found in the appendix of the instruction manual. Figure 4 The image plane (2D corner points) represents the pixel information of the image obtained after a single frame is captured by a color camera. Multiple visual markers belonging to the same tooling exist within the image (Tag A, Tag B, and Tag C in the image). Each square coded visual marker extracts four 2D corner points (e.g., Tag A-1 to Tag A-4), and all corner points have a corresponding marker ID, recording the 2D coordinates of each corner point within the image pixel plane. When the operator moves the tooling or there is partial occlusion, the complete set of visible 2D corner points within the image is still retained as visual input data. The rigid body model (3D corner points) is the pre-stored 3D rigid body geometric model data of the tooling. Based on the tooling processing dimensions and marker installation positions, the spatial 3D coordinates of the four 3D corner points corresponding to each marker in the tooling's own coordinate system are pre-stored. The relative positions of all Tag A, Tag B, and Tag C 3D corner points under the same tooling remain fixed, forming a unified 3D rigid body constraint, unaffected by shooting angle or occlusion. Then, relying on the unique ID of the marker, the 2D corner points extracted from the image plane are paired one by one with the 3D corner points pre-stored in the rigid body model. All visible marked corner points under the same fixture are included in the same matching set without splitting individual markers for independent calculation. All paired 2D and 3D corner point pairs are integrated for unified calculation, and multiple sets of spatial constraints are used to jointly solve the overall six-degree-of-freedom pose of the entire fixture relative to the camera.

[0038] Furthermore, to ensure the integrity and effectiveness of trajectory acquisition, the recognition status of the handheld acquisition tool's markers is monitored in real time while solving the pose: when situations such as hand occlusion, viewpoint shift, or markers moving out of the frame occur, resulting in the tool's visual markers being lost for a long time or multiple consecutive frames failing to be recognized normally, the recording process will be automatically paused to avoid generating invalid or discontinuous trajectory data; once the markers are recognized normally again, recording can be manually resumed as needed.

[0039] This application decouples trajectory acquisition from the reference coordinate system, eliminating the need for the reference reference fixture to be simultaneously present in the camera's imaging field of view throughout the teaching process. This eliminates the mandatory requirement for the reference marker to be continuously visible during the teaching phase and reduces constraints on the on-site shooting layout.

[0040] In step S3, after the massage trajectory acquisition is completed, the raw continuous trajectory data obtained in step S2 is immediately subjected to online quality screening. Each frame of trajectory data is verified one by one according to a preset threshold, with a focus on verifying the reprojection error corresponding to the single-frame pose, the average pixel side length of the visual marker, and the position jump and attitude jump variables between adjacent frames. Abnormal frames, jittery frames, and misidentified frames exceeding the threshold range are directly removed, retaining only valid trajectory data that meets all the standards. After screening, the valid trajectories in this set of camera coordinate systems are stored as the original source trajectories. The system can manage the repeated writing, overwriting, or modification of the source trajectories through at least one of the following: version identifier, time identifier, unique task identifier, overwrite confirmation, permission verification, or verification information, to improve the traceability and reliability of the source trajectory data. In some embodiments, the system can read historically saved source trajectory files and restore the coordinate system definition, pose expression method, time information, and quality information of the source trajectory. After completing the locking of the corresponding reference datum fixture and the determination of coordinate relationships, coordinate transformation can be performed again on the historical source trajectories.

[0041] See the instruction manual appendix Figure 5After the source trajectory is saved, the full-link trajectory post-processing process is automatically initiated, performing data processing operations in a predetermined order: First, isolated outliers and spike noise are removed from the trajectory; then, the entire trajectory is smoothed using a moving average filtering algorithm to eliminate small-amplitude attitude and positional jitter; subsequently, trajectory point distance downsampling is performed according to a set minimum distance threshold to simplify redundant data points; finally, equal arc length resampling is used to ensure that trajectory points are evenly distributed along the motion path. Each of the above processing steps independently generates a corresponding trajectory file: cleaned represents the trajectory after removing isolated outliers; filtered represents the trajectory after moving average filtering; downsampled represents the trajectory after minimum distance sparsification; resampled represents the trajectory after equal arc length resampling; and final represents the final six-DOF trajectory file obtained after post-processing. All files are archived and stored uniformly, preserving the original collected data completely while outputting optimized trajectories adapted to robot operation, and providing complete data support for subsequent coordinate transformation, fault verification, and data tracing.

[0042] In step S4, when it is necessary to convert the camera coordinate system trajectory into a trajectory file in the robot arm's base coordinate system, the operator issues a coordinate transformation trigger command to independently initiate the reference fixture visual recognition process, without requiring the fixture to be simultaneously displayed in the frame. The camera acquires images containing the reference fixture, and the real-time pose of the fixture is solved frame by frame using multi-marker joint PnP logic. Furthermore, a sliding window composed of multiple consecutive frames is used to conduct comprehensive stability judgment, and multiple quantitative indicators are verified simultaneously: the number of effective recognition marks within the window, the single-frame mark reprojection error, the average pixel side length of the mark imaging, the fixture translation deviation in consecutive frames, the pose deviation in consecutive frames, and the total number of abnormal outlier frames within the window. These multi-dimensional parameters jointly constrain the reference accuracy.

[0043] A sliding window mechanism is employed, and stability is determined based on one or more of the following: the number of visible markers, single-frame reprojection error, average pixel side length of markers, consecutive frame position deviation, consecutive frame pose deviation, and the number of outlier frames. When the number of valid frames or the proportion of valid frames meeting preset conditions within the window reaches a preset threshold, the locked pose of the reference fixture relative to the camera coordinate system is determined based on the average pose, weighted average pose, or robustly estimated pose of the valid frames, completing the intermediate coordinate reference calibration and providing a stable transformation reference for coordinate transformation. In one strict implementation, the locked pose is determined when all candidate frames within the sliding window meet the preset stability conditions; in another implementation, outlier frames are allowed to exist in no more than a preset number or preset proportion, and the locked pose is determined based on the remaining valid frames.

[0044] Step S5 mainly involves performing multiple coordinate transformations. See the appendix to the instruction manual. Figure 6The system comprises four core coordinate systems: the camera coordinate system C serves as the original reference for visual recognition; the Tool1 coordinate system T1 is bound to the handheld acquisition fixture, and the transformation relationship T_C_T1 between the two is obtained in real-time during the trajectory recording stage; the Tool2 reference coordinate system T2 is bound to a reference fixture rigidly fixed to the robot arm base, and T_C_T2_locked represents the stable locked pose after sliding window verification; and the robot base coordinate system B serves as the robot arm motion control reference, with T_B_T2 representing the fixed rigid body transformation relationship calibrated once after fixture installation. The entire architecture uses the Tool2 coordinate system as a bridge to map the massage trajectory from the camera coordinate system to the robot base coordinate system, thus decoupling trajectory acquisition from the coordinate reference.

[0045] Following the rigid body coordinate transformation logic, the matrix operations are performed step by step. In the embodiment where the massage application point P coincides with the origin of the handheld acquisition fixture coordinate system T1, the core transformation relationship is as follows: T_T2_T1=(T_C_T2_locked)^(-1)×T_C_T1 T_B_P=T_B_T2×T_T2_T1 It should be noted that in the above formula, the origin of the handheld acquisition tool coordinate system T1 is set at the center of the end spherical action point.

[0046] When the origin of the handheld acquisition fixture coordinate system T1 does not coincide with the massage point P, the fixed rigid body transformation T_T1_P between the handheld acquisition fixture coordinate system T1 and the massage point coordinate system P is predetermined, and the pose of the massage point relative to the robot arm base coordinate system is obtained by the following formula: T_B_P=T_B_T2×T_T2_T1×T_T1_P, where T_T1_P is the fixed rigid body transformation matrix from the handheld acquisition fixture coordinate system T1 to the massage point coordinate system P; when the origin of the handheld acquisition fixture coordinate system T1 coincides with the massage point P, T_T1_P is the unit transformation matrix.

[0047] In one embodiment, the position of the Tool2 coordinate system relative to the robot arm base coordinate system is (-93.34, 93.34 / 2, 0) mm, i.e., (-93.34, 46.67, 0) mm; if the axes of the two coordinate systems are aligned, the rotation matrix is ​​an identity matrix. If there are positional and / or attitude deviations during actual installation, the complete six-degree-of-freedom rigid body transformation relationship between the reference tooling and the robot arm base coordinate system can be determined through external calibration.

[0048] When the camera observes Tool1, T_C_T1 is obtained through multi-marker joint PnP; when tool2 is observed and locked stably as needed, T_C_T2_locked is obtained. In the current embodiment, the trajectory after T conversion is no longer stored on the disk as a Tool2 coordinate system file. Instead, the known T_B_T2 and T_C_T2_locked are used to synthesize the transformation from the camera coordinate system to the robot arm base coordinate system, and the saved camera trajectory is mapped as a whole to the robot arm base coordinate system.

[0049] This coordinate transformation process supports unlimited repeated triggering, eliminating the need for operators to re-record the teaching trajectory while holding the tool. When the robotic arm's installation position or the position of the reference tool relative to the camera changes, step S4 can be re-executed to obtain a new reference tool locking pose, and a corresponding robotic arm base coordinate system trajectory can be generated based on the same source trajectory. When the camera position changes, the rigid body transformation relationship between the original camera coordinate system and the current camera coordinate system can be determined through fixed world reference markers, external positioning devices, environmental feature matching, multiple reference toolings, or other camera repositioning methods. After updating the source trajectory coordinates according to the rigid body transformation relationship, the reference tool locking and trajectory transformation are then performed. Let T_Cnew_Cold be the rigid body transformation from the original camera coordinate system Cold to the current camera coordinate system Cnew, then the updated source trajectory pose is T_Cnew_T1 = T_Cnew_Cold × T_Cold_T1.

[0050] In step S6, the six-degree-of-freedom trajectory data, after coordinate transformation and placed in the robot arm's base coordinate system, is packaged in a standardized format. On one hand, a trajectory file is generated, which includes at least the three-dimensional spatial coordinates and rotational attitude of each trajectory point. The trajectory file or accompanying process data file may also include one or more of the following: timestamp, frame number, velocity parameters, acceleration parameters, quality indicators, coordinate system identifier, and action segment identifier. On the other hand, an accompanying trajectory quality summary file is automatically generated, which fully records all quality indicators, judgment thresholds, marker recognition status, and coordinate system information of the entire process of trajectory acquisition, filtering, and coordinate transformation, facilitating staff to verify trajectory accuracy and conduct source tracing and debugging.

[0051] The packaged trajectory execution file and quality summary file are then output to the storage path of the massage robot control system. The robot control system can perform one or more of the following processes based on the trajectory file: unit conversion, coordinate axis mapping, posture expression transformation, time parameter generation, velocity planning, inverse kinematics solution, safety verification, and control command generation. The generated control commands are then sent to the robotic arm execution mechanism.

[0052] This application provides a method for generating the trajectory of a massage robot based on non-coplanar coded visual markers. It constructs a rigid body using multiple non-coplanar stereo visual markers, solves the six-degree-of-freedom pose using a multi-marker joint PnP algorithm, and verifies the geometric consistency of the rigid body using reprojection error. This method helps reduce the risk of recognition interruption and pose instability caused by single-plane markers under large-angle rotation, partial occlusion, or complex viewing angles, improving the continuity and stability of pose estimation. It abandons the robotic arm dragging teaching mode, using a handheld fixture to simulate massage movements, which is simple to operate and highly comfortable for the human body. Simultaneously, it decouples trajectory acquisition from the reference coordinate system, eliminating the need for the reference fixture to continuously appear in the camera's field of view during recording, reducing limitations on site layout and shooting angles, and adapting to various human body massage teaching scenarios. Furthermore, the camera coordinate system source trajectory is only acquired and saved once, and subsequent reference fixture locking and coordinate transformation can be triggered multiple times. The same teaching trajectory can be repeatedly converted to adapt to different installation conditions and different robotic arm devices, eliminating the need for repeated handheld teaching and significantly improving the efficiency of robot debugging and motion replication.

[0053] Based on the same inventive concept, this application also provides a massage robot trajectory generation device based on non-coplanar coded visual markers. Since the principle of the device in this application is similar to the massage robot trajectory generation method based on non-coplanar coded visual markers described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0054] As per the instruction manual Figure 7 As shown in the figure, this application embodiment also provides a massage robot trajectory generation device based on non-coplanar coded visual markers, the device comprising: Configuration module 701 is used to configure the handheld acquisition fixture and the reference fixture; wherein, the handheld acquisition fixture and the reference fixture are provided with multiple non-coplanar coded visual markers with unique identifiers and corner features; The acquisition module 702 is used to simulate massage movements by hand-held acquisition fixtures. The camera detects visual markers frame by frame and distinguishes the fixture type. The multi-marker joint PnP algorithm is used to solve the six-degree-of-freedom pose of the point of action in the camera coordinate system to complete the acquisition of the massage trajectory. The storage module 703 is used to perform online quality screening of the recorded massage trajectory according to the trajectory processing threshold, and to store the selected valid trajectory as the source trajectory of the camera coordinate system once. The locking module 704 is used to detect the reference reference fixture after triggering the coordinate transformation command and perform stability verification by combining the sliding window and multiple indicators; after the indicators meet the standards, the locked pose of the reference reference fixture relative to the camera coordinate system is determined. The transformation module 705 is used to perform trajectory coordinate system transformation based on the source trajectory, locked pose, and fixed transformation relationship between the reference reference tooling and the robot arm base coordinate system; The generation module 706 is used to encapsulate the six-degree-of-freedom trajectory in the converted robot arm base coordinate system and generate a trajectory file containing position and attitude information for the robot control system to read.

[0055] The massage robot trajectory generation device based on non-coplanar coded visual markers described in this application uses a combination of two sets of non-coplanar stereo vision tooling and machine vision algorithms to achieve handheld teaching of massage trajectories, multiple reuses of a single data acquisition, reducing teaching difficulty and equipment cost, and improving trajectory accuracy, stability and robot reproduction effect.

[0056] Based on the same concept of the present invention, as shown in the appendix to the specification. Figure 8 As shown in the figure, an embodiment of this application provides the structure of an electronic device 800, which includes: at least one processor 801, at least one network interface 804 or other user interface 803, a memory 805, and at least one communication bus 802. The communication bus 802 is used to realize the connection and communication between these components. The electronic device 800 may optionally include a user interface 803, including a display (e.g., touch screen, LCD, CRT, holographic imaging, or projector, etc.), a keyboard, or a clicking device (e.g., mouse, trackball, touchpad, or touch screen, etc.).

[0057] Memory 805 may include read-only memory and random access memory, and provides instructions and data to processor 801. A portion of memory 805 may also include non-volatile random access memory (NVRAM).

[0058] In some implementations, memory 805 stores executable modules or data structures, or subsets thereof, or extended sets thereof: The 8051 operating system contains various system programs used to implement various basic business functions and handle hardware-based tasks. Application module 8052 contains various applications, such as launcher, media player, and browser, to implement various application functions.

[0059] In this embodiment, by calling the program or instructions stored in the memory 805, the processor 801 is used to execute steps such as a massage robot trajectory generation method based on non-coplanar coded visual markers.

[0060] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs steps such as those in a method for generating the trajectory of a massage robot based on non-coplanar coded visual markers.

[0061] Specifically, the storage medium can be a general-purpose storage medium, such as a portable disk or hard drive. When the computer program on the storage medium is run, it can reduce the difficulty of teaching and the cost of equipment, and improve the accuracy, stability and robot reproduction effect of the trajectory.

[0062] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, and the indirect coupling or communication connection of the apparatus or units may be electrical, mechanical, or other forms.

[0063] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0064] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0065] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0066] Finally, it should be noted that the above embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for generating the trajectory of a massage robot based on non-coplanar coded visual markers, characterized in that, The method includes the following steps: Configure a handheld acquisition fixture and a reference fixture; the handheld acquisition fixture and the reference fixture are equipped with multiple non-coplanar coded visual markers with unique identifiers and corner features; Massage motions are simulated by handheld acquisition fixtures. The camera detects visual markers frame by frame and distinguishes the fixture type. The six-degree-of-freedom pose of the point of action in the camera coordinate system is solved by multi-marker joint PnP algorithm to complete the acquisition of massage trajectory. The recorded massage trajectories are screened online according to the trajectory processing threshold, and the effective trajectories obtained from the screening are stored as the source trajectories of the camera coordinate system once. After triggering the coordinate transformation command, the reference reference fixture is detected and stability is verified by combining the sliding window and multiple indicators; once the indicators meet the standards, the locked pose of the reference reference fixture relative to the camera coordinate system is determined. Based on the source trajectory, locked pose, and the fixed transformation relationship between the reference reference tooling and the robot arm's base coordinate system, trajectory coordinate system transformation is performed; The six-degree-of-freedom trajectory in the converted robotic arm base coordinate system is encapsulated to generate a trajectory file containing position and attitude information for the robot control system to read.

2. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 1, characterized in that, The method of solving the six-DOF pose of the point of action in the camera coordinate system using the multi-marker joint PnP algorithm includes the following steps: After the camera captures a single frame image, it identifies all visible coded visual markers in the image one by one, and extracts the two-dimensional pixel corner point and unique ID corresponding to each coded visual marker. Retrieve the pre-stored 3D rigid body model of the handheld acquisition tool based on the unique ID, and match the coordinates of the 3D spatial corner points of the corresponding coded visual markers; The matching relationship is established for the two-dimensional and three-dimensional corner points corresponding to all visible coded visual marks belonging to the handheld acquisition tool, and all matching point pairs are integrated and uniformly substituted into the PnP algorithm to obtain the six-degree-of-freedom overall pose of the entire handheld acquisition tool relative to the camera coordinate system in one go.

3. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 2, characterized in that, The origin of the coordinate system of the handheld acquisition tool is set at the center of the spherical action point at its end. The pose solution directly represents the spatial position and attitude of the action point in the camera coordinate system.

4. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 2, characterized in that, The process of online quality screening of recorded massage trajectories according to trajectory processing thresholds, and storing the selected valid trajectories as source trajectories in a single pass using the camera coordinate system, includes the following steps: After the massage trajectory is collected, online quality screening is performed based on pre-configured multiple trajectory processing thresholds. Among them, the pre-configured trajectory processing thresholds include visual marker imaging threshold, inter-frame motion smoothness threshold, and effective marker recognition threshold. The selected valid trajectories are defined as source trajectories in the camera coordinate system and stored once. The stored source trajectories are then subjected to multi-level post-processing, which includes one or more of outlier removal, moving average filtering, distance downsampling, and equal arc length resampling. At least one of the following is set for the source trajectory: version identifier, time identifier, task unique identifier, overwrite confirmation mechanism, permission verification mechanism, or verification information, in order to manage the repeated writing, overwriting, or modification of the source trajectory.

5. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 4, characterized in that, in, A sliding window mechanism is used to determine stability based on one or more of the following: number of visible markers, single-frame reprojection error, average pixel side length of markers, position deviation of consecutive frames, pose deviation of consecutive frames, and number of outliers. When the number of valid frames or the proportion of valid frames that meet the preset conditions within the window reaches the preset threshold, the locked pose of the reference fixture relative to the camera coordinate system is determined based on the average pose, weighted average pose, or robust estimated pose of the valid frames.

6. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 5, characterized in that, The trajectory coordinate system transformation is performed using the following formula: T_T2_T1=(T_C_T2_locked)^(-1)×T_C_T1 T_B_P=T_B_T2×T_T2_T1×T_T1_P Wherein, T_C_T1 is the six-DOF rigid body transformation matrix of the handheld acquisition fixture relative to the camera coordinate system C, T_C_T2_locked is the rigid body transformation matrix of the reference fixture relative to the camera coordinate system C, T_T2_T1 is the rigid body transformation matrix of the handheld acquisition fixture relative to the reference fixture coordinate system, T_B_T2 is the fixed rigid body transformation matrix of the reference fixture relative to the robotic arm base coordinate system B, T_B_P is the final six-DOF rigid body transformation matrix of the massage application point relative to the robotic arm base coordinate system B, and T_T1_P is the fixed rigid body transformation matrix from the handheld acquisition fixture coordinate system T1 to the massage application point coordinate system P; when the origin of the handheld acquisition fixture coordinate system T1 coincides with the massage application point P, T_T1_P is the unit transformation matrix.

7. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 6, characterized in that, The reference reference fixture and the robot arm base are rigidly fixed by means of screw connection, positioning pin, rigid clamp, or integrated machining; if there are installation position and / or attitude deviations, the complete six-degree-of-freedom rigid body transformation relationship between the reference reference fixture and the robot arm base coordinate system is determined by external calibration.

8. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 1, characterized in that, The method further includes the following steps: Read the historically saved source trajectory file and restore the coordinate system definition, pose expression method, time information and quality information of the source trajectory; after completing the locking of the reference datum fixture and the determination of coordinate relationships, perform coordinate transformation on the historical source trajectory again.

9. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 1, characterized in that, The method further includes the following steps: When the camera position changes during the trajectory acquisition and coordinate transformation stages, the rigid body transformation relationship between the original camera coordinate system and the current camera coordinate system is determined by using fixed world reference markers, external positioning devices, environmental feature matching, multiple reference reference fixtures or their combinations; after updating the source trajectory based on the rigid body transformation relationship, reference reference fixture locking and trajectory transformation are then performed.

10. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 1, characterized in that, The method further includes the following steps: The corresponding robot adaptation parameter set is called for different robotic arm devices; the robot adaptation parameter set includes one or more of the following: coordinate axis mapping relationship, length unit conversion relationship, posture representation method, rigid body transformation relationship from reference tooling to robotic arm base coordinate system, trajectory file format rules and kinematic constraints.

11. The method for generating the trajectory of a massage robot based on non-coplanar coded visual markers according to claim 1, characterized in that, The robot control system performs one or more of the following processes based on the trajectory file: unit conversion, coordinate axis mapping, posture expression transformation, time parameter generation, velocity planning, inverse kinematics solution, safety verification, and control command generation.

12. A massage robot trajectory generation device based on non-coplanar coded visual markers, characterized in that, The device includes: The configuration module is used to configure the handheld acquisition fixture and the reference fixture; the handheld acquisition fixture and the reference fixture are equipped with multiple non-coplanar coded visual markers with unique identifiers and corner features; The acquisition module is used to simulate massage movements using a handheld acquisition fixture. The camera detects visual markers frame by frame and distinguishes the fixture type. The multi-marker joint PnP algorithm is used to solve the six-degree-of-freedom pose of the point of action in the camera coordinate system to complete the acquisition of the massage trajectory. The storage module is used to perform online quality screening of the recorded massage trajectories according to the trajectory processing threshold, and to store the selected valid trajectories as the source trajectories of the camera coordinate system once. The locking module is used to detect the reference reference fixture after triggering the coordinate transformation command and perform stability verification by combining the sliding window and multiple indicators; after the indicators meet the standards, the locked pose of the reference reference fixture relative to the camera coordinate system is determined. The transformation module is used to perform trajectory coordinate system transformation based on the source trajectory, locked pose, and fixed transformation relationship between the reference reference tooling and the robot arm base coordinate system; The generation module is used to encapsulate the six-degree-of-freedom trajectory in the converted robot arm base coordinate system, generating a trajectory file containing position and attitude information for the robot control system to read.

13. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of a massage robot trajectory generation method based on non-coplanar coded visual markers as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of a massage robot trajectory generation method based on non-coplanar coded visual markers as described in any one of claims 1 to 11.

Images