Interactive projection guidance control method, interactive mechanical arm and humanoid robot
By integrating projection, laser guidance, and lighting modules at the end of the robot's robotic arm, and combining them with voice interaction and environmental perception, an integrated closed-loop service is achieved, enabling the robot to move autonomously, project precisely, and provide intelligent supplementary lighting in dynamic office environments. This solves the problems of limited display methods, insufficient explanation capabilities, and high system integration difficulty in existing technologies.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- FUJIAN SECURE MEDICAL TECH
- Filing Date
- 2026-05-20
- Publication Date
- 2026-06-26
AI Technical Summary
In existing technologies, robots in office and other settings have limited demonstration methods, insufficient explanation capabilities, scattered functional modules, weak adaptability to dynamic environments, and high difficulty in system integration, lacking a unified and collaborative control system technology solution.
By integrating projection, laser guidance, and lighting modules at the end of the robotic arm, a unified collaborative control framework is established. Combined with voice interaction and environmental perception, an integrated closed-loop service is achieved, enabling autonomous movement, precise projection, laser explanation, and intelligent supplementary lighting.
It solves the problems of coordinate system drift, end effector jitter, content-level laser mapping and lighting environment coordination in mobile projection scenarios, and realizes the robot's autonomous movement, precise projection, laser explanation and safe recharging in dynamic office environment.
Smart Images
Figure CN122275003A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robotics, and in particular to an interactive projection guidance control method, an interactive robotic arm, and a humanoid robot. Background Technology
[0002] With the rapid development of humanoid robots and wheeled service robots, their application in offices, reception areas, conference rooms, exhibition halls, and high-end companion services is gradually increasing. Existing technologies typically disclose one or more of the following: a mobile chassis, voice interaction, projection device, laser pointing device, lighting device, or robotic arm actuator. However, most of these technologies only address single-function problems, as detailed below: (1) Limited display methods. Most robots rely on chest or head screens for information display, which has a limited display area and cannot meet the needs of high-rise offices, meeting spaces and reporting scenarios for large-size, close-to-wall, and flexible display.
[0003] (2) Insufficient explanation ability. Even if existing robots can broadcast voice messages, they usually lack mechanical instruction capabilities that are linked to the content being displayed.
[0004] (3) Dispersed functional modules. Projection, laser and lighting are usually handled by different devices, lacking unified integration and control on the robot end effector, which is not conducive to forming stable and coordinated display actions.
[0005] (4) Weak adaptability to dynamic environments. The walls, screens, desks, chairs and personnel positions in the office environment change frequently, making it difficult for fixed projection equipment or fixed display schemes to adapt flexibly.
[0006] (5) The system integration is challenging. Integrating a mobile chassis, environmental perception, robotic arm motion control, projection display, laser guidance, lighting adjustment, and voice interaction into a single robot platform and achieving stable, continuous, and safe collaborative operation involves strong coupling issues across technical domains, for which existing technologies lack mature solutions. For example, there are coupling errors between the mobile chassis and the robotic arm, and difficulties in spatial mapping between projection and laser.
[0007] Therefore, existing technologies lack a systematic technical solution that can unify and coordinate the control of voice interaction, knowledge processing, projection display, laser explanation, lighting, mobile navigation, and automatic recharging on a four-wheeled intelligent chassis humanoid robot platform. Summary of the Invention
[0008] To address the aforementioned problems in the prior art, this application provides an interactive projection guidance control method, an interactive robotic arm, and a humanoid robot, which realizes an integrated closed-loop service for autonomous robot movement, precise projection, laser explanation, intelligent supplementary lighting, and safe recharging in a dynamic office environment.
[0009] To achieve the above objectives, the technical solution adopted in this application is as follows: In a first aspect, this application provides an interactive projection guidance control method, comprising: Responding to user voice commands, it generates content to be displayed and corresponding explanation scripts; The four-wheeled intelligent chassis is controlled to move to the target display area and the target projection plane is identified. Based on the spatial parameters of the target projection plane, the positioning status of the four-wheeled intelligent chassis, and the current joint status of the interactive robotic arm, the desired pose of the end effector of the interactive robotic arm is jointly solved. The interactive robotic arm is controlled to move the projection module to the desired pose, and after performing geometric correction on the content to be displayed, it is projected onto the target projection plane; According to the explanation script, the laser guidance module is controlled to dynamically indicate or track the target area in the content to be displayed, and the lighting module is controlled to perform supplementary lighting according to the ambient illuminance, task mode and user position. After the task is completed, control the humanoid robot to perform standby, follow, or automatically return to charging.
[0010] The beneficial effects of this application are as follows: by integrating the projection module, laser guidance module and lighting module into the end effector of the robotic arm and establishing a unified collaborative control framework for the three, the problems of coordinate system drift, end effector jitter, content-level laser mapping, lighting environment coordination and multiple safety constraints near people in mobile projection scenarios are solved, realizing an integrated closed-loop service for autonomous movement, precise projection, laser explanation, intelligent supplementary lighting and safe recharging of the robot in dynamic office environments.
[0011] Optionally, the desired pose satisfies: ; in, This is the homogeneous transformation matrix from the robot base coordinate system to the robotic arm end effector coordinate system; Let be the homogeneous transformation matrix from the robot's base coordinate system to the target projection plane reference coordinate system; This is the desired pose transformation matrix required by the projection module relative to the target projection plane.
[0012] Optionally, the geometric correction employs a homography transformation matrix H, such that the original image coordinates (u,v) and the target projection plane coordinates (x,y) satisfy: ; In the formula, λ is the scaling factor.
[0013] Optionally, the dynamic tracking of the laser guidance module adopts the following objective function: ; In the formula, p t p is the location of the target indicator point. l This indicates the current location of the laser beam. C is the joint velocity vector of the robotic arm; obs The cost of obstacle avoidance; α, β, and γ are weighting coefficients. The output brightness of the lighting module satisfies: L out =k1L env +k2D -1 +k3P mode ; In the formula, L out L is the output brightness of the lighting. env D represents ambient illumination; P represents the distance between the robot and the user or target area; D represents the distance between the robot and the user or target area. mode k1, k2, and k3 are task mode coefficients; k1, k2, and k3 are weight coefficients.
[0014] Optionally, it also includes: Define a joint error state vector e, and establish the relationship between the error state vector e and the chassis pose x. b A linearized observation model of the robotic arm joint angle q is used to obtain the optical-mechanical joint observation model: ; ; In the formula, This is the deformation displacement vector of the projected image on the target projection plane; This is the laser impact point deviation vector; O represents the illuminance deviation in the illuminated area; O is the opto-mechanical coupled observation matrix, which is obtained through offline calibration and estimates unmodeled disturbances in real time and provides feedback correction. The semantic focus in the explanation script will be transformed into a discrete-time spacetime heat source point set {(x k ,y k ,t k ,q k Let the attention heat conduction field T(x,y,t) on the target projection plane satisfy the heat dissipation conduction equation: T(x,y,t+Δt)=T(x,y,t)+κ 2 T(x,y,t)Δt+ G(x x k ,y y k ,t t k ); In the formula, q kρ is the heat source intensity; x and y are the target projection plane coordinates; Δt is the discrete time step; κ is the thermal diffusivity; G is the two-dimensional Gaussian kernel; and p is the laser impact point. l (t) is constrained to be a local maximum point of the current time T(x,y,t).
[0015] Optionally, it also includes: Define the joint state X of the chassis pose and the robotic arm end effector pose, and the projection optical axis direction error δξ. proj and laser pointing error δξ laser Let the elements of the Lie algebra in the tangent space of the joint state Lie group be: X=(X b ,X e )∈SE(2)×SE(3); δξ proj ,δξ laser ∈se(2)×se(3); In the formula, SE(2) is a two-dimensional special Euclidean group, representing the pose of the four-wheeled intelligent chassis in the plane; SE(3) is a three-dimensional special Euclidean group, representing the pose of the robotic arm end effector in space; X b X is the homogeneous transformation matrix of the four-wheel intelligent chassis; e SE(2) and SE(3) are the homogeneous transformation matrix of the end effector of the robotic arm, respectively. SE(2) and SE(3) are the Lie algebras corresponding to SE(2) and SE(3), respectively. Their elements are six-dimensional or three-dimensional vectors used to represent infinitesimal pose perturbations. To minimize ||δξ proj || 2 +‖δξ laser || 2 To achieve the objective, solve for the joint optimal control law and directly output the chassis velocity vector v. b And the joint torque vector τ of the robotic arm: min vb,τ =‖δξ proj || 2 +‖δξ laser || 2 ; Based on the joint error state output by the optical-mechanical joint observation model and the personnel location information output by the environmental perception module, a safety potential field function is constructed. When the predicted potential field value at a future time is lower than the safety threshold, a deceleration, trajectory replanning, or laser shutdown command is executed in advance.
[0016] Optionally, it also includes: In each control cycle, the optimal control problem is solved within the predicted time domain T to obtain the cooperative control command. The optimal control problem is: ; u joint =[vb ,τ,H(t),p l ,l] T ; In the formula, J total The total cost function; u joint is the joint control vector; is the joint error state; Q and R are positive definite weight matrices; Φ(τ) is the safety potential field value; t is the current time; H(t) is the time-varying homography transformation matrix; p l Let l be the coordinates of the laser impact point, and l be the illumination parameter vector. The control commands are output to the chassis drive, robotic arm joints, projection correction unit, laser tracking servo, and lighting driver.
[0017] Secondly, this application provides an interactive robotic arm for performing corresponding operations in the control method of the first aspect, including: Robotic arm body; The composite actuator is located at the end of the robotic arm body. The composite actuator integrates a projection module, a laser guidance module and an illumination module. The three work together through a unified installation structure and a unified control interface. The optical axis of the laser guidance module is coaxial or parallel to the optical axis of the projection module. The illumination module is located at the peripheral lighting position of the projection module. The posture detection unit is used to provide real-time feedback on the joint angles and end-effector pose information of the robotic arm body.
[0018] Optionally, the composite actuator adopts an integrated end-effector structure, which includes a projection heat dissipation channel, a laser collimation mounting position, an illumination drive circuit board, and an end-effector mounting bracket. The attitude detection unit includes one or more of a rotary encoder, an inertial measurement unit, a visual calibration device, or an end-effector identification mark.
[0019] Thirdly, this application provides a humanoid robot, including the interactive robotic arm of the second aspect.
[0020] The technical effects of the interactive robotic arm provided in the second aspect and the humanoid robot provided in the third aspect are described in the relevant description of the interactive projection guidance control method provided in the first aspect. Attached Figure Description
[0021] Figure 1 This is a flowchart illustrating the interactive projection guidance control method according to an embodiment of this application; Figure 2 This is a schematic diagram of the frame of an interactive robotic arm according to an embodiment of this application. Detailed Implementation
[0022] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be understood more clearly and thoroughly, and that the scope of this application can be fully conveyed to those skilled in the art.
[0023] The embodiments of this application are applied to application scenarios that require humanoid robots to perform mobile information display and explanation, such as office companionship, presentation, reception and explanation, or high-end service interaction. In the prior art, mobile robot display and explanation solutions have problems such as single display methods, lack of linkage in explanation, scattered functional modules, weak dynamic adaptability, and high difficulty in system combination. In particular, when integrating projection, laser, and lighting into the end effector of a robotic arm, it faces combination difficulties such as coordinate system drift, end effector jitter sensitivity, lack of content-level mapping, insufficient lighting coordination, and multiple safety constraints near the human body.
[0024] Therefore, in various embodiments of this application, in response to user voice commands, the system generates content to be displayed and a corresponding explanation script; controls the four-wheeled intelligent chassis to move to the target display area and identifies the target projection plane; based on the spatial parameters of the target projection plane, the positioning state of the four-wheeled intelligent chassis, and the current joint state of the interactive robotic arm, jointly solves the desired pose of the end effector of the interactive robotic arm; controls the interactive robotic arm to move the projection module to the desired pose, and projects the content to be displayed onto the target projection plane after performing geometric correction; according to the explanation script, controls the laser guidance module to dynamically indicate or track the target area in the content to be displayed, and controls the lighting module to perform supplementary lighting according to the ambient illuminance, task mode, and user position; after the task is completed, controls the humanoid robot to perform standby, following, or automatic recharging. Thus, an integrated closed-loop service is achieved for the robot's autonomous movement, precise projection, laser explanation, intelligent supplementary lighting, and safe recharging in a dynamic office environment.
[0025] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.
[0026] like Figure 1 As shown, this application provides an interactive projection guidance control method, including: Step S1: In response to the user's voice command, generate the content to be displayed and the corresponding explanation script.
[0027] In one example, a user issues commands to the voice interaction module, such as "Please show today's operating data and explain the key points," "Please project the meeting minutes onto the wall," or "Please accompany me to the meeting area and introduce the project progress," etc. After recognizing the speech, the voice interaction module sends the semantic results to the central control unit. The semantic processing submodule in the central control unit extracts the display topic, display location, explanation method, and movement requirements, while the knowledge retrieval and generation submodule generates the content to be displayed and the explanation script based on the local knowledge base, enterprise database, or reasoning service.
[0028] The content to be displayed may include text summaries, organizational charts, business process diagrams, schedules, bar charts, line charts, project status diagrams, Q&A results, or meeting minutes.
[0029] Step S2: Control the four-wheel intelligent chassis to move to the target display area and identify the target projection plane. Based on the spatial parameters of the target projection plane, the positioning status of the four-wheel intelligent chassis, and the current joint status of the interactive robotic arm, jointly solve the desired pose of the end effector of the interactive robotic arm.
[0030] The task planning submodule in the central control unit generates chassis path planning and robotic arm motion planning based on the office map and target location. After the four-wheeled intelligent chassis moves to the appropriate position, the environmental perception module identifies the outer surface of the wall, whiteboard, screen, or projection screen, and selects a suitable target projection plane. The central control unit then drives the robotic arm to lift the end effector compartment to the target posture and simultaneously controls the projection module, laser guidance module, and lighting module to perform the display task.
[0031] Step S3: Control the interactive robotic arm to move the projection module to the desired pose, and after performing geometric correction on the content to be displayed, project it onto the target projection plane.
[0032] In this embodiment, the robot base coordinate system is preset to B, the target projection plane reference coordinate system to P, and the robotic arm end effector coordinate system to E. The environment perception module detects the target projection plane and extracts its plane normal vector, boundary corner points, and spatial position. The central control unit converts this result into the target projection plane reference coordinate system P, and simultaneously combines it with the robot base coordinate system B and the robotic arm end effector coordinate system E to establish the pose relationship. The desired pose satisfies: ; in, This is the homogeneous transformation matrix from the robot base coordinate system to the robotic arm end effector coordinate system; The homogeneous transformation matrix from the robot's base coordinate system to the target projection plane reference coordinate system is obtained from the environmental perception results; This is the desired pose transformation matrix required by the projection module relative to the target projection plane, which is determined based on the projection size, projection distance, and projection angle requirements.
[0033] In this process, after the interactive robotic arm reaches the target pose, the projection module outputs the image to be displayed. Since there is usually a certain angle between the projection optical axis and the target projection plane, a homography mapping relationship between the image coordinates and the projection plane coordinates is established to compensate for the perspective error between the projection module and the target plane. Therefore, geometric correction is required. In this embodiment, the geometric correction uses a homography transformation matrix H to ensure that the original image coordinates (u,v) and the target projection plane coordinates (x,y) satisfy the following: ; In the formula, λ is the scaling factor.
[0034] Step S4: According to the explanation script, control the laser guidance module to dynamically indicate or track the target area in the content to be displayed, and control the lighting module to perform supplementary lighting according to the ambient illuminance, task mode and user position.
[0035] After the projected content is displayed, the central control unit controls the laser guidance module to move the laser point to a key area in the projected content, such as chart peaks, process nodes, title areas, or important text paragraphs, according to the explanation script. To ensure that the laser point stably tracks the target area in the projected content, the dynamic tracking of the laser guidance module in this embodiment adopts the following objective function: ; In the formula, p t p is the location of the target indicator point. l This indicates the current location of the laser beam. C is the joint velocity vector of the robotic arm; obs The cost of obstacle avoidance is represented by α, β, and γ, which are weighting coefficients.
[0036] In the objective function described above, the first term is used to minimize the laser landing point error, the second term is used to suppress the jitter caused by the excessively fast movement of the robotic arm, and the third term is used to improve obstacle avoidance safety.
[0037] Therefore, the laser guidance module in this embodiment does not work independently, but relies on the coordinates of the projected image, the posture of the robotic arm end effector, and the explanatory script to jointly determine the laser trajectory.
[0038] It should be noted that when a user is detected approaching, there is a risk of mirror reflection, or a face appears in the laser path, the central control unit can turn off the laser output or use non-laser prompts such as projected highlight boxes or arrow animations.
[0039] In this embodiment, the lighting module is used not only for supplemental lighting in low-light environments but also for providing auxiliary lighting in robot companionship, desktop services, and nighttime guidance. To balance projection visibility, user comfort, and energy consumption control, the output brightness of the lighting module in this embodiment meets the following requirements: L out =k1L env +k2D -1 +k3P mode ; In the formula, L out L is the output brightness of the lighting. env D represents ambient illumination; P represents the distance between the robot and the user or target area; D represents the distance between the robot and the user or target area. mode k1, k2, and k3 are task mode coefficients; k1, k2, and k3 are weight coefficients.
[0040] Therefore, the lighting module and projection module in this embodiment are not independent of each other. If the lighting is too strong, it may reduce the visibility of the projection; if the lighting is too weak, it will be detrimental to office reading and walking. Therefore, the central control unit establishes a coupled control relationship between the lighting direction and the projection direction to maintain a balance between supplementary lighting and display.
[0041] Step S5: After the task is completed, control the humanoid robot to perform standby, follow, or automatic recharging.
[0042] When the central control unit detects that the battery pack's charge level is below a preset threshold, or when the current task has ended and the system enters an idle state, it controls the four-wheeled intelligent chassis to navigate to the wireless charging base for energy coupling charging via the wireless charging module. The interactive robotic arm automatically retracts to a safe position during recharging to reduce its footprint and minimize the risk of collision.
[0043] This embodiment integrates automatic recharging into a unified task loop, rather than treating it as an isolated power management function. This allows the system to remain in an office environment for extended periods, providing continuous service and automatic recovery, further enhancing its feasibility for engineering implementation.
[0044] In one embodiment, it also includes: Step S61: Define the joint error state vector e, and establish the relationship between the error state vector e and the chassis pose x. b A linearized observation model of the robotic arm joint angle q is used to obtain the optical-mechanical joint observation model: ; ; In the formula, This is the deformation displacement vector of the projected image on the target projection plane; This is the laser impact point deviation vector; O represents the illuminance deviation in the illuminated area; O is the opto-mechanical coupled observation matrix, which is obtained through offline calibration and estimates unmodeled disturbances in real time and provides feedback corrections.
[0045] The central control unit acquires the position and speed of the four-wheel intelligent chassis and the joint angles and angular velocities of the interactive robotic arm in real time. Through the offline calibrated optical-mechanical coupling observation matrix, it estimates the joint error state to estimate unmodeled disturbances such as uneven ground, flexible vibration of the robotic arm, and wind disturbance in real time. The estimated value of the joint error state is fed back to the subsequent control module for correction of projection correction, laser tracking and lighting output.
[0046] Step S62: Transform the semantic focus in the explanation script into a discrete-time heat source point set {(x k ,y k ,t k ,q k Let the attention heat conduction field T(x,y,t) on the target projection plane satisfy the heat dissipation conduction equation: T(x,y,t+Δt)=T(x,y,t)+κ 2 T(x,y,t)Δt+ G(x x k ,y y k ,t t k ); In the formula, q k ρ is the heat source intensity; x and y are the target projection plane coordinates; Δt is the discrete time step; κ is the thermal diffusivity; G is the two-dimensional Gaussian kernel; and p is the laser impact point. l (t) is constrained to be a local maximum point of the current time T(x,y,t).
[0047] The central control unit transforms the semantic focus in the explanation script into a discrete spatiotemporal heat source point set, enabling the laser guidance module to automatically follow the hot spot of the explanation's attention, forming a smooth and continuous explanation guidance trajectory, thus avoiding the frequent start-stop and shaking of the robotic arm caused by traditional discrete jump points.
[0048] Therefore, the central control unit performs optical-mechanical joint observation and attention heat conduction control to solve the problems of projection deformation, laser offset and lighting deviation caused by chassis movement and robotic arm movement.
[0049] Step S63: Define the joint state X of the chassis pose and the robotic arm end effector pose, and the projection optical axis direction error δξ. proj and laser pointing error δξ laser Let the elements of the Lie algebra in the tangent space of the joint state Lie group be: X=(Xb ,X e )∈SE(2)×SE(3); δξ proj ,δξ laser ∈se(2)×se(3); In the formula, SE(2) is a two-dimensional special Euclidean group, representing the pose of the four-wheeled intelligent chassis in the plane; SE(3) is a three-dimensional special Euclidean group, representing the pose of the robotic arm end effector in space; X b X is the homogeneous transformation matrix of the four-wheel intelligent chassis; e SE(2) and SE(3) are the homogeneous transformation matrix of the end effector of the robotic arm. SE(2) and SE(3) are the Lie algebras corresponding to SE(2) and SE(3), respectively. Their elements are six-dimensional or three-dimensional vectors used to represent infinitesimal pose perturbations.
[0050] Step S64: Minimize ||δξ proj || 2 +‖δξ laser || 2 To achieve the objective, solve for the joint optimal control law and directly output the chassis velocity vector v. b And the joint torque vector τ of the robotic arm: min vb,τ =‖δξ proj || 2 +‖δξ laser || 2 .
[0051] Among them, the optimal control law simultaneously optimizes the chassis motion and the robotic arm motion within the same Lie group framework, avoiding the cumulative error caused by multi-coordinate system transformation in traditional hierarchical control.
[0052] Step S65: Based on the joint error state output by the optical-mechanical joint observation model and the personnel location information output by the environmental perception module, construct a safety potential field function. When the predicted potential field value at a future time is lower than the safety threshold, execute the deceleration, trajectory replanning, or laser shutdown command in advance.
[0053] The safety potential field function includes the potential of the robot arm-person distance, the potential of the laser-eye area, the potential of the chassis-obstacles, and the potential of the projected light path-sensitive area. This predictive safety mechanism can proactively intervene before a collision or injury occurs, significantly improving the safety of robots in office companionship scenarios.
[0054] Therefore, the central control unit performs joint motion planning and predictive safety constraints based on Lie group Lie algebras to eliminate coordinate system transformation errors between the chassis and the robotic arm, and to avoid various safety risks in near-human scenarios in advance.
[0055] Step S66: In each control cycle, solve the optimal control problem within the predicted time domain T to obtain the cooperative control command. The optimal control problem is: ; u joint =[v b ,τ,H(t),p l ,l] T ; In the formula, J total The total cost function; u joint is the joint control vector; is the joint error state; Q and R are positive definite weight matrices; Φ(τ) is the safety potential field value; t is the current time; H(t) is the time-varying homography transformation matrix; p l Let l be the coordinates of the laser impact point, and l be the illumination parameter vector.
[0056] Among them, the central control unit adopts the Model Predictive Control (MPC) framework, which integrates optical-mechanical observation, attention heat conduction, Lie group motion planning and safety potential field constraints into the same optimization problem, so as to achieve synchronous optimal coordination of all control modules.
[0057] Step S67: Output control commands to the chassis drive, robotic arm joint, projection correction unit, laser tracking servo and lighting driver.
[0058] The central control unit employs sequential quadratic programming or alternating direction multiplier method to solve the optimization problem in real time, with a solution cycle controlled within 50ms, meeting the robot's real-time control requirements. The first control quantity of the obtained control command is output to the chassis drive, robotic arm joints, projection correction unit, laser tracking servo, and lighting driver, enabling projection correction, laser tracking, lighting adjustment, robotic arm movement, and chassis movement to respond simultaneously within the same optimization cycle, achieving synchronous and optimal coordination among multiple modules.
[0059] This embodiment overcomes the shortcomings of error accumulation and response lag in traditional serial control by using a unified MPC framework. When chassis movement causes the projection optical axis to shift, the MPC can simultaneously adjust the robot arm's posture and projection correction parameters in the prediction time domain to compensate. When laser tracking requires a rapid response to the narration script, the MPC will plan the robot arm's motion trajectory in advance to reduce lag. When the safety potential field predicts a potential collision, the MPC actively adjusts the control variables to avoid the risk. This deeply coupled control architecture enables the robot to stably and smoothly complete continuous tasks such as movement, projection, narration, lighting, and safe recharging in a dynamically changing office environment.
[0060] In one embodiment, such as Figure 2As shown, this application also provides an interactive robotic arm for performing corresponding operations in the control methods described above, including: Robotic arm body 1; The composite actuator 2 is located at the end of the robotic arm body 1. The composite actuator 2 integrates a projection module, a laser guidance module and an illumination module. The three work together through a unified installation structure and a unified control interface. The optical axis of the laser guidance module is set coaxially or parallel to the optical axis of the projection module, so that the laser landing point corresponds to the specified area in the projection content. The illumination module is set at the peripheral supplementary lighting position of the projection module. The posture detection unit 3 is used to provide real-time feedback on the joint angles and end-effector pose information of the robotic arm body 1.
[0061] The robotic arm body of the interactive robotic arm includes a robotic arm base, an upper arm assembly connected to the robotic arm base, a forearm assembly connected to the upper arm assembly, a wrist assembly located at the end of the forearm assembly, and a series of joints with 5 to 7 degrees of freedom, including shoulder joints, elbow joints, wrist joints, etc.
[0062] In one embodiment, when the interactive robotic arm is configured as a dual-arm system, the left robotic arm can perform the projection function, while the right robotic arm can perform the laser explanation or lighting function; alternatively, the left and right robotic arms can collaboratively display two projection areas respectively. This dual-arm extension scheme further demonstrates the system scalability and modular synergy of this embodiment.
[0063] In this embodiment, the composite actuator adopts an integrated end effector compartment structure, which internally includes a projection heat dissipation channel, a laser collimation mounting position, an illumination drive circuit board, and an end effector mounting bracket. Specifically, the projection module is one of a micro DLP projector, an LCOS projector, a laser projector, or other micro image projection devices. The laser guidance module is a low-power visible light laser, whose emission direction is coaxial with, parallel to, or at a preset angle to the optical axis of the projection module. The illumination module is one or more of an LED downlight, a spotlight unit, a dimmable illumination unit, or an adjustable color temperature illumination unit. The environmental perception module includes one or more of an RGB camera, a depth camera, a LiDAR, a millimeter-wave radar, an ultrasonic sensor, an inertial sensor, and an ambient illuminance sensor. The attitude detection unit includes one or more of a rotary encoder, an inertial measurement unit, a visual calibration device, or an end effector identification mark.
[0064] In one embodiment, this application also provides a humanoid robot, including the interactive robotic arm of the above embodiments. The interactive robotic arm of this embodiment further includes an environmental perception module.
[0065] The humanoid robot in this embodiment also includes a four-wheeled intelligent chassis, a humanoid upper body structure mounted on the four-wheeled intelligent chassis, a voice interaction module mounted on the humanoid upper body structure, at least one interactive robotic arm, and a central control unit.
[0066] The four-wheeled intelligent chassis includes a navigation and positioning module, a wireless charging module, a battery pack, and a chassis controller, providing environmental mapping, autonomous navigation, autonomous positioning, dynamic obstacle avoidance, and automatic recharging capabilities. The humanoid upper body structure houses the voice interaction module and the interactive robotic arm. The voice interaction module performs voice acquisition, wake-up, recognition, semantic parsing, and broadcasting. The interactive robotic arm's environmental perception module identifies the target projection plane, personnel position, obstacle position, and ambient lighting conditions. The central control unit performs semantic processing, knowledge generation, task planning, chassis control, robotic arm motion control, projection correction, laser guidance, safety control, and energy management. Thus, the four-wheeled intelligent chassis provides the mobility foundation, the environmental perception module identifies the target plane and dynamic obstacles, the interactive robotic arm performs precise end-effector positioning, the interactive robotic arm's projection module displays information, the interactive robotic arm's laser guidance module provides focused explanations, the interactive robotic arm's lighting module provides supplementary and accompanying lighting, the wireless charging module provides power, and the central control unit executes the control methods described in the above embodiments to achieve coordinated control of all the modules, including but not limited to: A unified coordinate mapping relationship is established between the projection module, the laser guidance module, and the target projection plane, so that the laser landing point corresponds to a specified area in the projected content. Based on the positioning results of the four-wheel intelligent chassis, the joint state of the interactive robotic arm, and the spatial parameters of the target projection plane, the target pose and projection correction parameters of the projection module are jointly solved. The display content and explanation script are generated according to the user's voice commands, and the projection module and laser guidance module are controlled synchronously according to the explanation script. When the environmental perception module detects the presence of a face area, eye area, specular reflection area, or high-risk area on the laser path, the central control unit turns off the laser guidance module or switches the laser prompt to a projection highlight prompt. The central control unit switches between demonstration mode, reporting mode, reception mode, companion mode, and night lighting mode according to user commands or scene status.
[0067] In a specific application scenario, within a private office, users can directly communicate with a humanoid robot using natural language. After a user submits a query, summary, or presentation request, the humanoid robot retrieves and organizes information using its knowledge processing module. It then moves to the vicinity of the target wall, where its interactive robotic arm raises its end effector to project the content onto the wall. A laser guidance module explains key points, and a lighting module provides supplementary illumination to the target area.
[0068] This embodiment only shows a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the interactive robotic arm or humanoid robot to which the present application solution is applied. A specific interactive robotic arm or humanoid robot may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements, such as power supplies, input / output interfaces, etc. Furthermore, the central control unit of this embodiment can operate on an operating system stored in memory, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.
[0069] In addition, the specific descriptions of the technical effects and steps of the interactive robotic arm and humanoid robot in the above embodiments are all based on the relevant descriptions of the embodiments in which the interactive projection guidance control method is described.
[0070] Since the systems / devices described in the above embodiments of this application are systems / devices used to implement the methods of the above embodiments of this application, those skilled in the art can understand the specific structure and modifications of the system / devices based on the methods described in the above embodiments of this application, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of this application fall within the scope of protection of this application.
[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.
[0073] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The words "a" or "an" preceding a component do not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.
[0074] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0075] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include the preferred embodiments as well as all changes and modifications falling within the scope of this application.
[0076] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, then this application should also include these modifications and variations.
Claims
1. An interactive projection guidance control method, characterized in that, include: Responding to user voice commands, it generates content to be displayed and corresponding explanation scripts; The four-wheeled intelligent chassis is controlled to move to the target display area and the target projection plane is identified. Based on the spatial parameters of the target projection plane, the positioning status of the four-wheeled intelligent chassis, and the current joint status of the interactive robotic arm, the desired pose of the end effector of the interactive robotic arm is jointly solved. The interactive robotic arm is controlled to move the projection module to the desired pose, and after performing geometric correction on the content to be displayed, it is projected onto the target projection plane; According to the explanation script, the laser guidance module is controlled to dynamically indicate or track the target area in the content to be displayed, and the lighting module is controlled to perform supplementary lighting according to the ambient illuminance, task mode and user position. After the task is completed, control the humanoid robot to perform standby, follow, or automatically return to charging.
2. The method according to claim 1, characterized in that, The desired pose satisfies: ； in, This is the homogeneous transformation matrix from the robot base coordinate system to the robotic arm end effector coordinate system; Let be the homogeneous transformation matrix from the robot's base coordinate system to the target projection plane reference coordinate system; This is the desired pose transformation matrix required by the projection module relative to the target projection plane.
3. The method according to claim 1, characterized in that, The geometric correction employs a homography transformation matrix H to ensure that the original image coordinates (u,v) and the target projection plane coordinates (x,y) satisfy the following: ； In the formula, λ is the scaling factor.
4. The method according to claim 3, characterized in that, The dynamic tracking of the laser guidance module adopts the following objective function: ； In the formula, p t p is the location of the target indicator point. l This indicates the current location of the laser beam. C is the joint velocity vector of the robotic arm; obs The cost of obstacle avoidance; α, β, and γ are weighting coefficients. The output brightness of the lighting module satisfies: L out =k1L env +k2D -1 +k3P mode ; In the formula, L out L is the output brightness of the lighting. env D represents ambient illumination; P represents the distance between the robot and the user or target area; D represents the distance between the robot and the user or target area. mode k1, k2, and k3 are task mode coefficients; k1, k2, and k3 are weight coefficients.
5. The method according to any one of claims 1 to 4, characterized in that, Also includes: Define a joint error state vector e, and establish the relationship between the error state vector e and the chassis pose x. b A linearized observation model of the robotic arm joint angle q is used to obtain a combined optical-mechanical observation model: ； ； In the formula, This is the deformation displacement vector of the projected image on the target projection plane; This is the laser impact point deviation vector; O represents the illuminance deviation in the illuminated area; O is the opto-mechanical coupled observation matrix, which is obtained through offline calibration and estimates unmodeled disturbances in real time and provides feedback correction. The semantic focus in the explanation script will be transformed into a discrete-time spacetime heat source point set {(x k ,y k ,t k ,q k Let the attention heat conduction field T(x,y,t) on the target projection plane satisfy the heat dissipation conduction equation: T(x,y,t+Δt)=T(x,y,t)+κ 2 T(x,y,t)Δt+ G(x x k ,y y k ,t t k ); In the formula, q k ρ is the heat source intensity; x and y are the target projection plane coordinates; Δt is the discrete time step; κ is the thermal diffusivity; G is the two-dimensional Gaussian kernel; and p is the laser impact point. l (t) is constrained to be a local maximum point of the current time T(x,y,t).
6. The method according to claim 5, characterized in that, Also includes: Define the joint state X of the chassis pose and the robotic arm end effector pose, and the projection optical axis direction error δξ. proj and laser pointing error δξ laser Let the elements of the Lie algebra in the tangent space of the joint state Lie group be: X=(X b ,X e )∈SE(2)×SE(3); right proj ,dx laser ∈se(2)×se(3); In the formula, SE(2) is a two-dimensional special Euclidean group, representing the pose of the four-wheeled intelligent chassis in the plane; SE(3) is a three-dimensional special Euclidean group, representing the pose of the robotic arm end effector in space; X b X is the homogeneous transformation matrix of the four-wheel intelligent chassis; e SE(2) and SE(3) are the homogeneous transformation matrix of the end effector of the robotic arm, respectively. SE(2) and SE(3) are the Lie algebras corresponding to SE(2) and SE(3), respectively. Their elements are six-dimensional or three-dimensional vectors used to represent infinitesimal pose perturbations. To minimize ||δξ proj || 2 +‖δξ laser || 2 To achieve the objective, solve for the joint optimal control law and directly output the chassis velocity vector v. b And the joint torque vector τ of the robotic arm: minutes vb,τ =‖δξ proj ‖ 2 +‖δξ laser ‖ 2 ; Based on the joint error state output by the optical-mechanical joint observation model and the personnel location information output by the environmental perception module, a safety potential field function is constructed. When the predicted potential field value at a future time is lower than the safety threshold, a deceleration, trajectory replanning, or laser shutdown command is executed in advance.
7. The method according to claim 6, characterized in that, Also includes: In each control cycle, the optimal control problem is solved within the predicted time domain T to obtain the cooperative control command. The optimal control problem is: ； u joint =[v b ,τ,H(t),p l ,l] T ; In the formula, J total The total cost function; u joint is the joint control vector; is the joint error state; Q and R are positive definite weight matrices; Φ(τ) is the safety potential field value; t is the current time; H(t) is the time-varying homography transformation matrix; p l Let l be the coordinates of the laser impact point, and l be the illumination parameter vector. The control commands are output to the chassis drive, robotic arm joints, projection correction unit, laser tracking servo, and lighting driver.
8. An interactive robotic arm for performing the corresponding operation in the control method of any one of claims 1 to 7, characterized in that, include: Robotic arm body; The composite actuator is located at the end of the robotic arm body. The composite actuator integrates a projection module, a laser guidance module and an illumination module. The three work together through a unified installation structure and a unified control interface. The optical axis of the laser guidance module is coaxial or parallel to the optical axis of the projection module. The illumination module is located at the peripheral lighting position of the projection module. The posture detection unit is used to provide real-time feedback on the joint angles and end-effector pose information of the robotic arm body.
9. The interactive robotic arm according to claim 8, characterized in that, The composite actuator adopts an integrated end-effector structure, which includes a projection heat dissipation channel, a laser collimation mounting position, an illumination drive circuit board, and an end-effector mounting bracket. The attitude detection unit includes one or more of a rotary encoder, an inertial measurement unit, a visual calibration device, or an end-effector identification mark.
10. A humanoid robot, characterized in that, Including the interactive robotic arm as described in any one of claims 7 to 9.