Robots and binocular perception method for adversarial scenarios

Abstract

The application discloses a robot and a binocular perception method for a confrontation scene. The method comprises the following steps: at least one storage medium stores instructions, and the instructions enable the robot to acquire a binocular image pair synchronously collected by a binocular camera; based on a preset confrontation semantic key point set, two-dimensional coordinates of each key point of the opponent in a left eye image and a right eye image are detected from the binocular image pair; according to the two-dimensional coordinates of each key point in the left eye image and the right eye image, binocular triangulation processing is performed to obtain three-dimensional coordinates of each key point; a relative coordinate system is constructed based on reference points and orientation information of both sides, and three-dimensional relative poses and distances between key positions between both sides are calculated according to the three-dimensional coordinates of each key point of both sides in the relative coordinate system; and a structured perception result is determined according to the three-dimensional relative poses and the distances between key positions. The application solves the technical problems of poor robustness and low perception accuracy of related methods in binocular perception.

Description

Technical Field

[0001] This application relates to the field of robot vision perception technology, and more specifically, to a robot and a binocular perception method for adversarial scenarios. Background Technology

[0002] In high-speed, dynamic combat scenarios such as robot boxing, accurate situational awareness is fundamental to ensuring safety and improving control performance. Related technologies often employ monocular vision, general human pose estimation, or rely on external optical motion capture, multi-camera arrays, and static calibration environments, primarily outputting a single target's two-dimensional skeleton or coarse-grained motion categories. However, these methods exhibit poor robustness when faced with frequent limb occlusion and drastic changes in perspective caused by camera movement.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a robot and a binocular perception method for adversarial scenarios, which at least solves the technical problems of poor robustness and low perception accuracy of related methods when performing binocular perception.

[0005] According to one aspect of the embodiments of this application, a robot is provided, comprising: a torso; two upper limbs and two lower limbs respectively connected to the torso; a binocular camera mounted on the head; at least one processor; and at least one storage medium, the at least one storage medium storing instructions, which, when executed by the at least one processor, cause the robot to: acquire a pair of binocular images synchronously acquired by the binocular camera; detect the two-dimensional coordinates of each key point of the opponent in the left and right binocular images based on a preset set of adversarial semantic key points, wherein the preset set of adversarial semantic key points includes target attack parts and target defense parts; perform binocular triangulation processing based on the two-dimensional coordinates of each key point in the left and right binocular images to obtain the three-dimensional coordinates of each key point; construct a relative coordinate system based on reference points and orientation information of both parties, and calculate the three-dimensional relative pose and distance between key parts of both parties based on the three-dimensional coordinates of each key point of both parties in the relative coordinate system; and determine the structured perception result based on the three-dimensional relative pose and distance between key parts.

[0006] Optionally, the preset set of adversarial semantic key points includes: attack side key points and defense side key points. Attack side key points include fist key points, and defense side key points include: head target area key points and torso target area key points.

[0007] Optionally, key points on the attacking side also include the outer edge of the fist; key points on the defensive side also include key points in the forearm defensive area, key points in the supporting foot area, and key points in the vulnerable areas of the fuselage.

[0008] Optionally, a relative coordinate system is constructed based on the reference points and orientation information of both parties, including: determining the origin reference point based on the torso reference points of both parties, and determining the coordinate axis direction based on the orientation vectors of both parties, wherein the orientation vector is determined by the direction of the line connecting the preset key point pairs; and constructing a relative coordinate system based on the origin reference point and the coordinate axis direction.

[0009] Optionally, when the instructions are executed by at least one processor, the robot also constructs a two-way interaction graph using semantic key points of both parties as nodes and the three-dimensional distance between key points on the attacking side and key points in the target area on the defending side as edge features, wherein the two-way interaction graph is used to calculate the distance between key parts.

[0010] Optionally, when the instructions are executed by at least one processor, the robot also: while detecting the two-dimensional coordinates of each key point of the other party, simultaneously outputs the visibility confidence of each key point; when the visibility confidence of a key point is lower than a preset visibility threshold, it corrects the three-dimensional coordinates of the key point.

[0011] Optionally, when executed by at least one processor, the instructions also cause the robot to: acquire the average reprojection error and pose confidence; publish the structured perception results normally in response to the average reprojection error being less than or equal to a preset error threshold and the pose confidence being greater than or equal to a preset confidence threshold; and publish the structured perception results in a downgraded manner in response to the average reprojection error being greater than the preset error threshold and / or the pose confidence being less than the preset confidence threshold.

[0012] Optionally, the structured perception results also include the safety distance calculated based on the relative translation vectors of the two reference points.

[0013] Optionally, the three-dimensional coordinates of the user's key points are obtained from the robot's sensor data and kinematic model, while the two-dimensional coordinates of the opponent's key points are detected from the binocular image pair.

[0014] According to another aspect of the embodiments of this application, a binocular perception method for adversarial scenarios is also provided. The method includes: acquiring synchronously acquired binocular image pairs; detecting the two-dimensional coordinates of each key point of both sides in the left and right eye images from the binocular image pairs based on a preset set of adversarial semantic key points, wherein the preset set of adversarial semantic key points includes target attack parts and target defense parts; performing binocular triangulation processing based on the two-dimensional coordinates of each key point in the left and right eye images to obtain the three-dimensional coordinates of each key point; constructing a relative coordinate system based on the reference points and orientation information of both sides, and calculating the three-dimensional relative pose and distance between key parts of both sides based on the three-dimensional coordinates of each key point of both sides in the relative coordinate system; and determining the structured perception result based on the three-dimensional relative pose and distance between key parts.

[0015] According to another aspect of the embodiments of this application, a binocular perception device for adversarial scenarios is also provided. The device includes: an acquisition module for acquiring synchronously acquired binocular image pairs; a detection module for detecting the two-dimensional coordinates of each key point of both sides in the left and right eye images from the binocular image pairs based on a preset set of adversarial semantic key points, wherein the preset set of adversarial semantic key points includes target attack parts and target defense parts; a triangulation module for performing binocular triangulation processing based on the two-dimensional coordinates of each key point in the left and right eye images to obtain the three-dimensional coordinates of each key point; a calculation module for constructing a relative coordinate system based on reference points and orientation information of both sides, and calculating the three-dimensional relative pose and distance between key parts of both sides based on the three-dimensional coordinates of each key point of both sides in the relative coordinate system; and a determination module for determining the structured perception result based on the three-dimensional relative pose and distance between key parts.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0019] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0020] In this embodiment, the robot's binocular cameras synchronously acquire binocular image pairs. Using a preset set of adversarial semantic key points, the robot detects the two-dimensional coordinates of the opponent's key points by identifying the target's attack and defense locations. After binocular triangulation, three-dimensional coordinates are obtained. A relative coordinate system is constructed based on the reference points and orientation information of both sides to calculate the three-dimensional relative pose and the distance between key points. Finally, the structured perception result is determined. This enables the robot to acquire the three-dimensional spatial position and relative pose of the opponent's key points in real time and accurately, relying solely on head-based binocular vision without external motion capture equipment. This provides precise situational awareness data for punching decisions, defensive choices, distance control, and safety constraints, improving the robot's robustness and perception accuracy in boxing scenarios. It also solves the technical problems of poor robustness and low perception accuracy in related methods when performing binocular perception. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a schematic diagram of an optional binocular perception system for adversarial scenarios according to an embodiment of this application;

[0023] Figure 2 This is a schematic diagram of an optional robot according to an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of an optional binocular perception method for adversarial scenarios according to an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of another optional binocular perception method for adversarial scenarios according to an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of another optional binocular perception method for adversarial scenarios according to an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of another optional binocular perception method for adversarial scenarios according to an embodiment of this application;

[0028] Figure 7 This is a schematic diagram of another optional binocular perception method for adversarial scenarios according to an embodiment of this application;

[0029] Figure 8 This is a structural block diagram of an optional binocular sensing device for adversarial scenarios according to an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] In robot combat scenarios, perception systems need to acquire real-time 3D spatial information of key opponent parts to support subsequent motion decisions and safety control. Related keypoint pose estimation schemes typically process a single target, outputting the target's 2D keypoint coordinates or 3D skeleton pose in the camera coordinate system. The skeleton definitions used in these schemes often follow standard skeleton models from general human pose estimation, with keypoint sets corresponding to human anatomical joints, but lacking semantic information such as attack areas, defense areas, or vulnerable areas required for specific combat scenarios.

[0033] When adversaries are in close-range, rapid interaction, frequent limb crossings and occlusions significantly increase the difficulty of detecting key points. Furthermore, if the visual sensor's perspective changes with the robot's movement, the stability of single-frame detection results often fails to meet the control system's requirement for continuous input. More critically, existing solutions only output independent pose information for each target, without establishing spatial relationships between them. This means that quantitative indicators such as relative distance, relative orientation, and reachable attack range required by the control system must be derived secondary by downstream modules from their individual coordinate data. This process lacks a unified coordinate reference and semantic constraints, easily introducing accumulated errors and failing to directly reflect the offensive and defensive spatial situation of the adversaries at the perception level. Consequently, it hinders the realization of punching decisions, defensive choices, and real-time security protection capabilities.

[0034] According to an embodiment of this application, a method embodiment for a robot is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0035] Figure 1 This is a schematic diagram of an optional binocular perception system for adversarial scenarios according to an embodiment of this application, such as... Figure 1 As shown, the system consists of a binocular input synchronization module, a keypoint detection and triangulation module, an interaction relationship and relative pose module, and a deployment and quality gating module. These modules are connected via data streams. The binocular input synchronization module receives left and right eye red-green-blue (RGB) images, timestamps, and calibration parameters. It generates synchronized image pairs through distortion correction, frame alignment, and synchronization quality checks, and then sends them to the keypoint detection and triangulation module. The keypoint detection and triangulation module performs two-dimensional keypoint detection on the synchronized image pairs based on the calibration parameters and calculates the three-dimensional coordinates of each semantic keypoint using a corrected binocular triangulation formula. Subsequently, it transmits the two-dimensional / three-dimensional keypoint data to the interaction relationship and relative pose module.

[0036] The interaction relationship and relative pose module receives the 3D keypoints and adversarial coordinate system definitions from both sides. During the training phase, it optimizes the network parameters using reprojection loss. During the deployment phase, it calculates the relative poses and interaction distances of both sides and updates the interaction relationship features. The relative poses of both sides include relative translation and relative rotation. Finally, the data enters the deployment and quality gating module. This module comprehensively evaluates the 3D keypoints, relative poses, reprojection errors, and pose confidence. Based on preset thresholds, it determines whether to publish normally or degrade the data, outputting structured perception results with timestamps and confidence levels for use by downstream control systems and to support post-competition data feedback and review.

[0037] Figure 2 This is a schematic diagram of an optional robot according to an embodiment of this application, such as... Figure 2 As shown. The robot 20 includes: a torso 21; two upper limbs 22 and two lower limbs 23 respectively connected to the torso; a binocular camera 24 mounted on the head; at least one processor 25; and at least one storage medium 26, the at least one storage medium storing instructions that, when executed by the at least one processor, cause the robot to:

[0038] Step S201: Obtain the binocular image pairs synchronously acquired by the binocular cameras;

[0039] Step S202: Based on a preset set of adversarial semantic key points, detect the two-dimensional coordinates of each key point of the opponent in the left and right eye images from the binocular image pair. The preset set of adversarial semantic key points includes the target attack parts and the target defense parts.

[0040] Step S203: Based on the two-dimensional coordinates of each key point in the left and right eye images, perform binocular triangulation to obtain the three-dimensional coordinates of each key point.

[0041] Step S204: Construct a relative coordinate system based on the reference points and orientation information of both parties. Under the relative coordinate system, calculate the three-dimensional relative pose between the two parties and the distance between key parts based on the three-dimensional coordinates of each key point of both parties.

[0042] Step S205: Determine the structured perception result based on the three-dimensional relative pose and the distance between key parts.

[0043] The robot described in this application embodiment can be a physical entity comprising a torso, upper limbs, lower limbs, and a perception and computing module, used to perform boxing combat tasks. The torso is the main structural part of the robot, used to connect the upper and lower limbs and serving as the main support for the body's center of gravity. The upper limbs are limb structures connected to both sides of the torso, including the shoulder, elbow, wrist, and fist, used to perform punching attacks. The lower limbs are limb structures connected to the bottom of the torso, including the hip, knee, ankle, and supporting foot, used to support the body and for movement and dodging.

[0044] In an optional embodiment of this application, the robot head is equipped with a binocular camera, including a left-eye camera and a right-eye camera lens. The baseline distance between the left-eye camera and the right-eye camera is calibrated at the factory, and the calibration parameters include the focal length f, the binocular baseline length B, and the principal point coordinates. , During deployment and operation, the binocular camera synchronously acquires images from the left eye at a preset frame rate. And right eye image Together, they form a stereo image pair. In an optional embodiment, the synchronization quality of the acquired stereo image pair is ensured by setting the time difference between the left and right eye acquisition times to not exceed a preset synchronization tolerance. If a frame exceeds this tolerance, it is marked as an unusable frame and skipped.

[0045] In one optional embodiment, the binocular cameras are mounted directly in front of the robot's head, with the left and right eye axes approximately parallel. This mounting method ensures that the opponent is within the overlapping area of ​​the binocular field of view at most engagement distances. If the robot's head experiences pitch or yaw motion, the camera extrinsic parameters can be updated in real time using readings from the robot's own joint encoders and the head-camera rigidity transformation relationship determined during calibration, but the camera intrinsic parameters and baseline length remain unchanged during deployment.

[0046] After acquiring the original images, distortion correction is performed on the left and right eye images separately to remove the effects of radial and tangential lens distortion, resulting in corrected synchronized image pairs. The distortion coefficients used for distortion correction also come from the offline calibration stage and remain frozen during deployment. After correction, the epipolar lines of the left and right eye images are aligned, allowing subsequent triangulation steps to search for corresponding points only within the same row.

[0047] The preset adversarial semantic keypoint set is defined offline, and its content is determined based on the robot's physical structure and adversarial rules. Unlike the general human skeleton model, this keypoint set not only includes basic skeletal keypoints, such as head, neck, shoulder, elbow, wrist, hip, knee, and ankle, but also attack-side and defense-side keypoints specific to the adversarial scenario.

[0048] After acquiring synchronized image pairs, the images are processed using a binocular adversarial keypoint perception network. This network comprises two functional components: an encoder and a detection head. The encoder extracts multi-scale visual features from the left and right eye images, respectively. and In one embodiment of this application, the left and right eye encoders share network weights. For example, the encoder can be a convolutional neural network (CNN). The shared weights design ensures that the left and right eye features reside in the same representation space, which is beneficial for matching the corresponding key points of the left and right eyes in the subsequent triangulation step. In another optional embodiment, the left and right eye encoders can also use independent weights, trained separately and then frozen.

[0049] The detection head receives multi-scale features output by the encoder and predicts the two-dimensional coordinates of each keypoint defined in the preset adversarial semantic keypoint set in the left eye image. , ) and the corresponding x-coordinate in the right eye image It also outputs the visibility confidence score of each key point. For example, the detection head can be a multi-layer perceptron (MLP).

[0050] In an optional embodiment of this application, the detection head can output keypoint coordinates using heatmap regression. Specifically, a Gaussian heatmap is predicted on the feature map for each keypoint, and the peak position of the heatmap is the sub-pixel coordinate of that keypoint. The response intensity of the heatmap peak can be directly used as an initial estimate of the visibility confidence of that keypoint.

[0051] After obtaining the two-dimensional coordinates of each keypoint in the left and right eye images, the corresponding points in the left and right eyes are corrected using camera calibration parameters through binocular triangulation to recover their three-dimensional coordinates in the left eye camera coordinate system. The triangulation calculation relationship is as follows:

[0052]

[0053]

[0054]

[0055] Where (X, Y, Z) are the three-dimensional coordinates of the key point in the left eye camera coordinate system. B is the focal length, and B is the binocular baseline length. , () represents the two-dimensional coordinates of the key point in the left eye image. Let x be the x-coordinate of the key point in the right eye image. , () are the coordinates of the main point. This refers to the horizontal parallax of corresponding points in the left and right eyes. The larger the parallax, the closer the key point is to the camera.

[0056] Triangulation requires that the left and right eye images have undergone epipolar correction, meaning that corresponding points are on the same line. The distortion correction and frame alignment described above already satisfy this condition.

[0057] The reference points for both sides are specific key points used to define the origin of the relative coordinate system, typically the torso center or hip center of both robots. Orientation information can be a direction vector representing the robot's body orientation, determined by the direction of the line connecting predefined key point pairs. The relative coordinate system, also known as the adversarial coordinate system, can be a three-dimensional coordinate system constructed based on the reference points and orientation information of both sides, used to uniformly describe the relative pose relationship between the two sides.

[0058] Three-dimensional relative pose refers to the spatial position and rotation angle of one robot relative to another, including relative translation and relative rotation. Distance between key components refers to the three-dimensional Euclidean distance between specific key points of the two robots, used to assess attack reachability and defense effectiveness.

[0059] After the relative coordinate system is established, the three-dimensional relative pose between the two parties is calculated.

[0060] In an optional embodiment, the relative translation between the two parties It can be defined as the vector difference between the opponent's reference point and one's own reference point, and the calculation formula is:

[0061]

[0062] in, The three-dimensional coordinates of the user's torso reference point in the left eye camera coordinate system. The three-dimensional coordinates of the opponent's torso reference point in the same coordinate system. Let be the relative translation vector between the two parties.

[0063] Relative rotation between the two sides Describes the rotational relationship between the opponent's torso orientation and the player's torso orientation. The calculation method is: based on the player's orientation vector... and the other party's orientation vector The result is obtained by calculating the orientation alignment rotation matrix. This calculation requires eliminating ambiguity regarding the rotational degrees of freedom about the orientation axis. In one embodiment of this application, the direction of gravity is introduced as an auxiliary constraint. The orientation vector is projected onto a horizontal plane, and then the yaw angle difference between the two is calculated, thereby simplifying the three-degree-of-freedom rotation into a relative rotation representation with the yaw angle difference as the main component. This simplification is reasonable in a boxing match scenario because both fighters' torsos remain in a nearly upright posture for most of the confrontation, with relatively small pitch and roll components.

[0064] In an optional embodiment, the three-dimensional relative pose can be obtained through post-processing calculation, direct regression by a neural network branch, or a strategy of fusing pre- and post-processing.

[0065] When two orientation vectors are approximately parallel or approximately opposite, i.e., the other is facing directly towards the other or with its back to the other, the yaw angle difference is close to zero or 180 degrees, and no numerical degradation will occur. If both are in a fallen state, causing the gravity projection assumption to fail, the frame can be marked as an abnormal frame by the confidence level in the subsequent gating steps.

[0066] After the relative coordinate system is constructed, the distance between key points on both sides is calculated. For each pair of key points (i, j) defined in the interaction diagram of both sides, the three-dimensional Euclidean distance is calculated, where i belongs to the key point on the attacking side and j belongs to the key point in the target area on the defending side. The calculation formula is as follows:

[0067]

[0068] in, Let i be the three-dimensional coordinates of our i-th key point. Let J be the three-dimensional coordinates of the j-th key point of the other party. This represents the three-dimensional distance between the keypoints.

[0069] Structured perception results refer to standardized data formats that include timestamps, confidence levels, three-dimensional relative poses, distances between key components, and release status identifiers, which are used by downstream control modules.

[0070] The calculated 3D relative pose and distances between key components are combined with reprojection error and pose confidence. A gating logic is then used to determine if safety and quality thresholds are met. If met, a structured perception result containing the aforementioned information and status identifiers is generated for use by downstream punching decisions, defensive selections, or training evaluation modules. If not met, a downgraded or conservatively corrected result is output.

[0071] Based on the above steps S201 to S205, this embodiment of the application synchronously acquires binocular image pairs using the robot's binocular camera, detects the two-dimensional coordinates of the opponent's key points using the target attack parts and target defense parts contained in the preset adversarial semantic key point set, obtains the three-dimensional coordinates through binocular triangulation processing, and constructs a relative coordinate system based on the reference points and orientation information of both parties to calculate the three-dimensional relative pose and the distance between key parts, and finally determines the structured perception result. This enables the real-time and accurate acquisition of the three-dimensional spatial position and relative pose relationship of the opponent's key parts under the condition of relying only on the head's binocular vision and without external motion capture equipment. This provides accurate situational awareness data for punching decisions, defense selection, distance control and safety constraints, improves the robustness and perception accuracy of the robot in boxing confrontation scenarios, and solves the technical problems of poor robustness and low perception accuracy of related methods when performing binocular perception.

[0072] Optionally, the preset set of adversarial semantic key points includes: attack side key points and defense side key points. Attack side key points include fist key points, and defense side key points include: head target area key points and torso target area key points.

[0073] Attacking-side keypoints can be key points specifically used to represent the intent to launch an attack or to execute a striking action from a pre-defined set of adversarial semantic keypoints. Attacking-side keypoints include fist keypoints, which can be specific geometric centers or edges pointing to the center of the fist, the leading edge of the knuckle shell, or the end of the wrist—points used for actual contact with the opponent. These could be the center of the left fist, the center of the right fist, etc. The selection of these points allows subsequent calculations to characterize the spatial position and direction of the attack.

[0074] Defensive keypoints can be specific points from a pre-defined set of adversarial semantic keypoints used to characterize the protection of vital points or for defensive blocking. Defensive keypoints include head target region keypoints and torso target region keypoints. The head target region can be the geometric feature points from the defensive keypoints used to define the effective striking area or head protection zone of the head, typically forming the boundary of a sphere, ellipsoid, capsule, or grid region; for example, it can be characterized by the center point and radius of a sphere or ellipsoid. The torso target region can be the geometric feature points from the defensive keypoints used to define the effective striking area or torso protection zone of the torso, typically forming a rectangle, plane, or grid region; for example, it can be characterized by the center point and dimensions of a capsule or bounding box. The boundary descriptions of these regions are loaded into the perception module during deployment as part of the semantic configuration.

[0075] For example, during the system initialization or configuration phase, a dedicated keypoint index table is defined and loaded based on the competition rules, robot mechanical structure, and safety level. This action first divides the keypoints into two main categories: "attack-side keypoints" responsible for offense and "defense-side keypoints" responsible for defense. The "attack-side keypoints" are further refined into "punch keypoints" for locating striking points, and the "defense-side keypoints" are further refined into "head target area keypoints" and "torso target area keypoints" for protecting vital organs. This establishes a semantic identification system closely bound to the logic of boxing tactics, providing clear output targets for the subsequent keypoint detection network.

[0076] Based on the above optional embodiments, this application embodiment defines a preset set of adversarial semantic key points that includes key points on the attacking side and key points on the defensive side. This enables the general limb skeleton recognition to be transformed into offensive and defensive situational awareness with clear tactical significance. This allows the perception system to directly output information on the fist position, which is crucial for punching decisions, and the head and torso protection areas, which are crucial for safety control. Thus, without additional labeling, it can automatically distinguish the spatial state of the offensive and defensive ends, improving the pertinence, interpretability, and direct usability of the perception results in boxing combat scenarios.

[0077] Optionally, key points on the attacking side also include the outer edge of the fist; key points on the defensive side also include key points in the forearm defensive area, key points in the supporting foot area, and key points in the vulnerable areas of the fuselage.

[0078] The key points on the attacking side also include the outer edge of the fist. The outer edge of the fist is defined as the farthest point of the fist shell along the direction of the punch.

[0079] Key defensive points also include key points for the forearm defensive area, key points for the supporting legs area, and key points for vulnerable areas of the fuselage. Key points for the forearm defensive area define the spatial boundary points of the area where the forearm is used to block opponent attacks. Key points for the supporting legs area define the spatial boundary points of the area where the legs are in contact with the ground or supporting the center of gravity. Key points for vulnerable areas of the fuselage define the vulnerable or non-combat-essential protected areas within the robot's structure.

[0080] For example, in the definition and configuration stage of the preset adversarial semantic key point set, the system further refines the granularity of the key points. On the basis of the original fist key points, "fist outer edge point" is added to mark the extreme position of the attack boundary. At the same time, "forearm defense area key point" is added to the defensive side key points to quantify the effectiveness of blocking, "supporting foot area key point" is added to monitor mechanical stability, and "fuselage vulnerable area key point" is added to identify hardware protection requirements. In this way, a key point system covering all aspects of semantics, including attack accuracy, defense effectiveness, body stability and hardware security, is constructed, providing fine-grained geometric constraints for subsequent high-precision three-dimensional pose estimation and situational awareness.

[0081] In an optional embodiment of this application, the specific composition of the preset adversarial semantic keypoint set can be reconfigured according to the robot's shell structure, adjustments to adversarial rules, or changes in security levels. For example, if the robot is equipped with knee armor, keypoints for the knee protection area can be added to the defensive side keypoints. The configurability of the keypoint set allows the perception module to adapt to fine-tuning of the keypoint definition without retraining the network; it only needs to adjust the number of output channels of the detection head and fine-tune with a small amount of incremental data.

[0082] Based on the above optional embodiments, this application embodiment expands the key points on the attack side to include the outer edge of the fist and the key points on the defense side to include the key points of the forearm defense area, the key points of the supporting foot area, and the key points of the vulnerable area of ​​the fuselage. This achieves an upgrade from single limb joint positioning to multi-dimensional semantic perception with tactical significance and physical constraints. This enables the system to accurately quantify the coverage of the attack, distinguish between effective blocking and direct impact, monitor the dynamic stability of the robot, and identify hardware protection requirements. In this way, it provides more comprehensive and detailed offensive and defensive situation information in complex and ever-changing boxing confrontation scenarios, directly improving the accuracy of the decision-making system in judging the effectiveness of the attack, the success rate of the defense, and the safety status of the robot itself.

[0083] Optionally, a relative coordinate system is constructed based on the reference points and orientation information of both parties, including: determining the origin reference point based on the torso reference points of both parties, and determining the coordinate axis direction based on the orientation vectors of both parties, wherein the orientation vector is determined by the direction of the line connecting the preset key point pairs; and constructing a relative coordinate system based on the origin reference point and the coordinate axis direction.

[0084] Reference points for both parties and These are the torso reference points for both the player's own side and the opponent's side. These reference points can be selected as the center of the torso, such as the midpoint of the line connecting the midpoints of the shoulders and the midpoints of the hips.

[0085] Orientation Vector and These represent the facing directions of both the user and the opponent, determined by the direction of the line connecting preset keypoint pairs. In one embodiment of this application, the facing vector is obtained by the cross product of the line connecting the two shoulders and the direction of the spine, pointing towards the front. If the robot's left shoulder keypoint is... The key point of the right shoulder is If the hip center key point is H and the neck key point is neck, then the orientation vector can be defined as ( The normalized unit vector of the cross product direction of (neck-H).

[0086] When determining the origin reference point using the torso reference points of both robots, semantic key points such as the torso center or hip center of both robots are selected as the spatial position reference reference. The three-dimensional coordinates of the reference points of both robots in the preset coordinate system are calculated, thereby determining the position reference origin in the relative pose calculation and providing a geometric reference for the subsequent calculation of the relative distance between the two robots.

[0087] When determining the coordinate axis direction using the orientation vectors of both parties, a unit vector representing the robot's orientation is calculated based on the geometric positional relationship of preset key points such as the robot's head, shoulders, and hips. This vector is then used to define the axis relative to the coordinate system, such as the X-axis or Z-axis, thereby establishing the orientation reference of both robots in space and solving the problem of ambiguous orientation definition under changing viewing angles in related technologies.

[0088] After determining the origin reference and coordinate axis directions, a relative coordinate system is constructed based on the origin reference and coordinate axis directions. Combining the determined position origin reference and the coordinate axis directions determined by the orientation vector, a unified adversarial coordinate system is constructed through a coordinate transformation algorithm. The coordinates of the three-dimensional key points of both robots are transformed into this relative coordinate system, so that the key points of both sides, the direction of the punch velocity, the head protection zone, and the reachable attack space are unified in a controllable three-dimensional coordinate system, realizing the standardized expression of the relative pose of both sides.

[0089] In an alternative embodiment, the relative coordinate system may also be a left-eye, right-eye, binocular midpoint, torso, foot support, or field coordinate system.

[0090] Based on the above optional embodiments, this application embodiment constructs a relative coordinate system by determining the origin reference using the torso reference points of both parties and the coordinate axis direction using the orientation vector of both parties determined by the direction of the line connecting the preset key point pairs. This unifies the key points, punch speed direction, head protection zone, and reachable attack space in a controllable three-dimensional coordinate system, solving the problem of lacking a clear definition of the three-dimensional coordinate system and geometric consistency constraints. This allows the perception results to directly output the relative pose quantities required by the control system, such as the relative translation and relative rotation of both parties. Thus, it provides conservative and usable distance estimation and offensive and defensive posture quantities in fast close-range and severely obstructed scenarios, directly serving punching decisions, defensive choices, distance control, risk assessment, and training evaluation, thereby improving the robustness and practicality of the perception results.

[0091] Optionally, when the instructions are executed by at least one processor, the robot also constructs a two-way interaction graph using semantic key points of both parties as nodes and the three-dimensional distance between key points on the attacking side and key points in the target area on the defending side as edge features, wherein the two-way interaction graph is used to calculate the distance between key parts.

[0092] A two-party interaction graph can be a graph data structure, with semantic key points of both parties as nodes of the graph and specific geometric relationships between nodes as edges of the graph.

[0093] In an optional embodiment of this application, when constructing the interaction graph between the two parties, all adversarial semantic key points of both parties are used as nodes, and the relationships between attack-side key points and defense-side target area key points are used as edges. Specifically, connecting edges are established between one's own attack-side key points and the opponent's defense-side key points, and the feature of each edge includes the corresponding three-dimensional distance. Conversely, symmetrical connections are also established between key points on the opponent's attack side and key points on the player's defense side. The topology of this graph structure remains fixed during deployment and does not change with frames.

[0094] The purpose of the interactive graph is to organize the distances between discrete key points into a structured relational representation. Downstream decision modules can directly query the distance values ​​of specific edges to determine the offensive and defensive posture, such as querying the distance from "the opponent's left fist to the effective striking area of ​​one's own head." If the distance is below the preset defensive distance, a defensive response will be triggered.

[0095] In an optional embodiment of this application, the edge set of the interaction graph is not limited to the connection between the attacking and defending sides. It can also include connecting edges between key points on the same side, such as the distance between one's left and right fists, to characterize the compactness of one's posture. Alternatively, connecting edges can be added between basic skeletal key points to assist in determining the positional relationship between the two sides. The specific composition of the edge set can be adjusted through a configuration file without retraining the detection network.

[0096] Based on the above optional embodiments, this application embodiment constructs a two-way interaction graph by using semantic key points of both parties as nodes and the three-dimensional distance between key points on the attacking side and key points in the target area on the defending side as edge features. This realizes the transformation of discrete three-dimensional key point coordinates into a topological structure with clear tactical semantics, enabling the system to explicitly model and calculate the spatial interaction relationship between the attacking and defending ends. This allows for the precise quantification of the threat level and reachability of each attacking part to each defending area, improving the pertinence and efficiency of distance calculation between key parts. It provides a structured, high semantic density perception basis for downstream decision-making, defensive choices, and security risk assessment.

[0097] Optionally, when the instructions are executed by at least one processor, the robot also: while detecting the two-dimensional coordinates of each key point of the other party, simultaneously outputs the visibility confidence of each key point; when the visibility confidence of a key point is lower than a preset visibility threshold, it corrects the three-dimensional coordinates of the key point.

[0098] While detecting the two-dimensional coordinates of each key point of the target, the detection head simultaneously outputs the visibility confidence level of each key point. Visibility confidence This characterizes whether the keypoint is occluded or outside the field of view in the current frame. When two robots are engaged in close combat, the fist frequently occludes the head and torso. In such cases, the visibility confidence of the occluded keypoint is determined. It will decrease accordingly.

[0099] In an optional embodiment, the detection head can also use a direct coordinate regression method, combined with an independent binary classification branch to output visibility determination.

[0100] For visibility confidence Keypoints with visibility below a preset visibility threshold have lower reliability in their triangulation results. In one embodiment of this application, when the visibility confidence of a keypoint is below the threshold, a conservative correction strategy is applied to the 3D coordinates of that keypoint. This involves using the 3D coordinates of the keypoint from the previous frame as the estimated value for the current frame and reducing the pose contribution weight of that point. In another optional embodiment, the conservative correction strategy can also be to increase the safety margin of the distance estimation corresponding to the keypoint, i.e., to superimpose a positive offset on the original triangulated distance result.

[0101] Based on the above optional embodiments, this application embodiment simultaneously outputs the visibility confidence of each key point when detecting the two-dimensional coordinates of each key point of the opponent, and corrects the three-dimensional coordinates of the key points when the visibility confidence of the key points is lower than the preset visibility threshold. This realizes real-time evaluation and adaptive correction of data quality during the perception process, effectively solves the occlusion problem caused by frequent limb crossing and rapid movement in robot boxing scenarios, avoids three-dimensional pose errors caused by low-quality two-dimensional coordinates, and thus improves the robustness of the binocular relative pose and adversarial key point perception method in complex dynamic environments and the reliability of the output results.

[0102] Optionally, when executed by at least one processor, the instructions also cause the robot to: acquire the average reprojection error and pose confidence; publish the structured perception results normally in response to the average reprojection error being less than or equal to a preset error threshold and the pose confidence being greater than or equal to a preset confidence threshold; and publish the structured perception results in a downgraded manner in response to the average reprojection error being greater than the preset error threshold and / or the pose confidence being less than the preset confidence threshold.

[0103] Before outputting the structured perception results, quality gating is performed. The gating decision is based on two metrics: mean reprojection error and pose confidence.

[0104] When calculating the average reprojection error, each 3D keypoint obtained through triangulation is backprojected back into the left and right eye images, respectively. The pixel deviation between the backprojected coordinates and the original detection coordinates is calculated, and the average value is taken for all visible keypoints. This error reflects the degree of consistency between the 2D detection results and the 3D geometric solution results. When the detection head outputs an incorrect left-right eye correspondence, or when the triangulation result produces a large depth error due to excessively small parallax, the reprojection error will increase accordingly.

[0105] In an optional embodiment of this application, when calculating pose confidence, the pose confidence is taken as the weighted average of the visibility confidence of all keypoints involved in the pose calculation. The weights are related to the contribution of each keypoint to the pose calculation, i.e., the weights of reference points and orientation keypoints are higher than the weights of limb end-effector keypoints. In another optional embodiment, the pose confidence can also be obtained by directly outputting a scalar branch from the network, which uses the pose estimation error as a supervision signal during training.

[0106] The preset error threshold can be used as the upper limit value to determine whether the average reprojection error is within an acceptable range; a value lower than or equal to this value indicates that the geometric reconstruction quality is acceptable. The preset confidence threshold can be used as the lower limit value to determine whether the pose confidence meets the release requirements; a value higher than or equal to this value indicates that the overall perception result is reliable.

[0107] During gating, if the average reprojection error does not exceed a preset error threshold and the pose confidence is not lower than a preset confidence threshold, the system is considered to be in a normal state and the structured perception result is published normally. Otherwise, if the average reprojection error exceeds the preset error threshold and / or the pose confidence is lower than the preset confidence threshold, the system is considered to be in an abnormal state and a downgraded publication is performed. The specific strategy for downgraded publication includes: using the result of the normally published result of the previous frame as the output of the current frame, while marking the pose confidence as a downgraded value to notify the downstream decision module that the perception result of the current frame has low reliability. After receiving the downgraded mark, the downstream module can choose to temporarily freeze the current action decision or adopt a more conservative distance estimation.

[0108] The gating determination formula is as follows:

[0109]

[0110] in, This is an indicator function that outputs 1 when the condition within the parentheses is met, indicating a normal release; otherwise, it outputs 0, indicating a downgraded release. This represents the average reprojection error, used to measure the geometric consistency of 3D reconstruction. This represents a preset error threshold, used to limit the maximum allowable reprojection error. This represents the pose confidence level, reflecting the reliability of the pose estimation under the current state. This indicates a preset confidence threshold, used to limit the minimum acceptable confidence level.

[0111] In an optional embodiment of this application, the preset error threshold and the preset confidence threshold are determined by statistical analysis on an offline validation set. Specifically, the reprojection error distribution of normal and abnormal frames is statistically analyzed on the validation set, and the quantile that makes the false alarm rate lower than a preset level is selected as the error threshold. The confidence threshold is determined in a similar manner. The above thresholds are fixed during deployment and can be adjusted through version updates.

[0112] Based on the above optional embodiments, this application embodiment obtains the average reprojection error and pose confidence, and publishes the structured perception results normally when the average reprojection error is less than or equal to a preset error threshold and the pose confidence is greater than or equal to a preset confidence threshold, and publishes the structured perception results in a downgraded manner when the average reprojection error is greater than the preset error threshold and / or the pose confidence is less than the preset confidence threshold. This achieves adaptive gating and hierarchical publishing of perception output quality, effectively distinguishing between high-reliability and low-reliability scenarios. While ensuring that the perception results directly serve control decisions under normal operating conditions, the downgraded publishing mechanism prevents safety accidents caused by erroneous information in high-risk scenarios such as occlusion or abnormal detection, thereby improving the perception robustness, decision safety, and overall system stability of the robot boxing combat system.

[0113] In an optional embodiment, this application clearly defines the boundaries of the four stages: training, inference, verification, and update, to ensure the security and maintainability of the system. During the training stage, the system uses labeled samples to perform backpropagation to update network parameters, supporting generative augmentation data and weakly supervised labels. During the inference or deployment stage, only forward computation is performed, taking real-time binocular images as input and outputting structured results and confidence levels; parameters are fixed or updated by version thresholds. During the verification stage, the system archives erroneous frames based on task execution results, security trigger events, or manual sampling, retaining qualified samples and feeding erroneous samples back to the second stage for sample selection or incremental training in the third stage. During the update stage, it is only executed in an offline environment, and a new version can only be released after verification against reprojection error, situational bias, and security thresholds, thereby ensuring the continuous optimization and stable operation of the perception system.

[0114] Optionally, the structured perception results also include the safety distance calculated based on the relative translation vectors of the two reference points.

[0115] After gating, the structured perception result output by the system includes the following fields: the 3D coordinates of each key point on both sides and the relative translation vector. Relative rotation The set of three-dimensional distances corresponding to each edge in the interaction graph { }, safe distance The visibility confidence, pose confidence, publication status flag, and timestamp of the current frame for each key point.

[0116] Among them, safe distance Defined as the magnitude of the relative translation vector, it is used to quantify the overall proximity between the two robots, and the calculation formula is:

[0117]

[0118] in, Let be the relative translation vector of the reference points of both parties. The straight-line distance between reference points on both sides' torsos is the safety distance. The safety distance is the most directly used quantitative indicator by downstream safety control modules. When the distance is below the preset safety distance threshold, the safety control module can limit large punching movements to avoid collision damage.

[0119] The entire structured perception result is published in the form of timestamped data packets, which can be used in parallel by downstream modules such as punch decision module, defense selection module, evasion control module, distance control module, and safety constraint module. Each downstream module reads the corresponding field in the data packet according to its own needs, without having to repeatedly calculate spatial relationships.

[0120] In an optional embodiment of this application, the structured perception result also includes historical window information, which retains sliding statistics of key quantities from the most recent few frames, such as... , The rate of change in direction is used by downstream modules to determine trends. For example, if... A continuous decrease over several frames indicates that the enemy is rapidly approaching, allowing the decision-making module to initiate defense preparations in advance. The historical window length is configurable, typically set to correspond to frames ranging from tens to hundreds of milliseconds.

[0121] The structured perception results are published at a fixed frequency, aligned with the frame rate of the stereo camera. Under typical configurations, the frame rate is tens of frames per second, corresponding to a processing latency on the order of milliseconds per frame. The end-to-end latency from stereo image acquisition to the publication of structured perception results determines the response speed of downstream decision-making modules to changes in the offensive and defensive situation. The lower the latency, the more timely the system's response to rapid attacks and sudden approach.

[0122] Based on the above optional embodiments, this application embodiment realizes the transformation of abstract three-dimensional relative pose into intuitive distance situation indicators by including the safety distance calculated based on the relative translation vector of the two reference points in the structured perception results. This enables the downstream control module to directly use the safety distance for distance control, collision warning or attack range judgment, thereby providing key quantitative basis for robot movement control and safety constraints without additional calculation, and improving the direct service capability of perception results to the underlying control and real-time response efficiency.

[0123] Optionally, the three-dimensional coordinates of the user's key points are obtained from the robot's sensor data and kinematic model, while the two-dimensional coordinates of the opponent's key points are detected from the binocular image pair.

[0124] The robot's sensor data can come from internal sensors such as encoders and inertial measurement units (IMUs), and is used to describe joint angles, motor states, or inertial information.

[0125] A kinematic model is a geometric and mathematical model that describes the motion relationships between the links and joints of a robot. It is usually based on the Unified Robot Description Format (URDF) and joint angle inputs.

[0126] Regarding the strategy for acquiring one's own key points, the 3D coordinates of one's own key points differ from those of the other party. Since the binocular camera is mounted on the robot's head, most of the robot's body is not within the camera's field of view, making it impossible to obtain one's own key points through image detection. In an optional embodiment of this application, the 3D coordinates of one's own key points are obtained based on the robot's own sensor data and kinematic model. Specifically, the angle values ​​of each joint encoder are read, and the 3D position of each joint in the robot's base coordinate system is calculated using forward kinematics. This position is then transformed into the left-eye camera coordinate system using the head-camera rigidity transformation relationship. The accuracy of the forward kinematics calculation depends on the joint encoder resolution and the calibration accuracy of the link geometry parameters, typically reaching millimeter-level accuracy on industrial-grade robot platforms.

[0127] The two-dimensional coordinates of the opponent's key points are detected from the binocular images, and then their three-dimensional coordinates are obtained through triangulation. The system acquires a binocular image pair consisting of left and right eye images simultaneously captured by the head-mounted binocular cameras. A key point detection network is used to detect the two-dimensional pixel coordinates of each key point of the opponent in both the left and right eye images, thereby obtaining the two-dimensional position information of the opponent's key points on the image plane. This information is then triangulated to obtain the three-dimensional coordinates. After unifying the two sets of three-dimensional coordinates into the same coordinate system—the left eye camera coordinate system—relative pose and interaction distance can be calculated.

[0128] Based on the above optional embodiments, this application embodiment obtains the three-dimensional coordinates of its own key points according to the robot's sensor data and kinematic model, and obtains the three-dimensional coordinates of the opponent's key points from the binocular image pair by detection and triangulation. This realizes the separation and complementary advantages of the data sources of self-state perception and opponent visual perception. It uses the data of the inner sensor to ensure the high accuracy and low latency of its own posture, and uses the visual data to obtain the spatial position of the opponent. Thus, under the condition of relying only on the head binocular vision, it fully constructs the adversarial perception foundation that includes its own accurate body posture and the opponent's visual observation, and provides accurate and complementary data input for the subsequent calculation of the relative pose of both parties and the modeling of the interaction relationship.

[0129] In an optional embodiment of this application, the latency distribution of each stage of the above-described end-to-end processing flow is roughly as follows: binocular synchronization and distortion correction account for a small proportion; encoder feature extraction and detection head inference account for the main computation time; triangulation and pose calculation are lightweight geometric operations with negligible time consumption; and gating determination is also a simple comparison operation. If the computing resources of the deployment platform are limited, lower latency can be achieved by reducing the number of channels in the encoder backbone network or the input resolution, at the cost of a slight decrease in keypoint detection accuracy. This accuracy-latency tradeoff can be determined by the specific deployment scenario requirements.

[0130] According to another aspect of the embodiments of this application, a binocular perception method for adversarial scenarios is also provided. Figure 3 This is a schematic diagram of an optional binocular perception method for adversarial scenarios according to an embodiment of this application, as shown below. Figure 3 As shown. The method includes:

[0131] Step S301: Acquire the synchronously acquired binocular image pairs;

[0132] Step S302: Based on a preset set of adversarial semantic key points, detect the two-dimensional coordinates of each key point of the adversarial side in the left and right eye images from the binocular image pair. The preset set of adversarial semantic key points includes the target attack part and the target defense part.

[0133] Step S303: Based on the two-dimensional coordinates of each key point in the left and right eye images, perform binocular triangulation to obtain the three-dimensional coordinates of each key point.

[0134] Step S304: Construct a relative coordinate system based on the reference points and orientation information of both parties. Under the relative coordinate system, calculate the three-dimensional relative pose between the two parties and the distance between key parts based on the three-dimensional coordinates of each key point of both parties.

[0135] Step S305: Determine the structured perception result based on the three-dimensional relative pose and the distance between key parts.

[0136] When acquiring binocular image pairs, the binocular cameras mounted on the robot's head are controlled to ensure that the left and right cameras capture the left and right RGB images of the scene simultaneously at the same time or after a strict time synchronization mechanism, forming a pair of visual data inputs for subsequent processing.

[0137] The acquired left and right eye images are input into the keypoint detection network. The target attack area and target defense area defined in the preset adversarial semantic keypoint set are used as detection targets. The two-dimensional pixel coordinates of these keypoints are predicted and output in the left and right eye images respectively, and the visibility confidence of each keypoint is also output.

[0138] Using pre-calibrated intrinsic and extrinsic parameters of the binocular camera, such as focal length, principal point, baseline length, and rotation and translation matrix, the two-dimensional coordinates of the same key point detected above in the left and right eye images are correlated. The three-dimensional coordinates of the key point in the camera coordinate system are calculated using the principles of trigonometric geometry, thus completing the conversion from two-dimensional visual information to three-dimensional spatial information.

[0139] Each robot selects its own reference point and orientation vector, and constructs a relative coordinate system based on one of them. The 3D coordinates of all key points on both sides are transformed into this relative coordinate system. The translation vectors between the reference points and the rotation matrix between the orientation vectors are calculated to obtain the 3D relative pose. Simultaneously, the 3D Euclidean distance between the key points on the attacking side and the key points in the target area on the defending side is calculated to obtain the distance between key components.

[0140] The calculated 3D relative pose, distance between key parts, timestamp, confidence level, and release status are combined into a standardized data structure to form structured perception results for use by downstream control modules.

[0141] Based on the above steps S301 to S305, this embodiment of the application synchronously acquires binocular image pairs using the robot's binocular camera, detects the two-dimensional coordinates of the opponent's key points using the target attack and defense parts contained in the preset adversarial semantic key point set, obtains the three-dimensional coordinates through binocular triangulation, and constructs a relative coordinate system based on the reference points and orientation information of both parties to calculate the three-dimensional relative pose and the distance between key parts, and finally determines the structured perception result. This enables the real-time and accurate acquisition of the three-dimensional spatial position and relative pose relationship of the opponent's key parts under the condition of relying only on the head's binocular vision and without external motion capture equipment. This provides accurate situational awareness data for punching decisions, defense selection, distance control and safety constraints, improves the robustness and perception accuracy of the robot in boxing confrontation scenarios, and solves the technical problems of poor robustness and low perception accuracy of related methods when performing binocular perception.

[0142] In one alternative embodiment, the implementation of the training phase is further described in conjunction with another embodiment.

[0143] The goal of the training phase is to enable the encoder and detector head to accurately detect the 2D coordinates and visibility of semantic key points in a boxing combat scenario. The training data comes from stereo images recorded synchronously during live combat, as well as weakly supervised labels generated based on the robot's own state and kinematic model.

[0144] One iteration of the training phase includes the following processes: sampling a batch of binocular image pairs and corresponding labels from the training set; feeding the images into the encoder and detector head for forward inference to obtain the predicted 2D coordinates and visibility of each keypoint; performing triangulation on the predicted coordinates to obtain the predicted 3D coordinates; backprojecting the predicted 3D coordinates back onto the left and right image planes to calculate the reprojection loss; simultaneously using the predicted 3D coordinates to calculate the relative poses of both eyes and comparing them with the labeled relative poses; and summarizing the reprojection loss and other supervision signals before performing backpropagation to update the network parameters.

[0145] The formula for calculating the reprojection loss is:

[0146]

[0147] in, and These represent the projection functions for the left and right eyes, respectively, used to map three-dimensional spatial points onto a two-dimensional image plane. Represents the three-dimensional coordinates of the i-th key point. and These represent the two-dimensional observed coordinates or ground truth coordinates of the i-th keypoint in the left and right eye images, respectively. N represents the number of valid keypoints. This represents the square of the Euclidean distance. This formula forces the network to learn a 3D pose that conforms to binocular geometric constraints by minimizing the reprojection error of the predicted 3D coordinates onto the left and right eye images. This improves the geometric consistency and accuracy of 3D keypoint prediction, providing high-quality input data for subsequent relative pose calculations.

[0148] In addition to reprojection loss, the training objective also includes a 2D keypoint detection loss. This loss directly supervises the deviation between the 2D coordinates output by the detection head and the labeled 2D coordinates. In one embodiment of this application, the 2D keypoint detection loss adopts a smoothed L1 form, calculating the coordinate deviation for each visible keypoint in both the left and right eyes and averaging the results. The visibility branch is supervised by a binary cross-entropy loss, and the labels are derived from the occlusion state determination in the weakly supervised annotation process.

[0149] The various loss terms are balanced using preset weighting coefficients. The reprojection loss typically has a higher weight because it directly constrains the consistency between the 2D detection results and the 3D geometry; the 2D coordinate loss has the next higher weight; and the visibility loss has the lowest weight but is indispensable. Lack of visibility supervision can cause the network to generate false high-confidence predictions at occluded keypoints.

[0150] Data augmentation strategies are also introduced into the training data to improve the model's generalization ability. In one embodiment of this application, data augmentation includes geometric augmentation and photometric augmentation. For example, geometric augmentation can be random cropping, scaling, or horizontal flipping, while photometric augmentation can be random perturbation of brightness, contrast, and hue. Since binocular image pairs need to maintain geometric consistency between the left and right eyes, geometric augmentation applies the same transformation parameters simultaneously to both eyes, while photometric augmentation can be applied independently to the left and right eyes to simulate the optical differences of binocular cameras.

[0151] In an optional embodiment of this application, generative augmentation based on 3D rendering can also be introduced. Specifically, using the robot's 3D model and known joint angle configurations, binocular image pairs under different viewpoints and lighting conditions are rendered in a virtual environment, while the precise 2D coordinates of each keypoint are automatically obtained as labels. The rendered samples can supplement the scarce extreme poses and rare occlusion patterns in the real machine data. When training with rendered samples mixed with real machine samples, the impact of inter-domain differences on model performance can be mitigated through domain adaptation techniques or simple sample weight adjustments.

[0152] After training, the model parameters are frozen and exported in deployment format. Before deployment, offline validation is required. This involves running the complete deployment inference process on an independent validation set, statistically analyzing metrics such as keypoint detection accuracy, 3D coordinate error, and relative pose deviation. Only after confirming that all metrics are above the preset release threshold can the model be deployed online. The validation set should include scene distributions similar to actual deployment conditions, i.e., combat segments with different distances, speeds, and degrees of occlusion.

[0153] Figure 4 This is a schematic diagram of another optional binocular perception method for adversarial scenarios according to an embodiment of this application, such as... Figure 4 As shown, this method sequentially executes four stages: boxing semantic key points, target region and adversarial coordinate system construction, training dataset construction, training of binocular adversarial key point perception network and interaction relationship modeling module, deployment inference and situational output.

[0154] In the first stage, the system defines a set of semantic key points for boxing, including key points on the attacking side and key points on the defending side, based on the robot's URDF structure, shell geometry and competition rules, and establishes an adversarial coordinate system and interaction graph topology.

[0155] The system then proceeds to the second stage, where it uses real machine combat videos, synchronized left and right eye images, and robot state data to construct a training dataset containing weakly supervised labels and generative augmented data through 3D keypoint calculation, visibility estimation, occlusion annotation, and strategy action stage analysis.

[0156] In the third stage, the system trains a binocular adversarial keypoint perception network based on the constructed training dataset. The network extracts features from the left and right eyes, predicts two-dimensional keypoints and visibility, uses reprojection loss for supervision, and learns the relative poses, interaction distances and interaction relationships between the two parties.

[0157] In the fourth stage, the system receives real-time synchronized binocular RGB images, performs 2D detection and binocular triangulation to obtain 3D coordinates, calculates the 3D relative pose and distance between key parts based on the adversarial coordinate system, and performs quality gating judgment by calculating the average reprojection error and pose confidence. If the average reprojection error is less than or equal to a preset error threshold and the pose confidence is greater than or equal to a preset confidence threshold, the structured perception results including the safety distance are published normally; otherwise, a downgraded publication is performed.

[0158] The process concludes with a feedback and review stage. The system archives error frames based on task execution results, security trigger events, or manual sampling. Qualified samples are retained, while error samples are returned to the second stage for sample screening or incremental training in the third stage. Meanwhile, the system only performs model updates in an offline environment after verification, thus forming a closed-loop link of "semantic definition - data construction - model training - deployment output - feedback and review".

[0159] Figure 5 This is a schematic diagram of another optional binocular perception method for adversarial scenarios according to an embodiment of this application, such as... Figure 5 As shown, the data flow in the system presents a closed-loop structure of "input → processing → output → feedback". The forward data flow starts from binocular observation and calibration parameters, passes through perception processing to obtain intermediate states and quality assessment results, and then, after gating, forms a structured release for use by downstream decision-making and control modules. The reverse data flow returns from the downstream usage stage to the perception processing stage, but is only triggered under offline conditions. When the downstream security module records an abnormal event or a malfunctioning frame is found during manual sampling, the corresponding image and perception output are archived and, after review, a decision is made on whether to include them in the subsequent training set. This feedback path does not affect the real-time processing at the deployment end; it is only used to support the offline iterative optimization of the model.

[0160] Figure 6 This is a schematic diagram of another optional binocular perception method for adversarial scenarios according to an embodiment of this application, such as... Figure 6As shown, this method encompasses the complete logic from 3D reconstruction to final deployment decision. After acquiring the observed and intermediate quantities, this embodiment calculates the 3D keypoint coordinates based on the corrected binocular triangulation formula, using the 2D coordinates of the left and right keypoints and camera calibration parameters. Subsequently, it enters the binocular keypoint reprojection supervision stage, calculating the deviation between the predicted reprojection error of the 3D keypoints on the left and right images and the observed values, evaluating geometric consistency, and using it for training optimization. Next, based on the relative pose and interaction distance estimation of both sides, an adversarial coordinate system is constructed based on the reference points and orientation information of both sides. The relative translation vector, relative rotation matrix, and 3D Euclidean distance between the attacking side keypoints and the defending side target area keypoints are calculated, thereby quantifying the offensive and defensive posture. Finally, a deployment deployment gating is used for final determination. The system calculates the average reprojection error and pose confidence in real time. If the average reprojection error is less than or equal to a preset error threshold and the pose confidence is greater than or equal to a preset confidence threshold, it is determined to be a normal deployment state; otherwise, a downgraded deployment mechanism or review is triggered to ensure that the structured perception results output to the downstream control module have sufficient security and reliability.

[0161] Figure 7 This is a schematic diagram of another optional binocular perception method for adversarial scenarios according to an embodiment of this application, such as... Figure 7 As shown. The system first receives synchronized left and right eye RGB images, i.e., the left eye image. Right eye image Simultaneously, calibration parameters between the left and right eyes are acquired. This data is then fed into a shared-weight CNN encoder, which is responsible for extracting multi-scale features. and The images are divided into left and right view images, respectively. The extracted features are then used to process keypoints in the left and right eyes, respectively. Each keypoint outputs its corresponding multi-scale features, 2D keypoint coordinates, and visibility confidence. Next, the keypoint information from both eyes is fed into a binocular triangulation correction module. This module uses a triangulation formula to accurately calculate 3D keypoints and introduces a reprojection-based training loss to optimize the model during training. After triangulation, the system calculates the relative pose and interaction distance, generating relative coordinates, i.e., adversarial coordinates. Finally, by deploying a gating output module, a gating decision formula is used for publishing or downgrading, completing the entire binocular adversarial keypoint perception process.

[0162] According to another aspect of the embodiments of this application, a binocular perception device for adversarial scenarios is also provided. Figure 8 This is a structural block diagram of an optional binocular sensing device for adversarial scenarios according to an embodiment of this application, such as... Figure 8As shown. The device includes: an acquisition module 801 for acquiring synchronously acquired binocular image pairs; a detection module 802 for detecting the two-dimensional coordinates of each key point of both sides in the left and right eye images from the binocular image pairs based on a preset set of adversarial semantic key points, wherein the preset set of adversarial semantic key points includes the target attack parts and the target defense parts; a triangulation module 803 for performing binocular triangulation processing based on the two-dimensional coordinates of each key point in the left and right eye images to obtain the three-dimensional coordinates of each key point; a calculation module 804 for constructing a relative coordinate system based on the reference points and orientation information of both sides, and calculating the three-dimensional relative pose and the distance between key parts of both sides based on the three-dimensional coordinates of each key point of both sides in the relative coordinate system; and a determination module 805 for determining the structured perception result based on the three-dimensional relative pose and the distance between key parts.

[0163] According to another aspect of the embodiments of this application, an electronic device is also provided. This electronic device can serve as the computing core of the robot described above, or it can be deployed as a standalone edge computing device. The electronic device includes a processor, a storage medium, a binocular camera interface, and a communication interface, and the components are connected via a bus.

[0164] The processor is used to execute instructions stored in the storage medium. The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a neural network accelerator, or a combination of these. When the processor is a neural network accelerator, the inference of the encoder and detector head can be accelerated at the hardware level, which is beneficial for meeting real-time requirements.

[0165] The storage medium contains computer-executable instructions. When the processor executes these instructions, it implements the various steps of the binocular perception method for adversarial scenarios. Model parameters are loaded in read-only mode during deployment unless a new version is released after offline verification.

[0166] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0167] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0168] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0169] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0170] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0171] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0172] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0176] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A robot, characterized in that, include: trunk; Two upper limbs and two lower limbs, respectively connected to the torso; A binocular camera mounted on the head; At least one processor; as well as At least one storage medium storing instructions that, when executed by the at least one processor, cause the robot to: Acquire the binocular image pairs synchronously acquired by the binocular camera; Based on a preset set of adversarial semantic key points, the two-dimensional coordinates of each key point of the opponent in the left and right eye images are detected from the binocular image pair. The preset set of adversarial semantic key points includes the target attack points and the target defense points. Based on the two-dimensional coordinates of each key point in the left and right eye images, binocular triangulation is performed to obtain the three-dimensional coordinates of each key point. A relative coordinate system is constructed based on the reference points and orientation information of both parties. Under the relative coordinate system, the three-dimensional relative pose between the two parties and the distance between key parts are calculated according to the three-dimensional coordinates of each key point of both parties. The structured perception result is determined based on the three-dimensional relative pose and the distance between the key parts.

2. The robot according to claim 1, characterized in that, The preset set of adversarial semantic key points includes: attack side key points and defense side key points. The attack side key points include fist key points, and the defense side key points include: head target area key points and torso target area key points.

3. The robot according to claim 2, characterized in that, The key points on the attack side also include the outer edge of the fist; the key points on the defense side also include key points in the forearm defense area, key points in the supporting foot area, and key points in the vulnerable areas of the fuselage.

4. The robot according to claim 1, characterized in that, The construction of a relative coordinate system based on the reference points and orientation information of both parties includes: The origin reference is determined by the reference points of both sides' torsos, and the direction of the coordinate axes is determined by the orientation vectors of both sides, wherein the orientation vectors are determined by the direction of the line connecting the preset key point pairs; The relative coordinate system is constructed based on the origin reference and the coordinate axis directions.

5. The robot according to claim 1, characterized in that, The instructions, when executed by the at least one processor, also cause the robot to: Using semantic key points of both sides as nodes and the three-dimensional distance between key points on the attacking side and key points in the target area on the defending side as edge features, an interaction graph between the two sides is constructed. The interaction graph is used to calculate the distance between the key parts.

6. The robot according to claim 1, characterized in that, The instructions, when executed by the at least one processor, also cause the robot to: When detecting the two-dimensional coordinates of each key point of the other party, the visibility confidence of each key point is output simultaneously; When the visibility confidence of the key point is lower than a preset visibility threshold, the three-dimensional coordinates of the key point are corrected.

7. The robot according to claim 1, characterized in that, The instructions, when executed by the at least one processor, also cause the robot to: Obtain the average reprojection error and pose confidence. In response to the average reprojection error being less than or equal to a preset error threshold, and the pose confidence being greater than or equal to a preset confidence threshold, the structured perception result is published normally. In response to the average reprojection error being greater than the preset error threshold, and / or the pose confidence being less than the preset confidence threshold, the structured perception result is downgraded and published.

8. The robot according to claim 1, characterized in that, The structured perception result also includes the safety distance calculated based on the relative translation vector of the two reference points.

9. The robot according to claim 1, characterized in that, The three-dimensional coordinates of the user's key points are obtained based on the robot's sensor data and kinematic model, while the two-dimensional coordinates of the opponent's key points are detected from the binocular image pair.

10. A binocular perception method for adversarial scenarios, characterized in that, include: Acquire synchronously acquired binocular image pairs; Based on a preset set of adversarial semantic key points, the two-dimensional coordinates of each key point of the adversarial parties in the left and right eye images are detected from the binocular image pair. The preset set of adversarial semantic key points includes the target attack position and the target defense position. Based on the two-dimensional coordinates of each key point in the left and right eye images, binocular triangulation is performed to obtain the three-dimensional coordinates of each key point. A relative coordinate system is constructed based on the reference points and orientation information of both parties. Under the relative coordinate system, the three-dimensional relative pose between the two parties and the distance between key parts are calculated according to the three-dimensional coordinates of each key point of both parties. The structured perception result is determined based on the three-dimensional relative pose and the distance between the key parts.

Images