Mobile robot system for performing checkout processes at unchanged checkout workstations and for stabilized interaction with stationary human-machine interfaces.

Abstract

Mobile robot system for carrying out checkout transactions at an unchanged checkout workstation designed for human operation, comprising - a mobile base, - at least one manipulator arm with an end effector, mounted on the mobile base, for operating input devices of the cash register workstation, - at least one camera for optically capturing a cash register screen and other display devices at the cash register workstation, - a control unit characterized in that - the control unit is set up to evaluate image data from the camera in order to detect graphical controls displayed on a cash register screen and to determine their positions in screen coordinates, - the control unit is further equipped to determine a transformation relationship between the screen coordinates and a robot coordinate system as part of a self-calibration procedure, - the manipulator arm can be controlled in such a way that the end effector mechanically actuates selected graphical controls with a predetermined contact force based on the transformation relationship, - the robot system is additionally set up to perform all the necessary operating actions for a checkout process at the checkout workstation, including item registration, entry of product group or price codes, confirmation of dialog boxes, and initiation and completion of cashless payment transactions by mechanically operating the input and output devices intended for human operation, - where no interface for exchanging article data, price data or operating commands is provided between the control unit of the robot system and a cash register system of the checkout workstation, so that all information required for the checkout process is derived exclusively from visually perceptible states of the input and output devices intended for human operation.

Description

Technical field

[0001] The invention relates to the use of mobile robot systems in retail and service-oriented environments. In particular, the invention relates to a mobile robot system for carrying out checkout processes at essentially unchanged checkout workstations designed for human operation, as well as for physical interaction with stationary human-machine interfaces, in particular touchscreen control panels, without mechanical fixation to the environment. State of the art

[0002] Various automation solutions for checkout processes are known from the current state of the art. Self-service checkouts, where customers scan items and initiate payments themselves, are widespread. In addition, fully integrated systems exist in which robots or automated conveyor systems are linked to merchandise management and point-of-sale software via digital interfaces. In such systems, item data, price data, and operating commands are typically exchanged via defined data protocols, application programming interfaces (APIs), or virtual input devices (e.g., emulated keyboard / mouse signals).

[0003] Furthermore, test systems for touchscreens are known from robotics, in which stationary or Cartesian industrial robots execute defined pressure movements on control panels using an end effector with force sensors. Known methods are used to calibrate the robot's path relative to the screen, determining a mapping between screen pixel coordinates and robot coordinates.

[0004] In the field of humanoid or mobile robots, methods for whole-body control and maintaining stability under external force are established.

[0005] In particular, the Zero Moment Point (ZMP) concept is well known, in which the position of a resulting tipping moment relative to a support surface is monitored and controlled by appropriate movements of the legs, upper body and arms.

[0006] Recently, model predictive control (MPC) methods and learning-based approaches, particularly those based on reinforcement learning, have been increasingly used for stabilizing and planning the motions of humanoid and mobile robots. These methods generate whole-body movements via optimization problems or by means of control policies trained in simulations, offering greater flexibility and robustness against disturbances and unstructured terrain compared to purely ZMP-based methods. The invention is independent of the specific choice of control paradigm and can utilize both classical ZMP-based control systems and model predictive and learning-based methods.

[0007] However, the known solutions have significant limitations: In integrated checkout and logistics systems, the robot is typically deeply integrated into the IT infrastructure; therefore, its use at existing, unchanged checkout workstations is not possible without adapting the checkout hardware or software.

[0008] Test robots for touchscreens are usually stationary and not designed to reach multiple different control panels in a real retail environment or to independently perform all the necessary operations for a checkout process.

[0009] Humanoid robots with whole-body control focus on locomotion and manipulation, without specifically addressing robust, freestanding interaction with fixed control panels in confined checkout environments.

[0010] At least no system is known that allows a conventional, human-operated checkout workstation to be fully operated by a mobile robot in its unchanged state, without any digital interface to the cash register system, so that the robot derives all information solely from visually perceptible states of the existing input and output devices. Likewise, at least no mobile robot system is described that, while operating stationary human-machine interfaces, remains freestanding without any mechanical fixation or support from its surroundings and exerts contact forces via whole-body control without losing its stability. Object of the invention

[0011] Based on this state of the art, the invention aims to provide a robot system that enables, - to automate an existing checkout workstation designed for human operation without structural or software changes, by having a mobile robot system take over all the operating actions required for a checkout process, namely • without any electronic or logical data interface between robot control and cash register system (“air gap”), • exclusively based on optically recorded states of the cash register screen, payment terminal and other input / output devices; - to provide a mobile robot system for physical interaction with stationary human-machine interfaces that can be operated freely without mechanical fixation to the environment and can nevertheless perform stable operating actions, in particular pressing movements on touch or keypads, without losing its stability. Summary of the invention

[0012] This problem is solved by a mobile robot system according to claim 1. A further advantageous embodiment of the invention is provided by a mobile robot system according to claim 8. Advantageous embodiments are described in the dependent claims.

[0013] In a first embodiment, the invention relates to a mobile robot system for performing checkout operations at an unmodified checkout workstation designed for human operation. The robot system comprises a mobile base, at least one manipulator arm with an end effector, one or more cameras, and a control unit. The control unit is configured to detect graphical user interface elements on a checkout screen using camera images, to determine their positions in screen coordinates, and to establish a transformation relationship between screen coordinates and a robot coordinate system through a self-calibration process. Based on this transformation relationship, the manipulator arm can be controlled so that the end effector mechanically actuates selected graphical user interface elements with a defined contact force.The robot system is designed to perform all necessary operations for a checkout process, including item registration, entry of product group or price codes, confirmation of dialog boxes, and initiation and completion of cashless payments, solely through mechanical operation of the input and output devices intended for human use. No interface for exchanging item data, price data, or operating commands is provided between the robot system's control unit and the point-of-sale system.

[0014] Further details include, in particular: - a visual self-calibration procedure using reference images or graphic reference elements of the cash register screen; - a classification of the detected graphical controls into functional categories and a modeling of the cash register dialog as a state machine based on this; - a force or impedance control of the manipulator arm to keep contact forces within a permissible range and to dampen vibrations of the mobile base induced by pressure movements; - an additional camera for optical detection of a spatially separate payment terminal, which allows the robot to also operate this terminal (e.g. PIN pad, confirmation buttons); - the continuous verification of the success of user actions by comparing images of the cash register screen before and after an input, as well as the triggering of error handling routines if no change in state occurs; - a learning mode in which the operating actions of a human cashier are recorded and used to derive mappings between screen states and end effector paths.

[0015] In a further embodiment, the invention relates to a mobile robot system for physical interaction with a stationary human-machine interface, in particular a touchscreen control panel. The robot system has a mobile base with free-floating base dynamics, at least one manipulator arm, and a control unit. It is designed to remain freely standing during interaction without mechanical fixation to the human-machine interface or its support structure. The control unit implements a whole-body control algorithm which, when a contact force is exerted by the end effector, regulates the position of the overall center of gravity relative to a support surface in such a way as to maintain a stability criterion, and which coordinates the pose of the manipulator arm and base to generate the desired contact force without compromising stability. Advantageous embodiments of the invention

[0016] In advantageous embodiments, the self-calibration method includes capturing a reference image of the cash register screen, in which predefined graphic reference elements (e.g., corners, marks, logos, frame elements) are automatically recognized. From their positions in image coordinates, the parameters of a transformation relationship (e.g., homographic mapping, affine or projective transformation) are determined, which allows a mapping between screen and robot coordinates. Alternatively or additionally, geometric features of the physical screen, such as visible frames, can be used.

[0017] The control unit can perform a semantic classification of the elements from the detected layout of graphical user interface elements, for example, into categories such as confirmation, cancellation, product group selection, price / PLU input, discount, payment method selection, and start / end of the checkout process. Based on this classification, a state machine is built that models the progressing checkout dialogue. For each state, permissible subsequent states are defined, which are triggered by specific user actions (e.g., pressing an "OK" button).

[0018] In further advantageous embodiments, the manipulator arm is equipped with force and / or torque sensors, for example, in the form of a force-torque sensor integrated into the end effector. The control unit implements force or impedance control, in which a target contact force is specified and control deviations are determined from measured actual forces in order to adjust the drives of the manipulator arm accordingly. This allows for a defined, limited pressure force on sensitive touchscreens and damping of vibrations of the mobile base. Vibrations introduced into the system by pressing on the cash register screen are also taken into account within the framework of whole-body control.

[0019] In one embodiment, the control unit for this whole-body control uses the same model predictive or learning-based methods as described for freestanding interaction with general human-machine interfaces, so that stability and comfort criteria are uniformly considered both during checkout processes and when operating other control panels.

[0020] The robot system can additionally include one or more cameras for capturing a separate payment terminal. The control unit recognizes graphical user interface elements such as payment method selection, amount display, and confirmation symbols, as well as physical keypads for PIN entry. Based on the determined transformation relationship between the terminal screen or keypad and the robot's coordinate system, the manipulator arm can mechanically actuate the user interface elements required for the payment transaction.

[0021] To increase robustness, the system repeatedly captures and compares image data from the cash register screen during the checkout process, particularly immediately before and after each action performed by the robot. Based on detected changes in the screen state, it is verified whether the intended effect has occurred. If not, the control unit can trigger an error handling routine, such as repeating the input, using an alternative input sequence, returning to a defined state, or requesting external support.

[0022] In a learning mode, the robot system can record the actions of a human cashier at the same checkout workstation by simultaneously capturing screen displays and the positions or paths of a human hand. For this purpose, the hand can be marked with a marker or tracked using 3D tracking (e.g., a depth camera). The control unit extracts sequences of screen states and corresponding hand paths from the data. Using imitation learning, images can be generated from this data that define suitable end-effector paths and operating sequences for specific recognized screen states. In one embodiment, reinforcement learning is used additionally or alternatively, in which the control unit tries out different operating strategies in a simulation environment or a secure test environment and rewards the system with a signal, such as...Taking into account successful checkout processes, short processing times, or error minimization, a policy is learned. The policies learned in this way can be transferred to the real robot system and executed there, or combined with model-based strategy components (sim-to-real approach).

[0023] In one embodiment of the invention for stabilized interaction with a stationary human-machine interface, the control unit can implement force or impedance control for the manipulator arm in addition to the whole-body control algorithm. The forces detected by contact force sensors are used to selectively control the force exerted by the end effector and simultaneously maintain the overall center of gravity of the robot system within a permissible stability range. In the event of deviations of the overall center of gravity from a permissible range caused by external disturbances (e.g., impact by a person, uneven ground), the control unit can initiate a compensatory movement of the mobile base, in particular a stepping movement in the case of a bipedal configuration, to restore stability.

[0024] In a preferred embodiment, the mobile base is designed as a bipedal walking robot with two legs. The whole-body control algorithm hierarchically coordinates the joint angles of the legs, the posture of the upper body, and movements of the manipulator arm to ensure both stability and the execution of a control action. Alternatively, the base can be designed as a self-balancing, wheeled robot (e.g., two wheels with dynamic stabilization). Brief description of the drawings

[0025] Drawings can be used to further explain the invention, in which: Fig. 1 shows a schematic perspective representation of a mobile robot system (1) at a conventional checkout workstation (8) with checkout screen (9) and separate payment terminal (10), wherein a manipulator arm (3) with an end effector (4) operates the checkout screen (9); Fig. 2 shows a block diagram of a control unit (7) of the robot system (1) which processes image data from cameras (5, 6), performs dialogue and motion planning in several control modules (16), outputs control signals to a mobile base (2), a manipulator arm (3) and an end effector (4) and receives measurement signals from a force sensor or force control (15) as feedback; Fig. Figure 3 shows a schematic representation of a free-standing robot (1) when operating a stationary human-machine interface (11), with the position of an overall center of gravity (13) relative to a support surface (12) and an exemplary stability criterion in the form of a ZMP (14). Detailed description of implementation examples; basic structure of the robot system for checkout processes

[0026] In a first embodiment, the robot system comprises a mobile base, which can be designed, for example, as a bipedal walking robot or as a mobile undercarriage. At least one manipulator arm is arranged on the mobile base, comprising several rotary and / or translational joints, and an end effector is arranged at its free end. The end effector can be designed as a finger element, stylus, soft elastic contact surface, or multi-finger gripper to reliably operate and handle touchscreens, physical buttons, and, if applicable, merchandise.

[0027] As in Fig. As shown in Figure 1, the robot system (1) in one embodiment comprises a mobile base (2) in the form of a humanoid walking robot, on whose upper body a manipulator arm (3) with an end effector (4) is arranged. A camera (5) is arranged on the head or upper body of the robot and captures the cash register screen (9) of a checkout workstation (8). In the embodiment shown, the checkout workstation (8) has a conveyor belt, the cash register screen (9), and a payment terminal (10) arranged spatially separately from it. The end effector (4) is designed to mechanically actuate graphical operating elements on the cash register screen (9), for example, by pressing a touchscreen. Alternatively, the mobile base (2) can be designed as a mobile undercarriage with the upper body mounted on it.

[0028] A first camera is positioned to capture the cash register screen with sufficient resolution and a suitable perspective. The camera can be rigidly attached to the robot's upper body or movably mounted by means of a separate actuator. In an advantageous embodiment, several cameras with different focal lengths are provided to obtain both an overall view of the cash register workstation and detailed views.

[0029] The control unit comprises at least one processor and memory on which programs for image processing, classification, calibration, and motion planning are stored. It can be fully integrated into the robot or partially distributed across external computing units.

[0030] Fig. Figure 2 shows a schematic block diagram of a control unit (7) of the robot system (1). The control unit (7) receives image data from a camera (5) for the cash register screen and, optionally, from another camera (6) for a payment terminal. Within the control unit (7), several control modules (16) are provided, arranged in a processing chain. A first control module (16) can, in particular, perform image evaluation and segmentation functions of the cash register screen, a downstream control module (16) can perform calibration and transformation calculations between screen and robot coordinates, and a further control module (16) can handle the state machine or policy control of the cash register dialog as well as whole-body motion planning.

[0031] A force sensor (15), which can be arranged on the manipulator arm (3) or the end effector (4), is coupled to the control unit (7) and detects the contact forces occurring when the cash register screen (9) or payment terminal (10) is operated. The force sensor (15) delivers measurement signals to the control unit (7), in which an associated force control module, also designated (15), is implemented as a further control module in the processing chain. The force control module (15) processes the measurement signals together with setpoints provided by the other control modules (16) and calculates manipulated variables for the mobile base (2), the manipulator arm (3), and the end effector (4). On the output side, the control unit (7) sends corresponding control signals to the mobile base (2), the manipulator arm (3), and the end effector (4).

[0032] The checkout workstation is designed as an existing, unmodified system, as is common in supermarket, discount store, or department store environments. It typically comprises a checkout screen (usually a touchscreen), one or more physical keyboards, a barcode scanner, a scale, a customer display, a receipt printer, and a separate payment terminal with a screen and / or keypad. The invention requires that the checkout workstation is not modified, either in terms of hardware or software, for the robot. Self-calibration procedure and transformation relationship

[0033] Before or during its first use at a checkout counter, the robot performs a self-calibration procedure. For this, the control unit positions the mobile base in a working position where the checkout screen is fully within the field of view of the first camera and reachable by the manipulator arm.

[0034] The control unit instructs the camera to capture one or more reference images. An image processing module then recognizes characteristic features of the cash register screen. These can be graphic reference elements (e.g., frames, logos, menu bars, corners) or geometric contours of the physical display. A screen coordinate system, such as pixel coordinates, is defined from the recognized features.

[0035] In parallel, the robot can move the end effector to defined physical positions, for example, to the detected corners of the screen, and record the respective robot coordinates. From corresponding pairs of points in screen and robot coordinates, a transformation relationship is determined using numerical methods (e.g., solving a system of equations for affine or projective transformations). This transformation relationship allows a desired screen position to be converted into a target end effector position in the robot coordinate system.

[0036] This self-calibration procedure can be repeated regularly or reactivated upon detection of a change in the field of view, such as if the mobile base has been accidentally moved. Detection and classification of graphical user interface elements

[0037] During normal operation, the camera continuously captures images of the cash register screen. An image processing module detects graphical user interface elements such as buttons, input fields, list boxes, warning messages, or dialog boxes. Detection can be performed using classic image processing, deep neural networks (such as convolutional neural networks or vision transformer architectures), or a combination of these methods.

[0038] The detected controls are classified into functional categories according to shape, color, labeling, or context, for example: - Confirmation elements (e.g., "OK", "Confirm", green check mark); - Demolition elements (e.g. "Cancel", "Return", red cross); - Elements for selecting a product group (e.g. fruit / vegetable buttons); - Input fields for numeric identifiers (e.g. PLU input); - Elements for selecting a payment method (e.g. "card", "cash", "voucher"); - Dialog elements for discount, cancellation, receipt splitting, etc.

[0039] In one embodiment, the graphical controls and the resulting dialog state are determined by an end-to-end trained neural network that directly derives a symbolic representation of the controls and their functional context from a camera image or image sequence. This allows the control unit to robustly handle varying layouts and graphic styles of different point-of-sale systems.

[0040] Based on the classified elements, the control unit builds a state machine whose states correspond to the typical dialog states of the cash register system (e.g. "Scan item", "Show total", "Select payment method", "Enter PIN", "Issue receipt").

[0041] Transitions between states are realized through operator actions performed by the robot.

[0042] In another embodiment, the state machine is at least partially replaced or supplemented by a learning-based decision logic. For this purpose, the control unit can use a policy trained via imitation learning from recorded user sequences and / or via reinforcement learning in simulation. This policy selects a suitable next user action for a recognized dialog state and a set of available controls. In this way, even complex or modified dialog sequences can be processed without having to explicitly model all states and transitions. Execution of mechanical operating procedures

[0043] To execute a user action, the control unit first calculates the desired screen position of the graphical user interface element to be activated. Using the transformation relationship, these screen coordinates are converted into target coordinates for the end effector in the robot coordinate system. Path planning generates a collision-free trajectory, in which the end effector approaches the screen surface with a suitable movement.

[0044] In the final phase of the trajectory, a force or impedance control system is activated, limiting the contact force to a predefined range. The control unit continuously measures the contact forces occurring at the end effector and adjusts the joint positions accordingly to maintain the desired contact force until the operator action is recognized as complete (e.g., after a certain period of time or upon receiving a screen signal). The end effector is then lifted from the screen.

[0045] Path planning can take the pose of the mobile base into account. For example, if the controls are located further away, the base can be moved in a coordinated manner, or the upper body can be adjusted to maintain an ergonomic working area. Interaction with separate payment terminal

[0046] In an advantageous embodiment, a second camera is additionally provided, which captures a payment terminal located spatially separate from the cash register screen. The control unit can calibrate the payment terminal analogously to the calibration of the cash register screen and detect graphical user interface elements of the terminal screen. With a separate PIN pad and physical keypad, the system can determine the position of the keys (e.g., numbers, confirmation, delete) based on visual characteristics or through a one-time manual learning procedure.

[0047] During a payment transaction, the steps taken by the point-of-sale system and, if applicable, the customer are tracked by optically capturing the terminal screen. The robotic system can, for example, enter data on the payment terminal for customer cards, vouchers, or in system-authorized processes. The invention leaves open whether and to what extent customer input is automated; it merely provides the technical capability for mechanical actuation. Performance monitoring and error handling

[0048] To ensure high reliability, the control unit typically captures an image of the cash register screen immediately before and after each user action. By comparing the two images, differences in the screen's status are detected, such as the disappearance of a dialog box, the appearance of a confirmation symbol, or a change in the total display. A failure to observe the expected change in status is interpreted as an indication of a failed input.

[0049] In such cases, the robot system can execute an error handling routine. This might include, for example: - a repetition of the operating action, possibly with a slightly varied position or increased contact force; - resetting the checkout dialog to a known initial state; - displaying a help dialog by pressing a help key; - signaling an error to a human supervisor. Learning mode

[0050] In a learning mode, a human cashier can operate the checkout as usual while the robot system records observational data. A camera captures the cashier's hand and the checkout screen. Optionally, gloves with markers, inertial sensors, or other tracking systems can also be used.

[0051] The control unit extracts sequences of screen states and corresponding hand paths from the data. Using imitation learning, these data can be used to generate mappings that specify suitable end-effector paths and operating sequences for certain recognized screen states. In one embodiment, reinforcement learning is used additionally or alternatively. In this approach, the control unit is trained in a simulation environment or in a combination of simulation and real-world trials. Based on a reward signal that considers, for example, successful cashier transactions, short processing times, or error minimization, the unit learns a policy. The policies learned in this way can be transferred to the real robot system and executed there or combined with model-based strategy components (sim-to-real approach). Whole-body control and stabilization during HMI interaction

[0052] Fig.Figure 3 shows a side view of a robot system (1) with a mobile base (2), which in the illustrated embodiment is designed as a bipedal walking robot with two legs. The manipulator arm (3) carries an end effector (4) at its free end, which performs a pressure movement on a wall-mounted touchscreen control panel serving as a human-machine interface (11). The feet of the mobile base rest on the floor and form a support surface (12). A center of gravity (13) is determined within the robot, the projection of which onto the support surface (12) is shown as a dashed line. In the illustrated embodiment, a stability criterion is used, which is represented by a zero-moment point (ZMP) (14) within the support surface (12).

[0053] Another embodiment of the invention relates to the stabilization of a free-standing robot during interaction with stationary human-machine interfaces.

[0054] The robot is equipped with sensors to determine its pose, joint angles, and the forces acting on the end effector. It also captures information about the contact between the base and the surface, such as foot contact points in the case of a bipedal robot. The control unit continuously calculates the position of the robot system's overall center of gravity relative to the support surface, i.e., the area encompassed by the contact points.

[0055] The whole-body control algorithm determines target movements for the legs, torso, and manipulator arm such that, on the one hand, the desired end-effector movement, such as pressing a touch panel, is executed, and on the other hand, a stability criterion is met that quantifies the position of the overall center of gravity relative to the support surface. This stability criterion can be based, in particular, on the position of a zero-moment point (ZMP), a center of pressure, a capture point, and / or similar parameters that ensure the resulting contact forces remain within a permissible range. Optimization methods can be used for this purpose, taking into account the permissible torques of the joint actuators, comfort criteria, and safety distances.

[0056] In one embodiment, the whole-body control algorithm is designed as model predictive control (MPC). Here, trajectories for center of gravity, joint angles, and contact forces are optimized over a finite planning horizon, with the stability criterion being included as a constraint in the optimization problem. For this purpose, the control unit solves an optimization problem at predetermined time intervals and implements only the first section of the calculated trajectory as a sequence of manipulated variables (receding horizon principle).

[0057] If a contact force acts on the end effector, the controller attempts to reduce this force to a target value by adjusting the pose of the manipulator arm and, if necessary, the base position. In the event of excessive tilting moments, the control unit can initiate a compensating movement, such as a step backward or sideways step, to increase the support area and bring the overall center of gravity back into a safe range.

[0058] In an alternative or supplementary embodiment, the whole-body control algorithm includes a learning-based control component. This can be implemented, in particular, as a policy trained by imitation learning and / or reinforcement learning, which directly derives actuator parameters for the drives from sensory state variables, especially joint angles, velocities, contact information, and measured contact forces. Such a policy can be trained in a physical simulation or in a combination of simulation and real-world testing and is executed in the control unit during operation, whereby the stability criterion is implicitly or explicitly taken into account.

[0059] The human-machine interface can be a touchscreen control panel of a machine, an elevator, an access control system, or another technical system. The invention enables a mobile robot to operate such interfaces without additional brackets or support devices, maintaining its stability solely through movement and control of its limbs. Reference symbol list 1 robot system in total 2 mobile base 3 Manipulator arm 4 End effector 5 cameras for cash register screen 6 cameras for payment terminal 7 Control unit 8 checkout workstations 9 cash register screen 10 Payment terminal / PIN pad 11 Human-machine interface / Touchscreen control panel 12 support surface 13 Overall focus 14 ZMP (Zero Moment Point) 15 Force sensors and associated force control module 16 control modules (e.g. image evaluation, calibration, state machine, whole body control)

Claims

[1] Mobile robot system for performing cashier transactions at an unchanged cashier workstation designed for human operation, comprising - a mobile base, - at least one manipulator arm with an end effector, mounted on the mobile base, for operating input devices of the cash register workstation, - at least one camera for optically capturing a cash register screen and other display devices at the cash register workstation, - a control unit, characterized by , that - the control unit is set up to evaluate image data from the camera in order to detect graphical controls displayed on a cash register screen and to determine their positions in screen coordinates, - the control unit is further equipped to determine a transformation relationship between the screen coordinates and a robot coordinate system as part of a self-calibration procedure, - the manipulator arm can be controlled in such a way that the end effector mechanically actuates selected graphical controls with a predetermined contact force based on the transformation relationship, - the robot system is additionally set up to perform all the necessary operating actions for a checkout process at the checkout workstation, including item registration, entry of product group or price codes, confirmation of dialog boxes, and initiation and completion of cashless payment transactions by mechanically operating the input and output devices intended for human operation, - where no interface for exchanging article data, price data or operating commands is provided between the control unit of the robot system and a cash register system of the checkout workstation, so that all information required for the checkout process is derived exclusively from visually perceptible states of the input and output devices intended for human operation. [2] Robot system according to claim 1, characterized by that the self-calibration procedure includes capturing at least one reference image of the cash register screen and determining the transformation relationship by recognizing predefined graphic reference elements and / or geometric features of the cash register screen. [3] Robot system according to one of the preceding claims, characterized by, that the control unit is set up to classify the detected graphical operating elements into functional categories, in particular confirmation, cancellation, product group selection and payment method selection, and to control the checkout process step by step on the basis of a state machine modeled thereby. [4] Robot system according to one of the preceding claims, characterized by , that the manipulator arm is controlled by means of force or impedance control in such a way that the contact force applied when the cash register screen is operated is kept within a specified force range and the vibrations of the mobile base originating from the contact are dampened. [5] Robot system according to any of the preceding claims, characterized by, that the robot system additionally includes at least one camera for the optical detection of a payment terminal spatially separate from the cash register screen and is set up to optically recognize and mechanically operate graphical control elements and input devices of the payment terminal, in particular fields for PIN entry and confirmation buttons. [6] Robot system according to one of the preceding claims, characterized by , that the control unit is set up to repeatedly capture image data from the cash register screen before and after an operation performed by the robot during the checkout process, to compare the image data with each other and to verify the success or failure of the operation based on a detected change in the state of the cash register screen and, if necessary, to trigger an error handling routine. [7] Robot system according to one of the preceding claims, characterized bythat the robot system has a learning mode in which the operating actions of a human cashier at the checkout workstation are recorded by simultaneously capturing screen displays and positions of a human hand, and that the control unit derives from the recordings a mapping between screen states and associated end effector paths for later autonomous operation. [8] Mobile robot system for physical interaction with a stationary human-machine interface, in particular a touchscreen control panel, comprising - a mobile base with a free-floating base dynamic, - at least one manipulator arm with an end effector, arranged on the mobile base, for exerting forces on human-machine interface controls, - at least one sensor device for recording the state of the human-machine interface, - a control unit, characterized by , that - the robot system is set up to remain freestanding without mechanical fixation to the human-machine interface or any supporting structure during interaction with the human-machine interface, - the control unit implements a whole-body control algorithm which, when a contact force is exerted by the end effector, controls the position of the overall center of gravity of the robot system relative to a support surface of the mobile base in such a way that a stability criterion, in particular a position of a ZMP (Zero Moment Point) (14) within the support surface, is met, - and the control unit is set up to coordinate the pose of the manipulator arm and the mobile base during a pressure movement of the end effector on a control element, so that the desired contact force is generated without losing the stability of the robot system. [9] Robot system according to claim 8, characterized by that the control unit additionally implements a force or impedance control for the manipulator arm, by means of which a target contact force is specified and actual forces detected by contact force sensors are adjusted during the interaction with the human-machine interface. [10] Robot system according to one of claims 8 or 9, characterized by , that the control unit is designed to initiate a compensatory movement of the mobile base, in particular a stepping movement, in order to restore the stability of the robot system in the event of deviations of the overall center of gravity from a permissible range caused by external disturbances. [11] Robot system according to one of claims 8 to 10, characterized by, that the mobile base is designed as a bipedal walking robot with two legs and the whole-body control algorithm hierarchically coordinates the joint angles of the legs, the posture of the upper body and movements of the manipulator arm in order to simultaneously ensure stability and the execution of an operating action at the human-machine interface.

Images