Robot control method and device and robot

Abstract

The invention provides a robot control method and device and a robot. The robot is provided with a mechanical arm, the mechanical arm is provided with a tail end executing mechanism, and the method comprises the steps that a voice grabbing instruction for classroom supplies is received, and the voice grabbing instruction is used for instructing the robot to grab the classroom supplies to a first area; in response to the voice grabbing instruction, controlling a mechanical arm to move a tail end execution mechanism to the classroom supplies, and controlling the tail end execution mechanism to grab the classroom supplies; and the mechanical arm is controlled to move and grab the tail end executing mechanism with the classroom supplies to the first area, and the tail end executing mechanism is controlled to place the classroom supplies in the first area. According to the invention, the robot can be controlled to automatically grab and place the classroom supplies, and the efficiency and quality of classroom management are improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robot control method, device and robot. Background Art

[0002] With the development of artificial intelligence, robots have entered a golden age of development, with more and more companies launching robots for use in a variety of scenarios.

[0003] In educational scenarios, classroom management is an important task, involving the delivery of items, the organization of desks, and other daily maintenance work. These tasks are mainly completed manually, which is not only time-consuming and laborious, but also prone to omissions, affecting the smooth progress of teaching activities. For example, teachers need to frequently pass teaching aids, books, and stationery, and students also need to regularly organize desks and public areas. These repetitive tasks increase the workload of teachers and students and reduce the efficiency and quality of classroom management. Summary of the invention

[0004] The embodiments of the present application provide a robot control method, device and robot, which can control the robot to automatically complete the grabbing and placement of classroom supplies, thereby improving the efficiency and quality of classroom management.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a robot control method, which is applied to a robot, wherein the robot has a robotic arm, and the robotic arm has an end effector. The method includes: receiving a voice grabbing instruction for classroom supplies, wherein the voice grabbing instruction is used to instruct the robot to grab the classroom supplies to a first area; in response to the voice grabbing instruction, controlling the robotic arm to move the end effector to the classroom supplies, and controlling the end effector to grab the classroom supplies; controlling the robotic arm to move the end effector that grabs the classroom supplies to the first area, and controlling the end effector to place the classroom supplies in the first area.

[0007] An embodiment of the present application provides a robot control device, which includes: an instruction receiving module, which receives a voice grabbing instruction for classroom supplies, and the voice grabbing instruction is used to instruct the robot to grab the classroom supplies to a first area; wherein the robot has a robotic arm, and the robotic arm has an end effector; a grabbing control module, which is used to control the robotic arm to move the end effector to the classroom supplies in response to the voice grabbing instruction, and control the end effector to grab the classroom supplies; a placement control module, which is used to control the robotic arm to move the end effector that grabs the classroom supplies to the first area, and control the end effector to place the classroom supplies in the first area.

[0008] The present application embodiment provides a robot, wherein the robot has a mechanical arm, the mechanical arm has an end effector, and the robot includes:

[0009] A memory for storing computer executable instructions or computer programs;

[0010] The processor is used to implement the robot control method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0011] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the robot control method provided in the embodiment of the present application when executed by a processor.

[0012] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the robot control method provided by the embodiment of the present application is implemented.

[0013] The embodiments of the present application have the following beneficial effects:

[0014] By receiving voice grabbing instructions for classroom supplies, the robot is controlled to grab the designated classroom supplies and place them in the first area. On the one hand, the robot can be controlled by simple voice commands without complicated operating steps, which improves the convenience of using the robot. On the other hand, the robot is controlled to grab and place classroom supplies instead of manually, which reduces the time and energy of manual operation, improves the efficiency and quality of classroom management, and realizes the automation and intelligence of classroom management. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the architecture of the robot control system provided in the embodiment of the present application;

[0016] Figure 2 is a schematic diagram of the robot structure provided in an embodiment of the present application;

[0017] Figure 3 is a schematic diagram of the structure of an electronic device for implementing a robot control method provided in an embodiment of the present application;

[0018] Figure 4 This is a schematic diagram of the process of the robot control method provided in the embodiment of the present application. Figure 1 ;

[0019] Figure 5 is a schematic diagram of the initial state of the robot provided in an embodiment of the present application;

[0020] Figure 6 is a schematic diagram of a robot arm provided in an embodiment of the present application moving an end effector to classroom supplies;

[0021] Figure 7 is a schematic diagram of controlling the end effector provided in an embodiment of the present application to grasp classroom supplies based on an adjusted posture;

[0022] Figure 8 is a schematic diagram of an image acquisition device provided in an embodiment of the present application;

[0023] Fig. 9 is a schematic diagram of a flange provided in an embodiment of the present application;

[0024] Fig.10 This is a schematic diagram of the process of the robot control method provided in the embodiment of the present application. Figure 2 ;

[0025] Fig.11 This is a schematic diagram of the process of the robot control method provided in the embodiment of the present application. Figure 3 ;

[0026] Fig.12 This is a schematic diagram of the process of the robot control method provided in the embodiment of the present application. Figure 4 ;

[0027] Fig.13 is a schematic diagram of an end effector mechanism grabbing classroom supplies and moving to a first area provided by an embodiment of the present application;

[0028] Fig.14 It is a schematic diagram of the training process of the target recognition model provided in the embodiment of the present application;

[0029] Fig.15 It is a flowchart of a robot provided in an embodiment of the present application performing a grasping task in a classroom scenario. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0031] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0034] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0036] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0037] 1) Robot: It is an automated mechanical device that can perform various tasks through pre-programming or artificial intelligence technology. The robot can imitate human behavior and has the ability to perceive the environment, make decisions and perform actions. In the embodiment of this application, the robot is used for object grabbing and placing tasks in a classroom environment.

[0038] 2) Robotic arm: It is a part of the robot, consisting of multiple rotatable or retractable joints, and can simulate the movements of human arms. The main function of the robotic arm is to move the end effector to a specified position in order to perform operations such as grabbing and carrying. In the embodiment of the present application, the robotic arm is used to move the end effector to the position of classroom supplies, and grab and place classroom supplies in a specified area.

[0039] 3) End effector: It is a device installed at the end of the robot arm, which is used to directly physically contact and operate the environment or objects. Common end effectors include grippers, suction cups, and manipulators. In the embodiment of the present application, the end effector is used to grab and release classroom supplies.

[0040] 4) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0041] 5) Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies.

[0042] 6) Classroom supplies: refers to various items used in the classroom, such as textbooks, notebooks, pens, rulers, teaching aids, pencil cases, etc. Classroom supplies usually need to be frequently passed and sorted during the teaching process.

[0043] 7) Posture: refers to the position and posture of an object in space, usually described by six parameters: three position parameters (x, y, z) represent the coordinates of the object in three-dimensional space, and three posture parameters (roll angle (roll), pitch angle (pitch), yaw angle (yaw)) represent the rotation angle of the object. In the embodiment of the present application, the posture of the end effector includes the coordinate position (x, y, z) of the end effector in three-dimensional space, and the rotation angle (roll, pitch, yaw) of the end effector in three-dimensional space.

[0044] 8) Wheeled form: refers to the form in which the robot uses wheels as a moving mechanism. Wheeled robots usually have a higher moving speed and lower energy consumption, and are suitable for fast movement on a flat ground. In the embodiment of the present application, the robot can quickly move to the location of classroom supplies in the wheeled form.

[0045] 9) Footed morphology: refers to the form in which the robot uses legs as a moving mechanism. Footed robots have good terrain adaptability and can walk on uneven or complex ground. In the embodiment of the present application, the robot can switch to the footed morphology when encountering obstacles or uneven ground to ensure smooth arrival at the designated location.

[0046] 10) Two-finger gripper: It is a common end effector mechanism consisting of two movable fingers that grasp and release objects through opening and closing movements. The two-finger gripper has a simple structure and flexible operation, and is suitable for objects of various shapes and sizes.

[0047] The embodiments of the present application provide a robot control method, device, robot, storage medium and program product, which can control the robot to automatically complete the grabbing and placement of classroom supplies, thereby improving the efficiency and quality of classroom management.

[0048] The robot control method provided in the embodiment of the present application is applied to a robot, wherein the robot has a mechanical arm, and the mechanical arm has an end effector, wherein a voice grabbing instruction for classroom supplies is received, and the voice grabbing instruction is used to instruct the robot to grab the classroom supplies to a first area; in response to the voice grabbing instruction, the mechanical arm is controlled to move the end effector to the classroom supplies, and the end effector is controlled to grab the classroom supplies; the mechanical arm is controlled to move the end effector with the classroom supplies to the first area, and the end effector is controlled to place the classroom supplies in the first area. In this way, the robot is controlled to grab and place classroom supplies instead of manually, which reduces the time and energy of manual operation, improves the efficiency and quality of classroom management, and realizes the automation and intelligence of classroom management.

[0049] See also Figure 1 , Figure 11 is a schematic diagram of the architecture of the robot control system provided in the embodiment of the present application. Exemplarily, the robot control system 100 includes a robot 400 and a control device 200. The control device 200 and the robot 400 are two relatively independent devices. The robot 400 is connected to the control device 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The robot 400 and the control device 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application.

[0050] Robot 400 is used to, when receiving a voice grabbing instruction for classroom supplies, wherein the voice grabbing instruction is used to instruct the robot to grab the classroom supplies to the first area, respond to the voice grabbing instruction, control the robot arm to move the end effector to the classroom supplies, and control the end effector to grab the classroom supplies, control the robot arm to move the end effector grabbing the classroom supplies to the first area, and control the end effector to place the classroom supplies in the first area.

[0051] The control device 200 is used to obtain a voice grabbing instruction according to a user's voice input operation before the control device 200 controls the robot 400, and the voice grabbing instruction can instruct the robot 400 to grab classroom supplies to the first area.

[0052] The control device 200 can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0053] In some embodiments, the control device 200 is a part of the robot 400 , and the control device 200 may be disposed in the body of the robot 400 , for example, the control device 200 is an internal processor in the robot 400 .

[0054] See also Figure 2 , Figure 24 is a schematic diagram of the robot structure provided by an embodiment of the present application. The robot 400 has a mechanical arm 401, an end actuator 402 at the end of the mechanical arm, a base 403, an image acquisition device 404 and a flange 405. The base 403 is used to install and fix the robot 400. The image acquisition device 404 is mounted on the mechanical arm 401 through the flange 405. The mechanical arm 401 can be composed of multiple joints and multiple connecting rods. Taking the mechanical arm 401 as an example of six degrees of freedom, the mechanical arm 401 may include a first joint, a second joint, a third joint, a fourth joint, a fifth joint, a sixth joint, a first connecting rod, a second connecting rod, a third connecting rod, a fourth connecting rod and a fifth connecting rod. Among them, the first joint is located above the base 403, allowing the mechanical arm 401 to rotate along a vertical axis. The first connecting rod connects the first joint and the second joint, is a part of the mechanical arm, and allows the mechanical arm to move forward and backward on a horizontal plane. The second joint connects the first connecting rod and the second connecting rod, allowing the mechanical arm to rotate along a horizontal axis. The second link connects the second joint and the third joint, and is another part of the robotic arm, allowing the robotic arm to move up and down on the vertical axis. The third joint connects the second link and the third link, allowing the robotic arm to rotate along the vertical axis. The third link connects the third joint and the fourth joint, and is the last part of the robotic arm, allowing the robotic arm to move up and down on the horizontal axis. The fourth joint connects the third link and the fourth link, allowing the robotic arm to rotate along the horizontal axis. The fifth joint connects the fourth joint and the fifth joint, allowing the robotic arm to rotate along the vertical axis. The fifth link connects the fifth joint and the sixth joint, and is the last section of the robotic arm, allowing the robotic arm to move up and down on the horizontal axis. The sixth joint connects the fifth link and the sixth link, allowing the robotic arm to rotate along the horizontal axis. The end actuator 402 is connected to the sixth joint, and the end actuator can be a two-finger gripper used to perform tasks such as grasping objects and assembly.

[0055] See also Figure 3 , Figure 3 It is a schematic diagram of the structure of an electronic device for implementing a robot control method provided in an embodiment of the present application. The electronic device may be a terminal or a server. Figure 3 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 3 Various buses are labeled as bus system 440 .

[0056] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0057] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0058] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0059] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0060] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0061] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0062] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0063] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0064] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0065] In some embodiments, the robot control device provided in the embodiments of the present application can be implemented in a software manner. Figure 3 The robot control device 455 stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: an instruction receiving module 4551, a grasping control module 4552, and a placement control module 4553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0066] In other embodiments, the robot control device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the robot control method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0067] The robot control method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the robot provided in the embodiment of the present application. Figure 4 This is a schematic diagram of the process of the robot control method provided in the embodiment of the present application. Figure 1 ,like Figure 4 As shown, the method includes the following steps 101 to 103:

[0068] In step 101, a voice capture instruction for classroom supplies is received.

[0069] Among them, the voice grabbing instruction is used to instruct the robot to grab classroom supplies to the first area.

[0070] Here, the robot can receive the user's voice grabbing instructions through a built-in microphone or other voice input device. In the smart classroom scenario, the user can be a teacher or a student. The teacher or student can ask the robot to grab a classroom supply through verbal commands. Classroom supplies refer to various items used in schools or educational institutions, such as textbooks, notebooks, pens, rulers, scissors, tape, paper, blackboard erasers, chalk, experimental equipment, computer equipment, teaching aids, etc. The first area may refer to a pre-set location or area for placing classroom supplies in a classroom environment. The first area may be specially divided for the convenience of management and use, such as next to the podium, in front of the student's desk, etc. Alternatively, the first area may be an area where classroom supplies are placed after being grabbed by the user through voice grabbing instructions.

[0071] For example, a teacher can give a voice grabbing command to the robot, "pass me the textbook", in which case the first area can be a pre-set area, such as the table on the podium. A student can give a voice grabbing command to the robot, "put the pen in the pencil case", in which case the first area can be the "pen case" area obtained by the robot through parsing the voice grabbing command.

[0072] In step 102, in response to the voice grabbing instruction, the robot arm is controlled to move the end effector to the classroom supplies, and the end effector is controlled to grab the classroom supplies.

[0073] Here, see Figure 5 , Figure 5 4 is a schematic diagram of the initial state of the robot provided in the embodiment of the present application. The robot 400 is in the initial state before receiving the voice grabbing instruction for the classroom object 600. The initial state means that the robot arm 401 is in a preset initial position, which is usually a safe position near the base 403 of the robot 400. The joints of the robot arm 401 are in a default relaxed state, and the end actuator 402 (such as a gripper) is in an open state, ready for the next operation.

[0074] After the robot receives the voice grabbing instruction, it can use voice recognition technology to convert the user's voice grabbing instruction into text form, and recognize the intention of the voice grabbing instruction through the built-in multimodal large language model. It should be noted that the embodiment of the present application does not limit the model structure of the multimodal large language model. The intention recognition process of the multimodal large language model is as follows: the multimodal large language model analyzes the text content obtained after the voice grabbing instruction is converted, and extracts key information from the text content, and the key information at least includes the classroom items that need to be grabbed. The key information can also include the area where the classroom items are located, and the first area where they need to be placed. If the key information is extracted, the multimodal large language model successfully recognizes the intention, for example, the multimodal large language model recognizes that the intention is "move the chalk box from the current position to the desk". If the multimodal large language model successfully recognizes the intention of the voice grabbing instruction, the robot will continue to the next step. If the intention is not successfully recognized, the robot can request the user to repeat the instruction or provide more information. If the multimodal large language model obtains the classroom items that need to be grabbed from the voice grabbing instruction, the robot can determine the location of the classroom items based on image recognition technology. If the multimodal large language model obtains the classroom items that need to be grasped and the area where the classroom items are located from the voice grasping instructions, the robot can determine the location of the classroom supplies based on the pre-stored map data and the area.

[0075] The classroom supplies location refers to a safe position above the classroom supplies, that is, a position at a preset height above the classroom supplies. The preset height h is a predefined safety height offset to ensure that the robotic arm does not directly contact the classroom supplies. Assuming that the coordinates of the position of the classroom supplies are (x, y, z), the coordinates of the safe position of the classroom supplies are (x, y, z+h). Controlling the robotic arm to move the end effector to the classroom supplies location means that after determining the position of the classroom supplies, the optimal path for the robotic arm to move from the current position to the safe position above the classroom supplies can be determined. Control the robotic arm to move the end effector to the safe position above the classroom supplies according to the calculated optimal path, and control the end effector to grab the classroom supplies. Figure 6 It is a schematic diagram of a robotic arm provided in an embodiment of the present application moving an end effector to classroom supplies.

[0076] In some embodiments, before controlling the end effector to grab classroom supplies, the posture of the end effector can also be adjusted, and the adjusted posture is adapted to the shape of the classroom supplies. Controlling the end effector to grab classroom supplies can be achieved in the following manner: controlling the end effector to grab classroom supplies based on the adjusted posture.

[0077] Here, the pose of the end effector includes the position and attitude (i.e., direction and angle) of the end effector in three-dimensional space. Specifically, the position of the end effector in three-dimensional space is the specific coordinates of the end effector in three-dimensional space, usually represented by (x, y, z). For example, if the end effector is located at a point in the classroom, its position can be represented as (1, 2, 0.5) meters. The pose of the end effector in three-dimensional space is used to describe the direction and angle of the end effector, usually represented by a rotation matrix, Euler angles or quaternions. For example, if the end effector needs to grab classroom items at a certain angle and direction, the attitude can be represented by Euler angles (α, β, γ), which represent the rotation angles around the x-axis, y-axis and z-axis respectively. The pose of the end effector can also include the angle at which the end effector is opened to adapt to the shape of classroom supplies.

[0078] The process of obtaining the shape of classroom supplies is as follows: the robot can obtain the visual image of the classroom supplies through the image acquisition device, and recognize the position and shape of the classroom supplies based on the visual image. For example, the robot determines that the chalk box is located on the bookshelf on the right side of the desk through image recognition technology, and its shape is a rectangular parallelepiped. Alternatively, a mapping table can be pre-stored, and the mapping table includes the association relationship between each classroom supply and each shape. The robot determines the classroom supplies to be grasped from the voice grasping instruction and obtains the shape of the classroom supplies from the mapping table.

[0079] After determining the position and shape of the classroom supplies, determine the position and shape of the end effector based on the position and shape of the classroom supplies. Determine the target angle of each joint of the robotic arm based on the inverse kinematics algorithm and the position and posture. The inverse kinematics algorithm converts the position and posture into the angle value of each joint. For example, calculate the angle that each joint of the robotic arm needs to rotate so that the end effector reaches the position of (1, 2, 0.6) meters and the posture of (0, 0, 45) degrees. Control the joints of the robotic arm so that it gradually moves to the calculated angle. For example, control the joints of the robotic arm to rotate in sequence so that the end effector gradually approaches the adjusted position and posture. Figure 7 Schematic diagram of controlling the end effector to grasp classroom supplies based on the adjusted posture provided by the embodiment of the present application. Figure 7 , control the end effector 402 to descend to the position of classroom supplies based on the adjusted posture, control the end effector 402 to close, and grab the classroom items 600.

[0080] In some embodiments, the robot has an image acquisition device. Before controlling the robotic arm to move the end effector to the classroom supplies, the robot control method provided in the embodiment of the present application may further include the following steps: first, controlling the image acquisition device to acquire a visual image containing classroom supplies; then, when the distance between adjacent classroom supplies and the classroom supplies in the visual image is less than a preset distance threshold, and the grasping position of the end effector for the classroom supplies is between the adjacent supplies and the classroom supplies, it is determined that the process of the end effector grasping the classroom supplies is interfered with by the adjacent classroom supplies.

[0081] Here, the image acquisition device may be a camera. In the embodiment of the present application, the image acquisition device is a depth camera, which is composed of a pair of depth sensors, an RGB (Red, Green, Blue) sensor and an infrared projector. Figure 2 , the image acquisition device 404 is installed on the mechanical arm 401 through the flange 405. Figure 8 It is a schematic diagram of an image acquisition device provided in an embodiment of the present application. Fig. 9 405 is a schematic diagram of a flange provided in an embodiment of the present application. First, the image acquisition device 404 is installed on the flange 405 , and then the flange 405 is installed on the mechanical arm 401 .

[0082] In response to the voice capture instruction, first, the image acquisition device is controlled to capture a visual image containing classroom supplies. Then, the position and shape of the classroom supplies, as well as the position and shape of the adjacent supplies of the classroom supplies, are identified from the visual image through image processing technology (such as edge detection, deep learning model, feature extraction, etc.). Exemplarily, for classroom supplies of a first color, the position and shape of the classroom supplies can be determined based on the depth information of the classroom supplies in the visual image. For classroom supplies of a second color, the position and shape of the classroom supplies can be determined based on the color information of the classroom supplies in the visual image. The first color is an opaque color, and the second color is a transparent color. Based on the position of the classroom supplies and the position of the adjacent supplies, the distance between the classroom supplies and the adjacent supplies is determined. It is determined whether the distance between the classroom supplies and the adjacent supplies is less than a preset distance threshold. The preset distance threshold is a pre-set distance value used to determine whether the distance between the two supplies is safe enough to avoid interference or collision during the capture process. When the distance between the classroom supplies and the adjacent supplies is greater than or equal to the preset distance threshold, the adjacent supplies will not interfere with the capture process of the classroom supplies, and the end actuator can be directly controlled to capture the classroom supplies based on the adjusted posture.

[0083] When the distance between classroom supplies and adjacent supplies is less than a preset distance threshold, it is determined whether the gripping position of the end actuator for classroom supplies is between the adjacent supplies and the classroom supplies. When the gripping position of the end actuator for classroom supplies is between the adjacent supplies and the classroom supplies, the adjacent supplies may interfere with the gripping process of the classroom supplies. The position in the adjusted posture of the end actuator can be determined as the gripping position of the end actuator for classroom supplies. Use a geometric method to determine whether the gripping position is between classroom supplies and adjacent supplies. For example, the straight-line distance from the gripping position to the classroom supplies and adjacent supplies can be calculated to determine whether the gripping position is between the two. Alternatively, the vector method or plane geometry method can be used to determine whether the gripping position is within the plane area formed by the classroom supplies and the adjacent supplies to determine whether the gripping position is between the classroom supplies and the adjacent supplies.

[0084] In some embodiments, see Fig.10 In step 102, controlling the end actuator to grab classroom supplies can be achieved by following steps 1021A to 1023A, which are described in detail below.

[0085] In step 1021A, in response to the process in which the end effector grasps the classroom supplies being interfered by adjacent classroom supplies, the end effector is controlled to grasp the adjacent supplies.

[0086] Here, if it is determined that the process of the end effector grabbing classroom supplies is interfered by adjacent supplies of the classroom supplies, the end effector can be controlled to grab the adjacent supplies. The specific process of controlling the end effector to grab the adjacent supplies is similar to the process of controlling the end effector to grab the classroom supplies, and can refer to step 102 of the above embodiment, which will not be repeated.

[0087] In step 1022A, the robot arm is controlled to move the end effector to the second area, and the end effector is controlled to place the adjacent products in the second area.

[0088] The distance between the second area and the first area is greater than the first distance.

[0089] Here, the second area refers to a temporary placement area selected by the robot to avoid interference during the grasping task, and is used to temporarily place adjacent supplies so that the robot can smoothly grasp classroom supplies. The first distance refers to a preset distance value, which is used to ensure that the distance between the second area and the first area is far enough to avoid interference of adjacent supplies on the grasping and placing operations of classroom supplies. The robot arm is controlled to move the end effector to the top of the second area, and the end effector is controlled to open so that the adjacent supplies fall to the second area.

[0090] In step 1023A, the robot arm is controlled to move the end effector to the classroom supplies again, and the end effector is controlled to grab the classroom supplies.

[0091] Here, the robot arm is controlled to move the end effector to the classroom supplies again, and the end effector is lowered based on the adjusted posture to grab the classroom supplies. The specific process of moving the end effector to the classroom supplies and controlling the end effector to grab the classroom supplies can refer to step 102 of the above embodiment, which will not be described here.

[0092] The embodiment of the present application controls the robotic arm to first place adjacent supplies that may cause interference in a second area away from classroom supplies, and then grabs the classroom supplies, thereby avoiding interference from adjacent supplies, improving the success rate of grabbing, and preventing adjacent supplies from being damaged.

[0093] In some embodiments, see Fig.11 In step 102, controlling the end actuator to grab classroom supplies can be achieved by following steps 1021B to 1022B, which are described in detail below.

[0094] In step 1021B, in response to the process in which the end effector mechanism grabs the classroom supplies being interfered with by adjacent classroom supplies, the end effector mechanism is controlled to push the adjacent supplies to the third area along the first direction.

[0095] There are no classroom supplies in the first direction, and the distance between the third area and the first area is greater than the second distance.

[0096] Here, the first direction refers to a direction selected by the robot to push adjacent supplies when the robot detects that the end actuator is interfered by adjacent supplies in the process of grabbing classroom supplies. There are no classroom supplies or other obstacles in the first direction. The third area refers to a temporary area where the robot places adjacent supplies after pushing the adjacent supplies. The second distance refers to a preset distance value, which is used to ensure that the distance between the third area and the first area is far enough to avoid interference of adjacent supplies with the grabbing and placing operations of classroom supplies. The end actuator can be controlled to descend to the same height as the adjacent supplies, and the end actuator can be controlled to push along the first direction, and a visual image is collected in real time by an image acquisition device. When it is identified from the visual image that the position of the adjacent supplies is located in the third area, the pushing is stopped.

[0097] In step 1022B, the robot arm is controlled to move the end effector to the classroom supplies again, and the end effector is controlled to grab the classroom supplies.

[0098] Here, the robot arm is controlled to move the end effector to the classroom supplies again, and the end effector is lowered based on the adjusted posture to grab the classroom supplies. The specific process of moving the end effector to the classroom supplies and controlling the end effector to grab the classroom supplies can refer to step 102 of the above embodiment, which will not be described here.

[0099] The embodiment of the present application controls the robotic arm to first push adjacent supplies that may cause interference to a third area away from classroom supplies, and then grabs the classroom supplies, thereby avoiding interference from adjacent supplies, improving the success rate of grabbing, and preventing adjacent supplies from being damaged.

[0100] In some embodiments, see Fig.12 In step 102, controlling the end actuator to grab classroom supplies can be achieved by following steps 1021C to 1022C, which are described in detail below.

[0101] In step 1021C, the end effector is controlled to grasp the classroom supplies with an initial grasping force.

[0102] Here, the initial grasping force may be a preset grasping force for classroom supplies. The initial grasping force may be the same or different for different classroom supplies. For example, when the type of classroom supplies is the first type, the initial grasping force may be the first grasping force; when the type of classroom supplies is the second type, the initial grasping force may be the second grasping force, the first grasping force is less than the second grasping force, and the weight of the first type of classroom supplies is less than the weight of the second type of classroom supplies.

[0103] In step 1022C, during the process of grabbing classroom supplies, the magnitude of the grabbing force for the classroom supplies is adaptively adjusted.

[0104] Here, the end actuator has a pressure sensor. Adaptive adjustment of the grasping force for classroom supplies can be achieved in the following ways: first, in the process of grasping classroom supplies, the pressure sensor is controlled to detect the pressure value between the end actuator and the classroom supplies; then, in response to the pressure value being less than or equal to the first pressure threshold, the end actuator is controlled to increase the initial grasping force; in response to the pressure value being greater than the second pressure threshold, the end actuator is controlled to reduce the initial grasping force; the first pressure threshold is less than the second pressure threshold. The embodiment of the present application does not limit the values ​​of the first pressure threshold and the second pressure threshold, and they can be set based on actual needs.

[0105] The embodiment of the present application adaptively adjusts the gripping force. If the robot detects that the gripping force is insufficient, it will increase the gripping force. If it detects that the gripping force is too large, the robot will reduce the gripping force to ensure that the gripping force is always in an optimal state to avoid objects slipping due to insufficient gripping force or damage to objects due to excessive gripping force.

[0106] In some embodiments, after controlling the end effector to grasp the classroom supplies, in response to the grasping force used by the end effector to grasp the classroom supplies being the target grasping force, the target grasping force is recorded. In response to receiving a voice grasping instruction for the classroom supplies again, the classroom supplies are grasped based on the recorded target grasping force.

[0107] By recording and using the target grasping force, the robot can significantly improve the efficiency and accuracy of the grasping operation in the embodiment of the present application.

[0108] In step 103, the robot arm is controlled to move the end effector that grabs the classroom supplies to the first area, and the end effector is controlled to place the classroom supplies in the first area.

[0109] Here, the first area refers to the target location where classroom supplies are finally placed, which is usually a plane area. For example, the first area can be an area on the desk with a coordinate range of (2, 2, 0.7) meters to (3, 3, 0.7) meters. Fig.13 , Fig.13 It is a schematic diagram of the end effector that grabs classroom supplies and moves to the first area provided in an embodiment of the present application. Controlling the robotic arm 401 to move the end effector that grabs classroom supplies to the first area 700 means controlling the robotic arm to move the end effector that grabs classroom supplies to above the plane area where the first area 700 is located. Control the end effector to release the classroom supplies 600 so that the classroom supplies 600 fall in the first area 700. Determine the optimal path for the robotic arm from the current grasping position to the first area. Control the robotic arm to move the end effector to the first area according to the calculated optimal path. According to the type of classroom supplies (such as fragility), select a suitable placement height and speed, and control the end effector to place the classroom supplies in the first area.

[0110] In some embodiments, controlling the end actuator to place classroom supplies in the first area in step 103 can be achieved in the following way: when the type of classroom supplies is the first type, controlling the end actuator to release the classroom supplies at a second distance vertically away from the first area, so that the classroom supplies fall into the first area; when the type of classroom supplies is the second type, controlling the end actuator to release the classroom supplies at a third distance vertically away from the first area, so that the classroom supplies fall into the first area; wherein the second distance is greater than the third distance, and the fragility of the first type of classroom supplies is less than the fragility of the second type of classroom supplies.

[0111] Here, the first type of classroom supplies have lower fragility and can withstand greater impact. If the classroom supplies are of the first type, the end actuator is controlled to release the classroom supplies at a second distance vertically away from the first area, that is, the classroom supplies are allowed to fall freely from a higher position to the first area. For example, the fragility of a chalk box is lower, and the second distance can be selected as 0.1 meters. The second type of classroom supplies have higher fragility and need to be handled with care to avoid damage. If the classroom supplies are of the second type, the end actuator is controlled to release the classroom supplies at a third distance vertically away from the first area, that is, the classroom supplies are allowed to fall freely from a lower position to the first area. For example, the fragility of a glass is higher, and the third distance can be selected as 0.05 meters.

[0112] The embodiment of the present application releases classroom supplies with lower fragility at a higher position and classroom supplies with higher fragility at a lower position so that the classroom supplies fall into the first area, thereby improving the efficiency of grabbing and placing while ensuring the safety of teaching supplies.

[0113] In some embodiments, controlling the end effector to place classroom supplies in the first area in step 103 can be achieved by controlling the robotic arm to move the end effector holding the classroom supplies to the first area at a target speed.

[0114] Among them, target speed was negatively correlated with the fragility of classroom supplies.

[0115] Here, the higher the fragility of the classroom supplies, the lower the target speed. Exemplarily, the fragility of the first type of classroom supplies is less than the fragility of the second type of classroom supplies. If the classroom supplies are of the first type, the robot arm is controlled to move the end effector with the classroom supplies to the first area at the first target speed; if the classroom supplies are of the second type, the robot arm is controlled to move the end effector with the classroom supplies to the first area at the second target speed, and the first target speed is greater than the second target speed.

[0116] The embodiment of the present application reduces the target speed of the end effector for moving and grasping classroom supplies for classroom items with higher fragility, and increases the target speed of the end effector for moving and grasping classroom supplies for classroom items with lower fragility, thereby improving the placement efficiency while ensuring the safety of teaching supplies.

[0117] In some embodiments, the robot is a movable robot, including a wheeled robot and a footed robot. Before controlling the robot arm to move the end effector to the classroom supplies, the method further includes:

[0118] Control the robot to move to the classroom supplies.

[0119] During the movement of the robot, in response to the movement switching condition being met, the robot is controlled to switch from a wheeled form to a footed form, or the robot is controlled to switch from a footed form to a wheeled form.

[0120] Here, the robot is a movable robot. The wheeled form of the robot is to install movable wheels under the base, and the number of wheels is not limited. The footed form of the robot is to install legs under the base, and the number of legs is not limited, for example, it can be two or four. After determining the location of the classroom supplies, the target path for the robot to move to the location of the classroom supplies can be determined by the path planning algorithm, and the robot can be controlled to move to the location of the classroom supplies based on the target path. Then, the robot arm is controlled to move the end actuator to the classroom supplies to grab the classroom supplies. The embodiment of the present application does not specifically limit the path planning algorithm.

[0121] In the embodiment of the present application, the movement switching condition may be receiving a form switching instruction. In response to receiving the form switching instruction, the robot is controlled to switch from a wheeled form to a footed form, or from a footed form to a wheeled form. The user may give the form switching instruction to the robot by voice.

[0122] In an embodiment of the present application, the mobile switching condition may also be that the road condition in the direction of movement of the robot does not match the current form of the robot. For example, the current form of the robot is a wheeled form, and the image acquisition device that controls the robot collects the road condition image in the direction of movement of the robot in real time. If the road condition image shows that there is an obstacle ahead, or the road is an uneven road (such as a stepped road), the road condition in the direction of movement of the robot does not match the current form of the robot, and the robot is controlled to switch from the wheeled form to the footed form. The current form of the robot is a footed form, and the image acquisition device that controls the robot collects the road condition image in the direction of movement of the robot in real time. If the road condition image shows that the road ahead is a flat road, the robot is controlled to switch from the footed form to the wheeled form.

[0123] The embodiment of the present application controls the robot to switch between a foot-type form or a wheel-type form, so that the robot can be applied to various complex classroom scenarios, thereby improving the efficiency and reliability of the robot in grasping and placing classroom supplies.

[0124] In some embodiments, the robot is a fixed-station robot, which is fixed to a desk or a lectern in the classroom. The end effector is a two-finger gripper. Controlling the end effector to place classroom supplies in the first area can be achieved in the following way: controlling the two-finger gripper to place classroom supplies in the first area of ​​the desk or the lectern.

[0125] Here, see Figure 5The robot 400 is fixed on a desk or a lectern 800 through a base 403. The end effector is a two-finger gripper. When the end effector is controlled to place classroom supplies in the first area, the two fingers of the two-finger gripper can be controlled to open at a certain angle, and the classroom supplies fall into the first area. It should be noted that in actual scenarios, the robot can also be fixed on any plane or inclined surface in the classroom, such as the ground and the wall.

[0126] The embodiment of the present application controls the robot to grab the designated classroom supplies and place them in the first area by receiving voice grabbing instructions for classroom supplies. On the one hand, the robot can be controlled by simple voice instructions without complicated operating steps, thereby improving the convenience of using the robot. On the other hand, the robot is controlled to grab and place classroom supplies instead of manually, thereby reducing the time and energy of manual operation, improving the efficiency and quality of classroom management, and realizing the automation and intelligence of classroom management.

[0127] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0128] With the deep integration of information technology and education and teaching, smart classrooms are entering a period of rapid development. Smart classrooms are mainly based on classroom recording, behavior analysis, data statistics, etc., and have not truly realized AI. To this end, the present application embodiment provides a robot control method that can be applied to the smart classroom scene, and the robot control method is applied to the embodied intelligent robot in the smart classroom scene. The robot can identify the commonly used teaching aids and stationery in the classroom, and can provide teaching aids to the teacher in time when the teacher is lecturing, and help students organize stationery after class, so as to improve teaching efficiency. In the present application embodiment, a model that can identify various teaching aids and stationery (corresponding to the classroom supplies in the above embodiment) is trained based on the target detection algorithm (You Only Look Once, YOLO), and then the classroom supplies are identified and the spatial posture of the classroom supplies is described through 3D visual input, and finally the fixed-stage mechanical arm robot is controlled by the robot operating system (Robot Operating System, ROS) system to grab the classroom supplies. Instructions are given to the robot via voice input, and the intention of the voice input is understood through a multimodal large model.

[0129] First, you can choose a dozen kinds of stationery, take more than 200 pictures of the stationery, use the labeling tool (Label Img) to label all the pictures to build a data set, and finally train a target recognition model with a recognition rate of more than 95% based on the target detection algorithm (YOLO10) by continuously adjusting the parameters. In the hardware design part, a robotic arm is selected as the robot body, a depth camera is used as the visual input of the robot (corresponding to the image acquisition device in the above embodiment), and a camera bracket (flange) for installing the camera is designed, and the depth camera is installed on the robotic arm through the camera bracket. Finally, the robot is operated through the ROS system. The embodiments of this application involve hardware design, data set construction of training models, etc., and apply embodied intelligent robot technology, computer vision technology, multimodal large model application technology, model training and reasoning technology, etc.

[0130] Fig.14 It is a schematic diagram of the training process of the target recognition model provided in the embodiment of the present application.

[0131] Step 801, take pictures of classroom supplies.

[0132] Among them, you can choose more than a dozen stationery items, take more than 200 pictures of the stationery items, and complete the original data collection.

[0133] Step 802, annotate the photo.

[0134] Among them, the labeling tool (Label Img) is used to label all pictures, obtain the type of stationery contained in each photo, and complete the construction of the data set.

[0135] Step 803, training the model.

[0136] Among them, the data set is trained based on the model training method provided by YOLO10 to obtain the target recognition model. The training parameters can be the number of training rounds epochs = 100 batch size batch = 16, and the size of the input image imgsz = 640. YOLO10 is a target detection algorithm that only uses a convolutional network to directly predict the category and position of different targets. It is characterized by fast running speed but relatively low accuracy. The robot can be operated to identify the visual input image based on the target recognition model to obtain all the operable objects in the classroom, and then the ROS system is used to operate the robotic arm to grab the objects in the identified scene and place them in the specified position.

[0137] YOLO uses a deep convolutional neural network to extract image features. This network usually contains multiple convolutional layers, pooling layers, and fully connected layers. The network outputs the probability that there may be an object at each position in the image and the probability that these objects belong to each category. By setting a threshold, the existence of the object can be determined and located. The YOLO algorithm uses a separate convolutional neural network model to achieve target detection. First, the input image is converted to a fixed size, then fed into the convolutional neural network model, and finally the network prediction results are processed to obtain the detected target. Specifically, YOLO's convolutional neural network model divides the input image into an S×S grid, and then each cell is responsible for detecting the targets whose center points fall within the grid. Each cell predicts N bounding boxes and the confidence of the bounding box. The confidence includes two aspects: one is the probability that the bounding box contains the target, and the other is the accuracy of the bounding box. When the bounding box does not contain the target, the confidence that it contains the target is 0. When the bounding box contains the target, the confidence that it contains the target object is 1. The accuracy of the bounding box can be characterized by the intersection of the predicted box and the actual box. Therefore, the confidence can be defined as the confidence of containing the target × the intersection of the predicted box and the actual box. The size and position of the bounding box can be represented by 4 values, namely x, y, w, and h, where x and y are the center coordinates of the bounding box, and w and h are the width and height of the bounding box. x and y are the offset values ​​relative to the coordinate point of the upper left corner of each cell, and the unit is relative to the cell size. The predicted values ​​of w and h of the bounding box are the ratio of the width and height of the entire image, so theoretically the size of the four elements should be in the range of [0,1]. In this way, the predicted value of each bounding box actually contains 5 elements: x, y, w, h, and c, of which the first 4 represent the size and position of the bounding box, and the last value is the confidence.

[0138] After training such a target recognition model, when the robot is working, it can identify the target object through the target recognition model in the visual input image and then perform a grasping operation on the object.

[0139] Step 804: Verify whether the model meets the performance requirements.

[0140] Among them, the target recognition model can be verified based on the validation set to evaluate whether the performance of the target recognition model on unseen data meets the requirements. By calculating the normalized confusion, P curve, R curve, mean average precision, F1 score and other indicators on the validation set, the generalization ability of the target recognition model for the target detection task can be understood. Based on the performance of the validation set, the hyperparameters of the model, such as learning rate, batch size, etc., can be adjusted to optimize the performance of the target recognition model. The best model configuration is found by continuous iteration and parameter adjustment. The validation set can also be used to detect whether the target recognition model is overfitting. If the performance of the target recognition model on the training set continues to improve, but the performance on the validation set no longer improves or begins to decline, it may mean that the target recognition model begins to overfit the training data. At this time, measures such as increasing data diversity and reducing model complexity need to be taken. In the YOLO algorithm, the validation set is an important tool for evaluating the performance of the target recognition model, adjusting hyperparameters, and detecting overfitting. By using the validation set reasonably, the model can be trained and optimized more effectively to improve the accuracy and generalization ability of target detection.

[0141] Each row of the normalized confusion matrix is ​​a predicted category, and each column is a true category. The accuracy of the target recognition model for each category can also be determined based on the P curve (P_curve). The P curve is the single-category accuracy, that is, the accuracy of each category recognition when the confidence is set to a certain value. When the confidence is large, the category detection is more accurate. Only when the confidence is very large, it is judged to be a certain category, and some categories with low confidence will be missed. The R curve (R_curve) is the relationship between the recall rate and the confidence. When the confidence is set to a certain value, the probability of each category being fully checked. It can be seen that when the confidence is smaller, the category detection is more comprehensive. Mean average precision: The higher the precision, the lower the recall rate. After the model is trained, it is expected to detect all categories as much as possible under the premise of high accuracy. Therefore, the curve of the mean average precision should be close to the point (1, 1), that is, it is hoped that the area of ​​the mean average precision curve is as close to 1 as possible. The F1 score is a measurement indicator for classification problems. It is the harmonic mean of precision and recall, with a maximum of 1 and a minimum of 0. 1 is the best and 0 is the worst.

[0142] Step 805, output the model.

[0143] Among them, if the target recognition model meets the performance requirements, the target recognition model is output.

[0144] Step 806, take a photo again.

[0145] Among them, if the target recognition model does not meet the performance requirements, the stationery is selected again to take photos. It should be noted that the size of the data set reconstructed this time is the same as the size of the original data set.

[0146] In the embodiment of the present application, the robot has a six-degree-of-freedom ultra-lightweight anti-personnel robotic arm. The total weight of the integrated control cabinet and the robotic arm is only 7.2KG, the rated load reaches 5KG, the peak load can reach 9KG, and the working radius is 610mm. The lightweight feature makes the robot more suitable for smart classroom scenarios. The two-finger gripper is selected as the end effector. The two-finger gripper is an electric gripper designed and produced using a small volume and high torque linear servo drive. A linear servo drive is integrated inside the gripper. The user interface uses an RS232 / RS485 communication interface and supports the serial communication protocol (Modbus Remote Terminal Unit, Modbus RTU) protocol. It has a built-in sensitive pressure sensor. By setting different pressure thresholds, it is convenient for users to grip objects of different hardness. The simple and efficient interface control instructions allow users to quickly control the gripper. The high-quality performance makes the gripper used in service robots, teaching aids and other fields.

[0147] The depth camera is selected as the visual input of the robot, which consists of a pair of depth sensors, RGB sensors and infrared projectors. With the help of rolling image shutter and narrow field of view, it can provide high depth resolution when the object size is small and more precise measurement is required. The depth camera is mounted on the flange and the flange is mounted on the robotic arm.

[0148] The camera needs to be calibrated before the robot is used. The calibration is mainly for the external parameters of the camera. Since the camera is installed on the robot, this calibration is the eye-in-hand calibration, which involves four coordinate systems: the robot base coordinate system (base), the robot end coordinate system (gripper), the camera coordinate system (cam), and the calibration object coordinate system (target). The conversion relationship between the coordinate systems is: the transformation matrix T (target-base) from the target coordinate system to the base coordinate system, the transformation matrix T (target-cam) from the target coordinate system to the camera coordinate system, the transformation matrix T (cam-gripper) from the camera coordinate system to the robot end coordinate system, and the transformation matrix T (gripper-base) from the robot end coordinate system to the base coordinate system. According to the properties of linear algebra matrix multiplication, the following formula is obtained: T (target-base) = T (target-cam) × T (cam-gripper) × T (gripper-base). The calibration process is as follows: Step 1, place the calibration plate in a fixed position; Step 2, move the robotic arm so that the camera can clearly capture the calibration plate, record the position of the robotic arm at this time, and take photos; Step 3, repeat the second step more than 20 times and save the data for later use.

[0149] According to the above calibration steps, during the calibration process, the position relationship between the calibration plate and the base coordinate system remains unchanged, that is, T (target-base) remains unchanged. Similarly, during the calibration process, the position relationship between the camera and the end of the robotic arm remains unchanged, that is, T (cam-gripper) remains unchanged. Therefore, by transforming the above formula, we can get AX=XB, where A represents the transformation of the target object coordinates to the camera coordinate system under different postures, and B represents the transformation of the robotic arm end coordinate system to the base coordinate system under different postures. Then, T (cam-gripper) can be calculated using multiple sets of data obtained from previous shooting, and then the first formula can be used to obtain T (target-base) corresponding to each photo. At this point, the hand-eye calibration is completed.

[0150] In the embodiment of the present application, the internal algorithm of the robot can be obtained by encoding using programming language (python). Programming language Python, as an object-oriented, cross-platform, open source interpreted programming language, has the advantages of being simple and easy to use, clean and elegant, and having low learning cost. It has many open source third-party libraries, which are powerful and can greatly improve work efficiency in the scene of algorithm verification. The coding this time is mainly divided into two parts, first of all, the control of vision, i.e., the camera, and secondly, the control of the robotic arm. The camera can be controlled by the real sense camera library (pyrealsense2 library), which is an interface operation camera based on the software toolkit (Software Development Kit, SDK), and ROS is not used to control the camera, because the camera is controlled by the publish-subscribe mode of ROS, which will cause the camera to be in working mode all the time, which will cause the camera to get hot and affect the life of the camera, and the communication efficiency is low. In the embodiment of the present application, the interface is called only once to take a photo when identifying classroom supplies, and the real-time stream of the camera is not required. In realizing the grasping of the robotic arm, the motion planning of the robotic arm is the most important part, which includes kinematic forward and inverse solution, collision detection, environmental perception, and motion planning. The motion planning of common robotic arms mostly adopts the mobile path planning (MoveIt planning) provided by the ROS system. MoveIt is a motion planning library in the ROS system that integrates component packages related to mobile operations. It contains most of the functions required in motion planning. At the same time, it provides a friendly configuration and debugging interface to facilitate the initialization and debugging of the robot on the ROS system.

[0151] Fig.15 It is a flowchart of a robot provided in an embodiment of the present application performing a grasping task in a classroom scenario.

[0152] See also Fig.15, step 901, system initialization. The robot system is initialized, including configuration (config) initialization, logger initialization, ROS node initialization, camera initialization, etc. Configuration (config) includes but is not limited to fixture configuration, camera parameters, robot kinematic parameters, etc. Logger initialization is to initialize the logger, which is used to record the operating status, errors and key events of the system. As the core of the system, the ROS node is used to manage the communication and data flow between various components. Step 902, the robot returns to the initial position. The robot's robotic arm and end effector are moved to the initialization posture. Step 903, waiting for instructions. The robot begins to wait for voice capture instructions.

[0153] Step 915, voice input. The user can input voice grabbing instructions by voice. Step 916, the multimodal large prediction model recognizes the voice input. Step 917, whether the recognition is successful. If the recognition is successful, jump to step 904; if not, jump to step 915. Step 904, the robot moves to the photo shooting position. Step 905, the camera takes a photo. Step 906, model inference. The target recognition model obtained through training identifies the target to be grabbed from the photo taken by the camera (corresponding to the classroom supplies in the above embodiment). Step 907, whether it is the target to be grabbed. For any object in the photo, if the model recognizes that the object is the target to be grabbed, jump to step 908; or, for each object in the photo, if the model recognizes that each object is not the target to be grabbed, return to step 905 to retake the photo. Alternatively, you can also return to step 904 and move the robot to another position to take photos. Step 908, principal component analysis. If the target to be grasped is identified, principal component analysis (PCA) is performed using the point cloud data related to the target to be grasped acquired by the camera, and the high-dimensional data is projected to a lower dimension to retain key features. The point cloud data related to the target to be grasped is a set of points in the three-dimensional space of the target to be grasped acquired by the RGBD camera. Step 909, calculate the pose. Calculate the pose based on the result of the principal component analysis. Step 910, open the clamp. Step 911, the robot moves to the target position. The robot moves to the target position based on the pose (corresponding to the classroom supplies in the above embodiment). Step 912, close the clamp. The end actuator grasps the target to be grasped. Step 913, the robot moves to the specified position (corresponding to the first area in the above embodiment). Step 914, open the clamp. The end actuator releases the target to be grasped. Since in a smart classroom scenario, the robot may need to handle multiple targets to be grasped (for example, the robot may need to grasp different classroom supplies multiple times and place them in designated locations), after executing step 914, it can jump to step 904 so that the robot can immediately enter the preparation stage of the grasping task of the next target to be grasped after completing the placement task of one target to be grasped, thereby improving the robot's work efficiency.

[0154] The embodiment of the present application trains a model specifically for teaching scenarios and recognizes various teaching aids and stationery. When the robot is used in a classroom scenario, it can help teachers select and pass teaching aids, and can help students organize stationery, making the application of embodied intelligent robots in smart classrooms possible.

[0155] The following is a description of an exemplary structure of a robot control device 455 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 3 As shown, the software modules stored in the robot control device 455 of the memory 450 may include:

[0156] The instruction receiving module 4551 receives a voice grabbing instruction for classroom supplies, and the voice grabbing instruction is used to instruct the robot to grab the classroom supplies to the first area; wherein the robot has a robotic arm, and the robotic arm has an end effector; the grabbing control module 4552 is used to control the robotic arm to move the end effector to the classroom supplies in response to the voice grabbing instruction, and control the end effector to grab the classroom supplies; the placement control module 4553 is used to control the robotic arm to move the end effector with the classroom supplies to the first area, and control the end effector to place the classroom supplies in the first area.

[0157] In some embodiments, the robot control device 455 also includes a posture adjustment module for adjusting the posture of the end effector, so that the adjusted posture is compatible with the shape of the classroom supplies.

[0158] In some embodiments, the grasping control module 4552 is also used to control the end effector to grasp classroom supplies based on the adjusted posture.

[0159] In some embodiments, the grasping control module 4552 is also used to control the end effector to grasp the adjacent supplies in response to the interference of adjacent supplies in the process of the end effector grasping the classroom supplies; control the robotic arm to move the end effector to the second area, and control the end effector to place the adjacent supplies in the second area, and the distance between the second area and the first area is greater than the first distance; control the robotic arm to move the end effector to the classroom supplies again, and control the end effector to grasp the classroom supplies.

[0160] In some embodiments, the grasping control module 4552 is also used to control the end effector to push the adjacent supplies to a third area along a first direction in response to interference from adjacent supplies during the process of the end effector grasping classroom supplies; wherein there are no classroom supplies in the first direction, and the distance between the third area and the first area is greater than the second distance; control the robotic arm to move the end effector back to the classroom supplies, and control the end effector to grasp the classroom supplies.

[0161] In some embodiments, the robot has an image acquisition device, and the robot control device 455 also includes an image acquisition module, which is used to control the image acquisition device to acquire a visual image containing classroom supplies; when the distance between adjacent classroom supplies and the classroom supplies in the visual image is less than a preset distance threshold, and the grasping position of the end effector for the classroom supplies is between the adjacent supplies and the classroom supplies, it is determined that the process of the end effector grasping the classroom supplies is interfered by the adjacent classroom supplies.

[0162] In some embodiments, the grasping control module 4552 is also used to control the end actuator to grasp the classroom supplies with an initial grasping force; in the process of grasping the classroom supplies, the grasping force for the classroom supplies is adaptively adjusted.

[0163] In some embodiments, the robot control device 455 also includes a memory module for recording the target grasping force in response to the grasping force used by the end actuator to grasp the classroom supplies as the target grasping force; the grasping control module 4552 is also used to grasp the classroom supplies based on the recorded target grasping force in response to receiving a voice grasping instruction for classroom supplies again.

[0164] In some embodiments, the placement control module 4553 is also used to control the end actuator to release the classroom supplies at a second distance vertically away from the first area when the type of classroom supplies is the first type, so that the classroom supplies fall to the first area; when the type of classroom supplies is the second type, control the end actuator to release the classroom supplies at a third distance vertically away from the first area, so that the classroom supplies fall to the first area; wherein the second distance is greater than the third distance, and the fragility of the first type of classroom supplies is less than the fragility of the second type of classroom supplies.

[0165] In some embodiments, the placement control module 4553 is also used to control the robotic arm to move the end effector grasping the classroom supplies to the first area at a target speed; wherein the target speed is negatively correlated with the fragility of the classroom supplies.

[0166] In some embodiments, the robot control device 455 also includes a moving module, which is used to control the robot to move to classroom supplies; during the movement of the robot, in response to the movement switching conditions being met, the robot is controlled to switch from a wheeled form to a footed form, or the robot is controlled to switch from a footed form to a wheeled form.

[0167] In some embodiments, the robot is a fixed-station robot, which is fixed to a desk or a lectern in the classroom. The end effector is a two-finger gripper, and the placement control module 4553 is also used to control the two-finger gripper to place classroom supplies in the first area of ​​the desk or lectern.

[0168] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the robot control method described in the embodiment of the present application.

[0169] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will be caused to execute the robot control method provided by the present application embodiment, for example, Figure 4 The robot control method is shown.

[0170] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0171] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0172] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0173] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0174] In summary, the embodiments of the present application can realize the automatic grabbing and placement of classroom supplies by robots in classroom scenarios, which can help teachers pass teaching aids, help students organize desks and public areas, improve the efficiency and quality of classroom management, and thus improve teaching efficiency.

[0175] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A robot control method, characterized in that: Applied to a robot, the robot has a mechanical arm, the mechanical arm has an end effector, the method comprises: receiving a voice grabbing instruction for classroom supplies, wherein the voice grabbing instruction is used to instruct the robot to grab the classroom supplies to a first area; In response to the voice grabbing instruction, controlling the robot arm to move the end effector to the classroom supplies, and controlling the end effector to grab the classroom supplies; The robot arm is controlled to move the end effector that grasps the classroom supplies to the first area, and the end effector is controlled to place the classroom supplies in the first area.

2. The method according to claim 1, characterized in that Before controlling the end effector to grab the classroom supplies, the method further includes: Adjusting the posture of the end effector so that the adjusted posture matches the shape of the classroom supplies; The controlling the end effector to grab the classroom supplies comprises: The end effector is controlled to grasp the classroom supplies based on the adjusted posture.

3. The method according to claim 1 or 2, characterized in that: The controlling the end effector to grab the classroom supplies comprises: In response to the process of the end effector grabbing the classroom supplies being interfered by the adjacent supplies of the classroom supplies, the end effector is controlled to grab the adjacent supplies, or the end effector is controlled to push the adjacent supplies to a third area along a first direction; wherein the classroom supplies do not exist in the first direction, and the distance between the third area and the first area is greater than the second distance; When the end effector is controlled to grab the adjacent product, the robot arm is controlled to move the end effector to a second area, and the end effector is controlled to place the adjacent product in the second area, and the distance between the second area and the first area is greater than the first distance; The robot arm is controlled to move the end effector to the classroom supplies again, and the end effector is controlled to grab the classroom supplies.

4. The method according to claim 3, characterized in that The robot has an image acquisition device. Before controlling the robot arm to move the end effector to the classroom supplies, the method further includes: Controlling the image acquisition device to acquire a visual image containing the classroom supplies; When the distance between the adjacent supplies of the classroom supplies in the visual image and the classroom supplies is less than a preset distance threshold, and the grasping position of the end actuator for the classroom supplies is between the adjacent supplies and the classroom supplies, it is determined that the process of the end actuator grasping the classroom supplies is interfered by the adjacent supplies of the classroom supplies.

5. The method according to any one of claims 1 to 4, characterized in that: The controlling the end effector to grab the classroom supplies comprises: Controlling the end effector to grasp the classroom supplies with an initial grasping force; In the process of grabbing the classroom supplies, the magnitude of the grabbing force for the classroom supplies is adaptively adjusted.

6. The method according to claim 5, characterized in that After controlling the end effector to grab the classroom supplies, the method further includes: In response to the gripping force used by the end effector to grip the classroom supplies being a target gripping force, recording the target gripping force; The method further comprises: In response to receiving a voice grasping instruction for the classroom supplies again, the classroom supplies are grasped based on the recorded target grasping force.

7. The method according to any one of claims 1 to 6, characterized in that: The controlling the end effector to place the classroom supplies in the first area includes at least one of the following: When the type of the classroom supplies is the first type, controlling the end actuator to release the classroom supplies at a second distance vertically away from the first area, so that the classroom supplies fall to the first area; When the type of the classroom supplies is the second type, controlling the end actuator to release the classroom supplies at a third distance vertically away from the first area, so that the classroom supplies fall to the first area; wherein the second distance is greater than the third distance, and the fragility of the first type of classroom supplies is less than the fragility of the second type of classroom supplies; Controlling the robotic arm to move the end effector grasping the classroom supplies to the first area at a target speed; Wherein, the target speed is negatively correlated with the fragility of the classroom supplies.

8. The method according to any one of claims 1 to 7, characterized in that: The robot is a fixed-table robot, the robot is fixed to a desk or a lecture table in the classroom, the end effector is a two-finger gripper, and the method of controlling the end effector to place the classroom supplies in the first area includes: The two-finger gripper is controlled to place the classroom supplies in the first area of ​​the desk or lectern.

9. A robot control device, characterized in that: The device comprises: An instruction receiving module receives a voice grabbing instruction for classroom supplies, wherein the voice grabbing instruction is used to instruct the robot to grab the classroom supplies to the first area; Wherein, the robot has a mechanical arm, and the mechanical arm has an end effector; A grasping control module, for controlling the robot arm to move the end effector to the classroom supplies in response to the voice grasping instruction, and controlling the end effector to grasp the classroom supplies; The placement control module is used to control the robot arm to move the end effector that grabs the classroom supplies to the first area, and control the end effector to place the classroom supplies in the first area.

10. A robot, characterized in that: The robot has a mechanical arm, the mechanical arm has an end effector, and the robot comprises: A memory for storing computer executable instructions or computer programs; The processor is used to implement the robot control method according to any one of claims 1 to 8 when executing the computer executable instructions or computer programs stored in the memory.

Images