Environmental perception method, device, environmental perception component and robot of a robot

Abstract

The present application discloses an environmental perception method, device, environmental perception component and robot for a robot, relating to the technical field of artificial intelligence. The method includes: obtaining motion data collected by an inertial measurement unit to determine pose change information of the robot body; obtaining an environmental image captured by a first camera to determine pose information of a target object in the environmental image in the coordinate system of the first camera; obtaining a tag image captured by a second camera to determine relative pose information between a visual tag and the second camera; determining relative pose information between the first camera and the robot body; determining pose information of the robot body and relative pose information between the robot body and the target object. By converting the pose of the target object in the coordinate system of the first camera to the coordinate system of the robot body, the present application improves the stability of the environmental perception system of the robot in a vibration environment, as well as the robustness and accuracy of the robot for environmental perception.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to an environmental perception method, device, environmental perception component and robot for a robot. Background Art

[0002] The environmental perception technology of a mobile robot is to sense the surrounding environment through sensors carried by the robot itself, and process the obtained environmental data to obtain specific information about the surrounding environment. However, since the robot will inevitably vibrate during movement, the environmental data collected by the sensors will inevitably encounter problems of motion blur and perspective mutation, and it is impossible to reliably perform perception tasks and subsequent navigation, positioning and other tasks based on perception.

[0003] In related technologies, by measuring the jitter parameters of the robot vision system, an anti-shake compensation control method is used to prevent and control the jitter of the vision sensor, and the jitter of the robot vision system is eliminated in real time to keep the sensor stable and achieve smooth acquisition of environmental information.

[0004] However, the above method of adjusting the anti-shake of the sensor pose will cause a change in the relative pose between the robot body and the sensor, thereby affecting the perception function of the robot and subsequent functions such as navigation and positioning. Summary of the Invention

[0005] Embodiments of this application provide an environmental perception method, device, environmental perception component and robot for a robot. The technical solutions are as follows:

[0006] According to one aspect of the embodiments of this application, an environmental perception method for a robot is provided. The robot includes a robot body and an environmental perception component. The environmental perception component includes: a fixed platform, an inertial measurement unit, a first camera, a second camera, a visual tag and an anti-shake gimbal; wherein, the inertial measurement unit is installed on the fixed platform; the first camera is installed on the fixed platform through the anti-shake gimbal; the second camera is installed on the fixed platform; the visual tag is set on the first camera, and the visual tag is within the field of view of the second camera; the fixed platform is connected to the robot body, and the movement of the fixed platform is consistent with that of the robot body;

[0007] The method includes:

[0008] Obtain the motion data collected by the inertial measurement unit, and determine the pose change information of the robot body according to the motion data;

[0009] Obtain an environmental image captured by the first camera of the environment where the robot body is located, and determine the pose information of the target object included in the environmental image in the coordinate system of the first camera;

[0010] Obtain a label image captured by the second camera of the visual tag, and determine the relative pose information between the visual tag and the second camera according to the label image;

[0011] Determine the relative pose information between the first camera and the robot body according to the relative pose information between the visual tag and the second camera;

[0012] Determine the pose information of the robot body and the relative pose information between the robot body and the target object according to the pose change information of the robot body, the pose information of the target object in the coordinate system of the first camera, and the relative pose information between the first camera and the robot body.

[0013] According to one aspect of the embodiments of the present application, an environmental perception component is provided. The environmental perception component includes: a fixed platform, an inertial measurement unit, a first camera, a second camera, a visual tag, and an anti-shake gimbal;

[0014] The inertial measurement unit is installed on the fixed platform;

[0015] The first camera is installed on the fixed platform through the anti-shake gimbal;

[0016] The second camera is installed on the fixed platform;

[0017] The visual tag is arranged on the first camera, and the visual tag is within the field of view of the second camera.

[0018] According to one aspect of the embodiments of the present application, an environmental perception device for a robot is provided. The robot includes a robot body and an environmental perception component. The environmental perception component includes: a fixed platform, an inertial measurement unit, a first camera, a second camera, a visual tag, and an anti-shake gimbal; wherein, the inertial measurement unit is installed on the fixed platform; the first camera is installed on the fixed platform through the anti-shake gimbal; the second camera is installed on the fixed platform; the visual tag is arranged on the first camera, and the visual tag is within the field of view of the second camera; the fixed platform is connected to the robot body, and the movement of the fixed platform is consistent with that of the robot body;

[0019] The device includes:

[0020] A first determination module, configured to obtain motion data collected by the inertial measurement unit, and determine pose change information of the robot body according to the motion data;

[0021] A second determination module, configured to obtain an environmental image captured by the first camera of the environment where the robot body is located, and determine pose information of a target object included in the environmental image in the coordinate system of the first camera;

[0022] A third determination module, configured to obtain a label image captured by the second camera of the visual label, and determine relative pose information between the visual label and the second camera according to the label image;

[0023] A fourth determination module, configured to determine relative pose information between the first camera and the robot body according to the relative pose information between the visual label and the second camera;

[0024] A fifth determination module, configured to determine pose information of the robot body and relative pose information between the robot body and the target object according to the pose change information of the robot body, the pose information of the target object in the coordinate system of the first camera, and the relative pose information between the first camera and the robot body.

[0025] According to one aspect of the embodiments of the present application, a robot is provided. The robot includes a processor and a memory. A computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the above-mentioned environmental perception method of the robot.

[0026] According to one aspect of the embodiments of the present application, a computer-readable storage medium is provided. A computer program is stored in the storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned environmental perception method of the robot.

[0027] According to one aspect of the embodiments of the present application, a computer program product is provided. The computer program product includes a computer program, and the computer program is loaded and executed by a processor to implement the above-mentioned environmental perception method of the robot.

[0028] The technical solutions provided by the embodiments of the present application can bring the following beneficial effects:

[0029] By using a second camera to detect the pose change of a visual tag, a coordinate transformation link from the first camera to the robot body is formed, and the pose of the target object in the environment image collected by the first camera in the first camera coordinate system is transformed to the coordinate system of the robot body, so as to obtain the pose of the target object in the coordinate system of the robot body; and the motion data of the robot is obtained through an inertial measurement unit, and a motion model of the robot is established, so that the pose of the robot body at different times can be obtained; according to the pose of the target object in the coordinate system of the robot body and the pose of the robot body at the current moment, the positioning information of the robot body in the environment at the current moment can be determined. This avoids the problem of motion blur caused by the vibration generated during the movement of the robot during the acquisition of the environment image, as well as the problem of the uncertain relative pose between the first camera and the robot body in a vibrating environment, improves the stability of the environmental perception component to determine the relative pose between the target object and the robot body, and also improves the robustness and accuracy of the robot's environmental perception in a vibrating environment. Description of the Drawings

[0030] Figure 1 is a schematic diagram of an environmental perception component provided by an embodiment of the present application;

[0031] Figure 2 is a schematic diagram of an environmental perception component provided by another embodiment of the present application;

[0032] Figure 3 is a flowchart of an environmental perception method of a robot provided by an embodiment of the present application;

[0033] Figure 4 is a schematic diagram of a robot provided by an embodiment of the present application;

[0034] Figure 5 is a schematic diagram of a visual tag provided by an embodiment of the present application;

[0035] Figure 6 is a schematic diagram of a coordinate transformation link provided by an embodiment of the present application;

[0036] Figure 7 is a specific flowchart of an environmental perception method of a robot provided by an embodiment of the present application;

[0037] Figure 8 is a block diagram of an environmental perception device of a robot provided by an embodiment of the present application;

[0038] Figure 9 is a block diagram of the structure of a computer device provided by an embodiment of the present application. Detailed Embodiments

[0039] To make the objectives, technical solutions and advantages of this application more clear, the following will further describe the embodiments of this application in detail with reference to the accompanying drawings.

[0040] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning and decision-making.

[0041] Artificial intelligence technology is an interdisciplinary subject, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0042] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields. For example, common ones include smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless, autonomous driving, drones, robots, smart healthcare, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0043] The technical solution of this application mainly relates to robot technology in artificial intelligence technology, and mainly relates to robot environmental perception. A robot is a mechatronic device that combines mechanical transmission and modern microelectronics technology and can imitate a certain skill of a human. Robots are developed on the basis of electronics, machinery and information technology. A robot does not necessarily have to look like a human. As long as it can autonomously complete the tasks and commands given to it by humans, it belongs to the members of the robot family. A robot is an automated machine that has some intelligent capabilities similar to humans or living things, such as perception ability, planning ability, motion ability and cooperation ability. It is an automated machine with high flexibility. With the development of computer technology and artificial intelligence technology, robots have been greatly improved in terms of function and technical level. Mobile robots and technologies such as the vision and touch of robots are typical representatives.

[0044] Please refer to Figure 1, which shows a schematic diagram of an environment perception component provided by an embodiment of the present application. The environment perception component includes: a fixed platform 107, an inertial measurement unit 102, a first camera 105, a second camera 101, a visual tag 104, and an anti-shake gimbal 106.

[0045] The fixed platform 107 is a carrier for carrying other elements of the environment perception component, and other elements can be installed on the fixed platform 107. The installation of the environment perception component is achieved by installing the fixed platform 107 on a moving object.

[0046] The inertial measurement unit 102 is installed on the fixed platform 107. The inertial measurement unit 102 is a device for measuring the angular velocity and acceleration of a moving object during movement, and the pose of the moving object can be calculated by measuring the angular velocity and acceleration of the moving object in three-dimensional space.

[0047] In some embodiments, as Figure 1 shown, the environment perception component further includes a first shock absorber column 103, and the inertial measurement unit 102 is installed on the fixed platform 107 through the first shock absorber column 103. The first shock absorber column 103 is used to reduce the vibration generated by the inertial measurement unit 102 mounted on it due to the movement of the moving object when the moving object moves, and can effectively control the vibration of the inertial measurement unit 102 at a certain frequency, improving the robustness of the measurement of the inertial measurement unit 102.

[0048] The first camera 105 is installed on the fixed platform 107 through the anti-shake gimbal 106. The anti-shake gimbal 106 keeps the position of the first camera 105 fixed on it unchanged relative to the fixed platform 107 through the relative movement of the gimbal and the gimbal bracket. For example, when the gimbal moves forward, the gimbal bracket moves backward relatively, and when the gimbal moves left, the gimbal bracket moves right relatively, and the attitude of the first camera 105 mounted on it is ensured to be stable through motion control compensation. The first camera 105 is used to capture images of the environment where the moving object itself is located.

[0049] In some embodiments, the first camera 105 is a color camera with a depth sensor, and can simultaneously capture depth images and color images. Compared with the pictures taken by an ordinary color camera that can record all objects within the camera's view angle, the depth images taken by the first camera 105 can detect the depth distance of the shooting space. Through the data obtained from the depth images taken by the first camera 105, the distance between each point in the image and the camera of the first camera 105 can be known, and together with the two-dimensional coordinates of these points in the two-dimensional space of the color image, the three-dimensional space coordinates of each point in the coordinate system of the first camera 105 can be calculated.

[0050] The second camera 101 is mounted on the fixed stage 107, and the second camera 101 can be used to capture a tag image containing the visual tag 104.

[0051] In some embodiments, the second camera 101 is a global shutter camera, which can expose the entire scene simultaneously at the same time, and can capture the entire image area in one shot, and can clearly capture fast-moving objects in the scene.

[0052] In some embodiments, the environmental perception component further includes a second shock absorber column 108, and the second camera 101 can be mounted on the inertial measurement unit 102 through the second shock absorber column 108. As Figure 1 shown, the second shock absorber column 108 can enable the second camera 101 mounted thereon to reduce the vibration generated by the movement of the object. Then, according to the first shock absorber column 103, the vibration of the inertial measurement unit 102 at a certain frequency can be reduced, so as to reduce the noise of the information collected by the sensor. By mounting the second camera 101 on the inertial measurement unit via the second shock absorber column 108, the only components directly connected to the fixed stage 107 are the first shock absorber column 103 and the anti-shake pan-tilt 106, which can reduce the area occupied by the components on the fixed stage 107, thereby reducing the area size of the fixed stage 107, that is, reducing the occupied area of the environmental perception component.

[0053] In some embodiments, the environmental perception component further includes a third shock absorber column 109, and the second camera 101 is mounted on the fixed stage 107 through the third shock absorber column 109. As Figure 2 shown, the second shock absorber column 108 can enable the second camera 101 mounted thereon to reduce the vibration generated by the movement of the object, so as to keep the relative pose between the second camera 101 and the fixed stage 107 as unchanged as possible. By directly mounting the second camera 101 on the fixed stage 107 via the third shock absorber column 109, the transformation relationship from the coordinate system of the second camera 101 to the coordinate system of the moving object body can be directly calculated subsequently, and the calculation steps are more convenient.

[0054] The visual tag 104 is disposed on the first camera 105, and the visual tag 104 is within the field of view of the second camera 101. The visual tag 104 can be pasted on the first camera 105, and it is a tag tool for the second camera 101 to perform positioning and recognition on the first camera 105. The second camera 101 can determine the pose of the first camera 105 by detecting the pose of the visual tag 104. The visual tag 104 is always within the field of view of the second camera 101, and can be as Figure 1 and Figure 2As shown in the figure, if the shooting direction of the first camera 105 is the first direction, the visual tag 104 is pasted on the back of the first camera 105, that is, in the opposite direction of the first direction of the first camera 105. The shooting direction of the second camera 101 is also the first direction, and it can face the visual tag 104 directly. At the same time, the pose of the second camera 101 can be adjusted according to the height of the visual tag 104 to ensure that the visual tag 104 can always be within the field of view of the second camera 101. The visual tag 104 can be an AprilTag, a Matrix tag, or an ARTag. This application does not make specific limitations on this.

[0055] Regarding the environmental perception method based on the environmental perception component, reference can be made to the following method embodiments.

[0056] The technical solution provided by this application uses the second camera to detect the pose change of the visual tag, and converts the pose of the target object in the environmental image collected by the first camera from the first camera coordinate system to the coordinate system of the second camera; and obtains the motion data of the moving object through the inertial measurement unit, constructs the coordinate system of the moving object body, so as to obtain the pose of the moving object at different times; converts the pose of the target object in the second camera coordinate system to the coordinate system of the moving object body, and obtains the relative pose between the moving object body and the target object according to the pose of the moving object body. It avoids the uncertainty problem of the relative pose between each component of the environmental perception component and the moving object body in a vibrating environment, and improves the stability of the environmental perception component to determine the relative pose between the target object and the moving object body.

[0057] In addition, by using shock-absorbing columns, the vibration impact on the inertial measurement unit and the second camera installed on the object during movement can be suppressed, improving the accuracy of the motion data collected by the inertial measurement unit and the clarity of the tag images collected by the second camera.

[0058] In addition, by setting the second camera as a global shutter camera, the entire scene can be exposed simultaneously at the same time, and the entire image area can be captured at one time, and the fast-moving objects in the scene can be clearly captured, thus ensuring the clarity of the tag images it captures.

[0059] In addition, using an anti-shake gimbal can perform motion control compensation by controlling the gimbal bracket of the anti-shake gimbal to keep the pose of the first camera installed on it stable, so that the first camera can smoothly collect environmental images, avoiding the problem of motion blur caused by the vibration generated during the movement of the moving object to the collection of environmental images.

[0060] Please refer to Figure 3 , which shows the flowchart of the environmental perception method of the robot provided by an embodiment of this application. As Figure 4As shown, the robot 40 includes a robot body 420 and an environment perception component 410. For the introduction of the environment perception component, refer to the above embodiments. The method may include at least one of the following steps 210 to 250:

[0061] Step 210, obtain the motion data collected by the inertial measurement unit, and determine the pose change information of the robot body according to the motion data.

[0062] The motion data includes angular velocity data and acceleration data during the robot's movement. Refer to Figure 7 , the motion state of the robot can be monitored in real time through the inertial measurement unit, and the real-time motion data of the robot during the movement can be obtained. According to the obtained motion data, a motion model of the robot can be established to calculate and predict the position of the robot body and its possible pose changes. The motion model of the robot refers to a formula that can calculate the position and pose changes of the robot based on the motion data of the robot, that is, input the current motion data of the robot, and the coordinates of the robot in its body coordinate system can be obtained, and this coordinate can represent the specific position and specific pose of the robot body.

[0063] Step 220, obtain the environment image obtained by the first camera shooting the environment where the robot body is located, and determine the pose information of the target object included in the environment image in the coordinate system of the first camera.

[0064] The first camera is installed on the fixed platform through an anti-shake gimbal. Based on its own physical characteristics, the anti-shake gimbal can perform motion control compensation by controlling the gimbal bracket to keep the pose of the first camera stable, so that the first camera can smoothly collect environmental information. Refer to Figure 7 , the first camera shoots the environment where the robot body is located, thereby obtaining an environment image, which can also be called an environment image frame. The environment image contains a target object, and the target object can be a scene object or a human object. By identifying and detecting the currently captured environment image, the target object in the environment image can be extracted, and the pose of the target object in the current frame in the coordinate system of the first camera can be determined. This pose can be expressed as (R j , p j ), where R j and p jIt is the rotation matrix and position vector of the target object in the coordinate system of the first camera. The rotation matrix is used to represent the pose of the target object in the coordinate system of the first camera, and the position vector is used to represent the position of the target object in the coordinate system of the first camera. If the poses of the same static target object in the coordinate system of the first camera in two adjacent frames are matched and associated, the motion state of the first camera between the two adjacent frames can be calculated, and this motion state can be expressed as (R, t). Since the first camera is mounted on a fixed platform through an anti-shake gimbal, the motion state of the robot body in the robot coordinate system can be obtained through the coordinate transformation in step 230 below.

[0065] Step 230: Obtain the label image obtained by the second camera shooting the visual label, and determine the relative pose information between the visual label and the second camera according to the label image.

[0066] The visual label is set on the first camera, so the pose change of the first camera can be judged according to the pose change of the visual label. And the visual label is within the field of view of the second camera, so the image content of the visual label must be included in the label image obtained by the second camera shooting the environment with a fixed field of view.

[0067] In some embodiments, obtain the corner point information of the visual label in the label image, and the corner point information is used to identify and determine the visual label; determine the pose information of the visual label in the coordinate system of the visual label according to the corner point information of the visual label; according to the pose information of the visual label in the coordinate system of the visual label and the physical distance between the visual label and the second camera, determine the relative pose information between the visual label and the second camera. The relative pose information between the visual label and the second camera includes the transformation relationship between the coordinate system of the visual label and the coordinate system of the second camera.

[0068] The corner point information refers to the intersection points formed at the junctions of different color blocks in the visual label or the points on the formed junction lines, such as Figure 5 the hollow dot 501 in. Refer to Figure 7, by detecting the corner information of the captured label image, the distribution of different color blocks in the label image can be determined. According to the distribution of different color blocks therein and referring to the distribution of different color blocks of all visual labels in the visual label database, the image part of the visual label can be identified and extracted therefrom. Encode the color block distribution of the visual label, query the visual label database to obtain the true physical size of the current visual label, form the correspondence between the corner coordinates of the visual label in the coordinate system of the label image and the physical coordinates of the true visual label, and solve the homography transformation matrix between the corresponding points of the visual label in the label image and the true visual label according to the pre-calibrated second camera internal parameters, and calculate the position and pose of the visual label in the second camera coordinate system. And according to the true physical size of the visual label, determine the pose of the visual label in the coordinate system centered on it, so that the transformation relationship between the coordinate system of the visual label and the coordinate system of the second camera, as well as the relative pose between the visual label and the second camera can be determined.

[0069] Step 240, determine the relative pose information between the first camera and the robot body according to the relative pose information between the visual label and the second camera.

[0070] In some embodiments, according to the transformation relationship between the coordinate system of the first camera and the coordinate system of the visual label, the transformation relationship between the coordinate system of the visual label and the coordinate system of the second camera, and the transformation relationship between the coordinate system of the second camera and the coordinate system of the robot body, determine the relative pose information between the first camera and the robot body; wherein, the relative pose information between the first camera and the robot body includes the transformation relationship between the coordinate system of the first camera and the coordinate system of the robot body.

[0071] Optionally, the transformation relationship between the coordinate system of the first camera and the coordinate system of the visual label, and the change relationship between the coordinate system of the second camera and the coordinate system of the robot body in the above embodiments can be obtained by off-line calibration. The transformation relationship between the coordinate system of the visual label and the coordinate system of the second camera can be obtained by referring to Step 230.

[0072] In some embodiments, the transformation relationship between the coordinate system of the second camera and the coordinate system of the robot body can be determined according to the transformation relationship between the coordinate system of the second camera and the coordinate system of the inertial measurement unit, and the transformation relationship between the coordinate system of the inertial measurement unit and the coordinate system of the robot body.

[0073] Optionally, the transformation relationships between the coordinate systems of the first camera and the visual tag, between the coordinate systems of the second camera and the inertial measurement unit, and between the coordinate system of the inertial measurement unit and the robot body in the above embodiments can be obtained through offline calibration. The transformation relationship between the coordinate system of the visual tag and the coordinate system of the second camera can be obtained with reference to step 230.

[0074] Reference Figure 6 , which shows a schematic diagram of the coordinate transformation link of the environmental perception method of the environmental perception component. According to the transformation relationships between the above coordinate systems, a coordinate transformation link from the coordinate system of the first camera to the coordinate system of the robot body can be formed This transformation link includes the transformation from the coordinate system C of the first camera to the coordinate system T of the visual tag Then the transformation from the coordinate system T of the visual tag to the coordinate system G of the second camera Then the transformation from the coordinate system G of the second camera to the coordinate system I of the inertial measurement unit Finally, the transformation from the coordinate system I of the inertial measurement unit to the coordinate system R of the robot body Through this transformation link, the pose of the target object in the coordinate system of the first camera can be converted into the pose of the target object in the coordinate system of the robot body, so as to eliminate the influence of the vibration generated during the movement of the robot on the relative pose between the first camera and the robot body.

[0075] Step 250, determine the pose information of the robot body and the relative pose information between the robot body and the target object according to the pose change information of the robot body, the pose information of the target object in the coordinate system of the first camera, and the relative pose information between the first camera and the robot body.

[0076] In some embodiments, determine the pose information of the robot body at the current moment according to the pose information of the robot body at the previous moment and the pose change information of the robot body.

[0077] The motion model established through the motion data obtained by the inertial measurement unit can output the poses of the robot body at different moments. Calculate the pose change of the robot body over time, so as to obtain the pose change law of the robot. According to the pose of the robot body at the previous moment, the pose of the robot body at the current moment can be determined.

[0078] In addition, according to this coordinate transformation link And combined with the motion state (R, t) of the first camera between two adjacent frames obtained in step 220, the pose change of the robot body between two adjacent frames can be obtained, and its calculation method is For example, It can be represented as a 4×4 homogeneous transformation matrix for the transformation from coordinate system 2 to coordinate system 1, which includes rotation information R and translation information t. The motion data is obtained from the inertial measurement unit with a high frequency, but the calculated pose data of the robot has poor accuracy. While the pose information of the robot is obtained from the coordinate transformation link with a low frequency, but the obtained pose data has high accuracy. Therefore, it can refer to Figure 7 , and fuse the motion data of the inertial measurement unit and the pose obtained from the coordinate transformation link to obtain a pose with higher accuracy for the optimized robot. Since the inertial measurement unit obtains data with a high frequency, the pose of the robot can first refer to the pose obtained from the motion model of the robot established by the motion data of the inertial measurement unit. When the pose of the coordinate transformation link is calculated, the pose of the robot is updated to the pose of the coordinate transformation link. Before the pose result of the coordinate transformation link is calculated next time, the pose obtained from the motion model of the robot can continue to be referred to.

[0079] In some embodiments, according to the pose information of the target object in the coordinate system of the first camera and the transformation relationship between the coordinate system of the first camera and the coordinate system of the robot body, the pose information of the target object in the coordinate system of the robot body is determined. Refer to Figure 6 In the coordinate transformation link in , according to the pose of the target object in the coordinate system of the first camera, the pose of the target object in the coordinate system of the first camera can be transformed into the pose of the target object in the coordinate system of the robot body via the coordinate transformation link.

[0080] According to the pose information of the target object in the coordinate system of the robot body and the pose information of the robot body at the current moment, the pose information of the robot body in the environment at the current moment is determined. Refer to Figure 7 , according to the poses of the target object and the robot body at the same coordinate, that is, in the coordinate system of the robot body, at the current moment, through the data fusion of multiple sensors, the relative pose between the robot body and the target object at the current moment can be calculated.

[0081] The technical solution provided by this application forms a coordinate conversion link from the first camera to the robot body by using the second camera to detect the pose change of the visual tag, and converts the pose of the target object in the environmental image collected by the first camera in the coordinate system of the first camera to the coordinate system of the robot body, so as to obtain the pose of the target object in the coordinate system of the robot body; and obtains the motion data of the robot through the inertial measurement unit, and establishes a motion model of the robot, so that the pose of the robot body at different times can be obtained; according to the pose of the target object in the coordinate system of the robot body and the pose of the robot body at the current moment, the relative pose between the robot body and the target object at the current moment can be determined. This avoids the problem of motion blur caused by the vibration generated during the movement of the robot to the acquisition of the environmental image, as well as the problem of the uncertain relative pose between the first camera and the robot body in a vibrating environment, improves the stability of the environmental perception component to determine the relative pose between the target object and the robot body, and also improves the robustness and accuracy of the robot's environmental perception in a vibrating environment.

[0082] In some embodiments, this application also provides a robot, which includes a robot body and an environmental perception component. The fixed platform in the environmental perception component is connected to the robot body and moves in unison with the robot body.

[0083] Reference Figure 4 , in the robot 40, the installation between the environmental perception component 410 and the robot body 420 is completed by installing the fixed platform of the environmental perception component 410 on the robot body 420. The fixed platform is fixedly installed with the robot body 420, so it can be determined that the movement of the fixed platform is consistent with that of the robot body 420.

[0084] For the introduction and description of the environmental perception component and the corresponding environmental perception method, reference can be made to the above embodiments.

[0085] The technical solution provided by this application installs an environmental perception component that can accurately perceive the environment in a vibrating environment on the robot body, enabling the robot to improve its recognition and perception functions of the surrounding environment, and thus realizing subsequent functions such as navigation and positioning.

[0086] The following is an embodiment of the device of this application, which can be used to execute the method embodiment of this application. For details not disclosed in the device embodiment of this application, please refer to the method embodiment of this application.

[0087] Please refer to Figure 8, which shows a block diagram of an environmental perception device of a robot provided by an embodiment of the present application. The robot includes a robot body and an environmental perception component, and the environmental perception component can refer to the description of the above embodiment. The device 800 may include: a first determination module 810, a second determination module 820, a third determination module 830, a fourth determination module 840, and a fifth determination module 850.

[0088] The first determination module 8610 is configured to obtain motion data collected by the inertial measurement unit and determine pose change information of the robot body according to the motion data.

[0089] The second determination module 820 is configured to obtain an environmental image obtained by the first camera photographing the environment where the robot body is located and determine pose information of a target object included in the environmental image in the coordinate system of the first camera.

[0090] The third determination module 830 is configured to obtain a tag image obtained by the second camera photographing the visual tag and determine relative pose information between the visual tag and the second camera according to the tag image.

[0091] The fourth determination module 840 is configured to determine relative pose information between the first camera and the robot body according to the relative pose information between the visual tag and the second camera.

[0092] The fifth determination module 850 is configured to determine pose information of the robot body and relative pose information between the robot body and the target object according to the pose change information of the robot body, the pose information of the target object in the coordinate system of the first camera, and the relative pose information between the first camera and the robot body.

[0093] In some embodiments, the relative pose information between the visual tag and the second camera includes a transformation relationship between the coordinate system of the visual tag and the coordinate system of the second camera. The fourth determination module 840 is configured to determine relative pose information between the first camera and the robot body according to the transformation relationship between the coordinate system of the first camera and the coordinate system of the visual tag, the transformation relationship between the coordinate system of the visual tag and the coordinate system of the second camera, and the transformation relationship between the coordinate system of the second camera and the coordinate system of the robot body; wherein, the relative pose information between the first camera and the robot body includes a transformation relationship between the coordinate system of the first camera and the coordinate system of the robot body.

[0094] In some embodiments, the third determination module 830 is configured to: obtain corner point information of the visual tag in the tag image, where the corner point information is used to identify and determine the visual tag; determine pose information of the visual tag in the coordinate system of the visual tag according to the corner point information of the visual tag; and determine relative pose information between the visual tag and the second camera according to the pose information of the visual tag in the coordinate system of the visual tag and the physical distance between the visual tag and the second camera.

[0095] In some embodiments, the fifth determination module 850 is configured to: determine the pose information of the robot body at the current moment according to the pose information of the robot body at the previous moment and the pose change information of the robot body; determine the pose information of the target object in the coordinate system of the robot body according to the pose information of the target object in the coordinate system of the first camera and the transformation relationship between the coordinate system of the first camera and the coordinate system of the robot body; and determine the pose information of the robot body in the environment at the current moment according to the pose information of the target object in the coordinate system of the robot body and the pose information of the robot body at the current moment.

[0096] In some embodiments, the transformation relationship between the coordinate system of the second camera and the coordinate system of the robot body is determined according to the transformation relationship between the coordinate system of the second camera and the coordinate system of the inertial measurement unit and the transformation relationship between the coordinate system of the inertial measurement unit and the coordinate system of the robot body.

[0097] The technical solution provided by this application forms a coordinate conversion link from the first camera to the robot body by using the second camera to detect the pose change of the visual tag, and converts the pose of the target object in the environment image collected by the first camera in the coordinate system of the first camera to the coordinate system of the robot body, so as to obtain the pose of the target object in the coordinate system of the robot body; and obtains the motion data of the robot through the inertial measurement unit and establishes a motion model of the robot, so that the pose of the robot body at different moments can be obtained; according to the pose of the target object in the coordinate system of the robot body and the pose of the robot body at the current moment, the relative pose between the robot body and the target object at the current moment can be determined. This avoids the problem of motion blur caused by the vibration generated during the movement of the robot to the acquisition of the environment image, and the problem of uncertainty of the relative pose between the first camera and the robot body in a vibrating environment, improves the stability of the environmental perception component to determine the relative pose between the target object and the robot body, and also improves the robustness and accuracy of the robot's environmental perception in a vibrating environment.

[0098] Please refer to Figure 9, which shows a structural block diagram of a robot 900 provided by an embodiment of the present application.

[0099] Generally, the computer device 900 includes: a processor 901 and a memory 902.

[0100] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may further include an AI processor, and the AI processor is used to process computational operations related to machine learning.

[0101] The memory 902 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store a computer program, and the computer program is configured to be executed by one or more processors to implement the above-mentioned environment perception method of the robot.

[0102] Those skilled in the art can understand that Figure 9 the structure shown in does not constitute a limitation on the computer device 900, and it may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements.

[0103] In a schematic embodiment, a computer-readable storage medium is further provided. A computer program is stored in the storage medium. When the computer program is executed by a processor of a computer device, the above-mentioned environment perception method of the robot is implemented. Optionally, the above-mentioned computer-readable storage medium may be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0104] In an exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned environment perception method of the robot.

[0105] It should be noted that the information and data involved in this application (including but not limited to user identity information, identity tags, travel data, travel information, etc.) are all obtained with the authorization of the user or with the full authorization of all parties, and the collection, use, and processing of the relevant information and data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0106] It should be understood that "a plurality of" mentioned herein means two or more. " / ", describing the association relationship of associated objects, indicates that three relationships may exist. For example, A and / or B may indicate: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. In addition, the step numbers described in this article only exemplarily show a possible execution sequence between steps. In some other embodiments, the above steps may not be executed in the order of the numbers. For example, two steps with different numbers are executed simultaneously, or two steps with different numbers are executed in the reverse order of the illustration. The embodiments of the present application do not limit this.

[0107] The above are only exemplary embodiments of the present application, and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An environmental perception method for a robot, characterized in that The robot includes a robot body and an environment perception component. The environment perception component includes: a fixed platform, an inertial measurement unit, a first camera, a second camera, a visual tag, and an anti-shake pan-tilt head; wherein, the inertial measurement unit is installed on the fixed platform; the first camera is installed on the fixed platform through the anti-shake pan-tilt head; the second camera is installed on the fixed platform; the visual tag is set on the first camera, and the visual tag is within the field of view of the second camera; the fixed platform is connected to the robot body, and the movement of the fixed platform is consistent with that of the robot body; the anti-shake pan-tilt head is used to perform motion control compensation by controlling the pan-tilt head bracket to keep the pose of the first camera stable; The method includes: Obtaining motion data collected by the inertial measurement unit, and determining the pose change information of the robot body according to the motion data; Obtaining an environmental image obtained by the first camera photographing the environment where the robot body is located, and determining the pose information of the target object included in the environmental image in the coordinate system of the first camera; Obtaining a tag image obtained by the second camera photographing the visual tag, and determining the relative pose information between the visual tag and the second camera according to the tag image; Determining the relative pose information between the first camera and the robot body according to the relative pose information between the visual tag and the second camera; Determining the pose information of the robot body and the relative pose information between the robot body and the target object according to the pose change information of the robot body, the pose information of the target object in the coordinate system of the first camera, and the relative pose information between the first camera and the robot body.

2. The method according to claim 1, wherein The relative pose information between the visual tag and the second camera includes the transformation relationship between the coordinate system of the visual tag and the coordinate system of the second camera; The determining the relative pose information between the first camera and the robot body according to the relative pose information between the visual tag and the second camera includes: Determining the relative pose information between the first camera and the robot body according to the transformation relationship between the coordinate system of the first camera and the coordinate system of the visual tag, the transformation relationship between the coordinate system of the visual tag and the coordinate system of the second camera, and the transformation relationship between the coordinate system of the second camera and the coordinate system of the robot body; wherein, the relative pose information between the first camera and the robot body includes the transformation relationship between the coordinate system of the first camera and the coordinate system of the robot body.

3. The method according to claim 2, wherein The determining the relative pose information between the visual tag and the second camera according to the tag image includes: Obtaining the corner point information of the visual tag in the tag image, and the corner point information is used to identify and determine the visual tag; Determining the pose information of the visual tag in the coordinate system of the visual tag according to the corner point information of the visual tag; Determine the relative pose information between the visual tag and the second camera based on the pose information of the visual tag in the coordinate system of the visual tag and the physical distance between the visual tag and the second camera.

4. The method according to claim 2, wherein The determining of the pose information of the robot body and the relative pose information between the robot body and the target object according to the pose change information of the robot body, the pose information of the target object in the coordinate system of the first camera, and the relative pose information between the first camera and the robot body includes: Determine the pose information of the robot body at the current moment according to the pose information of the robot body at the previous moment and the pose change information of the robot body; Determine the pose information of the target object in the coordinate system of the robot body according to the pose information of the target object in the coordinate system of the first camera and the transformation relationship between the coordinate system of the first camera and the coordinate system of the robot body; Determine the pose information of the robot body in the environment at the current moment according to the pose information of the target object in the coordinate system of the robot body and the pose information of the robot body at the current moment.

5. The method according to claim 2, wherein The transformation relationship between the coordinate system of the second camera and the coordinate system of the robot body is determined according to the transformation relationship between the coordinate system of the second camera and the coordinate system of the inertial measurement unit and the transformation relationship between the coordinate system of the inertial measurement unit and the coordinate system of the robot body.

6. An environmental perception component, characterized in that, The environment perception component includes: a fixed platform, an inertial measurement unit, a first camera, a second camera, a visual tag, and an anti-shake gimbal; The inertial measurement unit is installed on the fixed platform; The first camera is installed on the fixed platform through the anti-shake gimbal; the anti-shake gimbal is used to perform motion control compensation by controlling the gimbal bracket to keep the pose of the first camera stable; The second camera is installed on the fixed platform; The visual tag is set on the first camera, and the visual tag is within the field of view of the second camera.

7. The environmental perception component according to claim 6, wherein The environment perception component further includes a first shock absorber, and the inertial measurement unit is installed on the fixed platform through the first shock absorber.

8. The environmental perception component according to claim 6, wherein The first camera is a color camera with a depth sensor.

9. The environmental perception component according to claim 6, wherein The second camera is a global shutter camera.

10. The environment perception component according to claim 6, wherein The environment perception component further includes a second shock absorber, and the second camera is installed on the inertial measurement unit through the second shock absorber to realize the installation of the second camera on the fixed platform; Or, The environment perception component further includes a third shock absorber, and the second camera is installed on the fixed platform through the third shock absorber.

11. A robot, characterized in that, The robot includes a robot body and the environment perception component according to any one of claims 6 to 10, wherein the fixed platform in the environment perception component is connected to the robot body, and the movement of the fixed platform is consistent with that of the robot body.

12. An environmental perception device for a robot, characterized in that, The robot includes a robot body and an environment perception component. The environment perception component includes: a fixed platform, an inertial measurement unit, a first camera, a second camera, a visual tag, and an anti-shake pan-tilt head. Among them, the inertial measurement unit is installed on the fixed platform; the first camera is installed on the fixed platform through the anti-shake pan-tilt head; the second camera is installed on the fixed platform; the visual tag is set on the first camera, and the visual tag is within the field of view of the second camera; the fixed platform is connected to the robot body, and the movement of the fixed platform is consistent with that of the robot body; the anti-shake pan-tilt head is used to perform motion control compensation by controlling the pan-tilt head bracket to keep the pose of the first camera stable. The device includes: A first determination module, configured to obtain motion data collected by the inertial measurement unit, and determine the pose change information of the robot body according to the motion data; A second determination module, configured to obtain an environment image obtained by the first camera photographing the environment where the robot body is located, and determine the pose information of the target object included in the environment image in the coordinate system of the first camera; A third determination module, configured to obtain a tag image obtained by the second camera photographing the visual tag, and determine the relative pose information between the visual tag and the second camera according to the tag image; A fourth determination module, configured to determine the relative pose information between the first camera and the robot body according to the relative pose information between the visual tag and the second camera; A fifth determination module, configured to determine the pose information of the robot body and the relative pose information between the robot body and the target object according to the pose change information of the robot body, the pose information of the target object in the coordinate system of the first camera, and the relative pose information between the first camera and the robot body.

13. A robot, characterized in that, The robot includes a processor and a memory. A computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the environment perception method of the robot according to any one of claims 1 to 5.

14. A computer-readable storage medium, characterized in that, A computer program is stored in the storage medium, and the computer program is loaded and executed by a processor to implement the environment perception method of the robot according to any one of claims 1 to 5.

15. A computer program product, characterized in that, The computer program product includes a computer program, and the computer program is loaded and executed by a processor to implement the environment perception method of the robot according to any one of claims 1 to 5.

Images