Display method, device and equipment for perception information of mobile robot and storage medium
By reading and processing camera and sensor data in the external storage device of the mobile robot, generating display videos and uploading them to the backend server, the problems of high communication performance requirements and high operating costs in the mobile robot's perceived data display are solved, and data transmission is improved Efficiency and information accuracy.
Patent Information
- Application Number
- CN202411174146.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2024-11-12
AI Technical Summary
During the autonomous activities of mobile robots, too much data is obtained by sensors, which leads to dependence on communication status, increases labor costs, and affects the observation of perceived data. The existing technology requires too high communication performance and has too high operating costs.
Read camera data and sensor data in the external storage device of the mobile robot, locate and build maps through preset positioning and mapping methods, correct perceived map information, generate target display videos, upload display videos directly to the background server, and reduce the cost Dependence on communication status.
It improves data reading efficiency and information accuracy, reduces the dependence of mobile robots and background servers on communication status, enhances data transmission efficiency, and meets customers' needs for read perceived data.
Smart Images

Figure CN118938934A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of mobile robots, and in particular to a method, device, equipment and storage medium for displaying perceived information of a mobile robot. Background Art
[0002] Mobile robots are robots that can move autonomously or through wireless remote control. Mobile robots can be used in multiple industries. In the process of autonomous activities, mobile robots rely on the sensors set by the mobile robots to perceive the surrounding environment, and then use the robot's computing unit to calculate the data perceived by the sensors according to the algorithm. The mobile robot locates and builds an environmental map, and then plans paths and avoids obstacles to achieve autonomous activities. However, with the enhancement of the autonomous activity ability of mobile robots, the sensors obtain too much data. The background control personnel need to process the data transmitted back by the mobile robot to obtain the perception information of the mobile robot during the autonomous activity. However, the mobile robot needs to be in a communication state at all times, and the background staff also need to monitor the mobile robot in real time, resulting in excessively high labor costs when the mobile robot is autonomously active. In addition, it will be affected by the communication state, resulting in fluctuations in data transmission, which affects the observation of the perception data of the mobile robot. Summary of the invention
[0003] The present invention provides a method, device, equipment and storage medium for displaying perception information of a mobile robot, so as to solve the technical problems that the mobile robot has too high requirements on communication performance and too high operating cost when displaying perception data.
[0004] According to one aspect of the present invention, a method for displaying perception information of a mobile robot is provided, comprising:
[0005] Reading robot camera data and robot sensor data in an external storage device of the mobile robot;
[0006] Performing positioning and mapping based on the robot sensor data through a preset positioning and mapping method to obtain perception map information;
[0007] Correcting the perception map information according to the robot camera data to obtain target map information;
[0008] A target display video is generated according to the robot camera information and the target map information, and the perception information of the mobile robot is displayed through the target display video.
[0009] According to another aspect of the present invention, a device for displaying perception information of a mobile robot is provided, comprising:
[0010] An external data reading module, used for reading robot camera data and robot sensor data in an external storage device of the mobile robot;
[0011] A perception module, used to perform positioning and mapping based on the robot sensor data by a preset positioning and mapping method to obtain perception map information;
[0012] A perception correction module, used to correct the perception map information according to the robot camera data to obtain target map information;
[0013] The robot information display module is used to generate a target display video according to the robot camera information and the target map information, and display the perception information of the mobile robot through the target display video.
[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0015] at least one processor; and
[0016] a memory communicatively connected to the at least one processor; wherein,
[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for displaying perception information of a mobile robot described in any embodiment of the present invention.
[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for displaying perception information of a mobile robot as described in any embodiment of the present invention when executed.
[0019] The technical solution of the embodiment of the present invention is to read the robot camera data and the robot sensor data in the external storage device of the mobile robot, store the camera data and the sensor data of the mobile robot through the external storage device, so as to read the data when using the relevant data, improve the efficiency of data reading, and provide a data basis for the display of the perception information of the mobile robot; perform positioning and mapping based on the robot sensor data through a preset positioning and mapping method to obtain perception map information. During the movement, the mobile robot will perform positioning and mapping based on the stored sensor data to identify the surrounding environment, and then determine the perception information to be displayed, thereby improving the information validity of the perception information display; correct the perception map information according to the robot camera data to obtain the target map information, and correct the perception map information through the robot camera data to improve the information accuracy of the perception information to be displayed; generate a target display video according to the robot camera information and the target map information, and display the perception information of the mobile robot through the target display video. By combining the robot's camera information and target map information to generate the corresponding display video, the corresponding display video can be effectively generated directly in the mobile robot, and then uploaded to the background server for display, which solves the technical problems of too high communication performance requirements and too high operating costs when the mobile robot displays perception data. The mobile robot side autonomously stores camera data and sensor data, and automatically processes the camera data and sensor data to obtain the display video corresponding to the perception data, and directly uploads the display video to the background server side without the need to transmit camera data and sensor data to the background server, thereby reducing the dependence of the mobile robot and the background server on the communication status, and when the background server needs camera data and sensor data, it can directly take out the external storage device and read the corresponding data, effectively improving the efficiency of data transmission and meeting customers' needs for reading perception data.
[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0022] Figure 1A flowchart of a method for displaying perception information of a mobile robot is provided for an embodiment of the present invention;
[0023] Figure 2 A flowchart of another method for displaying perception information of a mobile robot provided by an embodiment of the present invention;
[0024] Figure 3 A flowchart of another method for displaying perception information of a mobile robot provided by an embodiment of the present invention;
[0025] Figure 4 A schematic diagram of the structure of a device for displaying perception information of a mobile robot provided by an embodiment of the present invention;
[0026] Figure 5 A schematic structural diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0029] Figure 1 A flowchart of a method for displaying perception information of a mobile robot is provided for an embodiment of the present invention. This embodiment is applicable to the case where the mobile robot processes camera data and sensor data to generate a display video during operation and displays the display video. The method can be executed by a display device for displaying perception information of a mobile robot. The display device for displaying perception information of a mobile robot can be implemented in the form of hardware and / or software. The display device for displaying perception information of a mobile robot can be configured in an electronic device. Figure 1 As shown, the method includes:
[0030] S110: Reading robot camera data and robot sensor data from an external storage device of the mobile robot.
[0031] Among them, the mobile robot can be a robot that can move autonomously or semi-autonomously in the environment; the mobile robot is used to perform designated tasks autonomously or semi-autonomously without the need for continuous control by staff. It should be noted that the mobile robot can usually rely on the installed cameras and sensors to perceive the surrounding environment, and perform autonomous navigation and movement based on the perceived environmental information. The mobile robot is provided with a computing unit and a communication unit. The computing unit can perform calculations of various types of data, and the communication unit can perform wireless and / or wireless remote communication and data transmission.
[0032] Among them, the external storage device can be a device for storing data. The external storage device can exist independently of the mobile robot and is connected to the mobile robot through an interface of the external storage device, thereby storing data transmitted by the robot to the external storage device.
[0033] Optionally, in an embodiment of the present invention, the mainboard of the mobile robot provides at least one device interface corresponding to an external storage device, and the external storage device can be connected to the mainboard of the mobile robot through the device interface to receive data transmitted by the mobile robot. Exemplarily, the external storage device can be a hard disk, a solid state drive, a USB flash drive, an optical disk, and a magnetic tape.
[0034] The robot camera data may be visual data collected by a camera of the mobile robot; the robot camera data may be used to perform image recognition of objects in the environment, and to model the environment and understand the scene. It should be noted that in the embodiment of the present invention, the camera installed on the mobile robot may be any type of camera, such as a depth camera, an infrared camera, or a thermal imaging camera.
[0035] Among them, the robot sensor data can be the data collected by the sensors in the mobile robot. The data collected by different types of sensors are not the same. It should be noted that multiple different types of sensors are installed in the mobile robot, specifically including laser radar, ultrasonic sensor, inertial measurement unit, torque sensor and gas sensor; and the robot sensor data includes laser radar data for ranging and building high-precision environmental maps; ultrasonic sensor data for short-distance ranging; inertial measurement unit measurement data for posture estimation; torque sensor data for measuring contact force; and gas sensor measurement data for detecting environmental conditions.
[0036] Optionally, in the present invention, during autonomous or semi-autonomous movement, the mobile robot will constantly collect surrounding environmental information through installed cameras and sensors, thereby generating robot camera data and robot sensor data, and storing the robot camera data and robot sensor data in an external storage device.
[0037] Optionally, when the mobile robot stores the robot camera data and the robot sensor data in an external storage device, the robot camera data and the robot sensor data are usually rolled over in the form of timestamps according to a preset data file size and quantity. During the storage process, if a single robot camera data and / or robot sensor data reaches the set size of a single data file, the system continues to create a new file, and then continues to write the robot camera data and / or robot sensor data; the file naming is based on the order in which the files are created and the file index, and a rolling storage cyclic overwrite mechanism is also set during the storage process to save the latest robot camera data and robot sensor data.
[0038] Specifically, an external storage device connected to the mobile robot is accessed, and robot camera data and robot sensor data are read from the external storage device.
[0039] S120. Perform positioning and mapping based on the robot sensor data using a preset positioning and mapping method to obtain perception map information.
[0040] The preset positioning and mapping method may be a method pre-set for constructing a SLAM (Simultaneous Localization and Mapping) map. It should be noted that after obtaining the robot sensor data, data feature extraction is performed based on the robot sensor data, and then data fusion is performed on a variety of different robot sensor data, and then initial positioning and mapping are performed to construct a SLAM map.
[0041] The perception map information may be a SLAM map constructed by the mobile robot. The perception map information may be used to help the mobile robot to perform self-positioning in an unknown environment.
[0042] S130. Correct the perception map information according to the robot camera data to obtain target map information.
[0043] The target map information may be a SLAM map modified and updated based on the perception map information. It should be noted that, during the movement of the mobile robot, the sensors installed in the mobile robot are disturbed by the environment, and the accuracy of the sensors will decrease, which will lead to data errors in the robot perception data of the mobile robot, resulting in errors in the perception map information, and the inability to accurately feedback the environmental information around the mobile robot. Therefore, the perception map information needs to be verified and corrected through the robot camera data to improve the accuracy of the perception map information.
[0044] Specifically, after the perception map information is obtained, the perception map information is error corrected through the robot camera data to obtain the corrected target map information.
[0045] S140. Generate a target display video according to the robot camera information and the target map information, and display the perception information of the mobile robot through the target display video.
[0046] The target display video may be a display of the perception information of the mobile robot during movement; the perception information may be environmental information perceived by the mobile robot through a camera and a sensor. The perception information may include video data and sensor data.
[0047] Optionally, since the mobile robot has many types of sensors and many types of sensor data, based on the target map information, the data of multiple sensors can be combined for fusion processing and redistribution, and then combined with the robot camera data to generate a target display video, and the perception information of the mobile robot can be displayed through the target display video.
[0048] The technical solution of the embodiment of the present invention is to read the robot camera data and the robot sensor data in the external storage device of the mobile robot, store the camera data and the sensor data of the mobile robot through the external storage device, so as to read the data when using the relevant data, improve the efficiency of data reading, and provide a data basis for the display of the perception information of the mobile robot; perform positioning and mapping based on the robot sensor data through a preset positioning and mapping method to obtain perception map information. During the movement, the mobile robot will perform positioning and mapping based on the stored sensor data to identify the surrounding environment, and then determine the perception information to be displayed, thereby improving the information validity of the perception information display; correct the perception map information according to the robot camera data to obtain the target map information, and correct the perception map information through the robot camera data to improve the information accuracy of the perception information to be displayed; generate a target display video according to the robot camera information and the target map information, and display the perception information of the mobile robot through the target display video. By combining the robot's camera information and target map information to generate the corresponding display video, the corresponding display video can be effectively generated directly in the mobile robot, and then uploaded to the background server for display, which solves the technical problems of too high communication performance requirements and too high operating costs when the mobile robot displays perception data. The mobile robot side autonomously stores camera data and sensor data, and automatically processes the camera data and sensor data to obtain the display video corresponding to the perception data, and directly uploads the display video to the background server side without the need to transmit camera data and sensor data to the background server, thereby reducing the dependence of the mobile robot and the background server on the communication status, and when the background server needs camera data and sensor data, it can directly take out the external storage device and read the corresponding data, effectively improving the efficiency of data transmission and meeting customers' needs for reading perception data.
[0049] Figure 2 This is a flowchart of another method for displaying perception information of a mobile robot provided by an embodiment of the present invention. The relationship between this embodiment and the above embodiment is a specific method for correcting perception map information through robot camera data. Figure 2 As shown, the method includes:
[0050] S210: Read robot camera data and robot sensor data from an external storage device of the mobile robot.
[0051] S220: Perform positioning and mapping based on the robot sensor data using a preset positioning and mapping method to obtain perception map information.
[0052] S230. Detecting feature position deviations of the perception map information according to the robot camera data, and determining perception deviation information.
[0053] The perception deviation information may be error information between the perception map information and the robot camera data. Optionally, when the mobile robot constructs the perception map information, it usually detects identifiable feature points in the perceived robot sensor data, uses these feature points as reference points for constructing the map information, and then obtains the perception map information based on the reference points; after obtaining the perception map information, when the perception map information is detected through the robot camera data, the position errors of each reference point in the constructed perception map information and the robot camera data collected by the real camera are determined, and these errors are used as the perception deviation information.
[0054] Specifically, based on the perceived map information, the position deviation of the feature points in the perceived map information is calculated through the robot camera data to obtain the perceived deviation information.
[0055] Optionally, in another optional embodiment of the present invention, the detecting the feature position deviation of the perception map information according to the robot camera data and determining the perception deviation information includes:
[0056] Acquire the position and posture information of the mobile robot; predict the characteristics of the mobile robot according to the perception map information and the position and posture information to determine the predicted characteristic information; perform deviation detection on the predicted characteristic information according to the robot camera data to obtain the perception deviation information.
[0057] Among them, the posture information can be the position and direction information of the mobile robot in space. The posture information can be applied to the motion control and spatial navigation of the mobile robot. In the present invention, the posture information can be used to construct perception map information.
[0058] Optionally, in an embodiment of the present invention, the posture of the mobile robot is estimated by using the measurement data of the inertial measurement unit, and an initial posture calculation is performed on the measurement data collected by the inertial measurement unit to perform a preliminary posture estimation, and the result of the preliminary posture estimation is fused with the laser radar data through a sensor fusion algorithm to obtain posture information. It should be noted that in an embodiment of the present invention, the posture information is updated in real time as the posture of the mobile robot changes and moves to ensure the accuracy and effectiveness of the posture information.
[0059] The predicted feature information may be feature point position information obtained by predicting the position of each feature of the environment in the robot video information by the mobile robot. It should be noted that each feature point in the predicted feature information is a reference point detected by the mobile robot, and then the position of these reference points in the robot video information is predicted to obtain the predicted feature information.
[0060] Optionally, after obtaining the perception map information, the positions of various reference points detected in the environment where the mobile robot is located in the robot video data are predicted based on the current posture information and perception map information of the mobile robot to obtain predicted feature information.
[0061] Optionally, after obtaining the predicted feature information of the mobile robot, the predicted feature information is compared one by one with the actual position information in the robot camera data collected by the camera. In the case of deviation, the deviation of each reference point is calculated one by one to obtain the perception deviation information.
[0062] Optionally, when there is a deviation between the predicted feature information of the mobile robot and the position information in the robot camera data actually collected by the camera, the deviation between the two is calculated, and the deviation is compared with a preset deviation threshold. If the deviation is greater than the deviation threshold, the deviation is used as the perception deviation information. The deviation can be obtained by adding the vector difference between the position information of each reference point in the predicted feature information and the position information in the robot camera data.
[0063] Specifically, the position and posture information of the mobile robot is obtained, and feature prediction of the mobile robot is performed based on the perception map confidence and position and posture information of the mobile robot to obtain predicted feature information, and deviation detection of the predicted feature information is performed through the robot camera data to obtain perception deviation information.
[0064] Optionally, in another optional embodiment of the present invention, performing deviation detection on the predicted feature information according to the robot camera data to obtain the perception deviation information includes:
[0065] Perform frame extraction on the robot camera data to determine video frame information; perform feature detection on the video frame information to determine real feature information; perform deviation calculation based on the real feature information and the predicted feature information to determine the perception deviation information.
[0066] Among them, the video frame information can be a single image that constitutes the video in the robot camera data. It should be noted that the robot camera data is usually composed of continuous video frames, and the robot camera data can be extracted frame by frame through frame extraction to obtain the video frame information.
[0067] Optionally, since the video frame information is static image information, it is possible to perform feature recognition and position detection on the reference points in the video frame information to obtain the real features and real position information corresponding to each reference point, that is, the real feature information. The real feature information may be the real features of the reference points and the reference point position information in the video frame image.
[0068] Optionally, deviations are calculated one by one based on the position information of the real feature information and the predicted feature information in the robot camera data, thereby obtaining the perception deviation information.
[0069] S240: Correct the perceived map information according to the perceived deviation information to determine the target map information.
[0070] Specifically, the position information of the reference point in the perceived map information is corrected by using the perceived deviation information corresponding to the perceived map information, and then the perceived map information is updated to obtain the target map information.
[0071] Optionally, in another optional embodiment of the present invention, the correcting the perception map information according to the perception deviation information to determine the target map information includes:
[0072] The posture information is corrected according to the perception deviation information to determine the target posture information; the perception map information is corrected according to the target posture information and the robot camera data to determine the target map information.
[0073] The target posture information may be the actual posture information of the mobile robot in the environment.
[0074] Optionally, the perception deviation information may indicate that the position and posture information of the mobile robot has certain errors, and it is necessary to re-estimate the position of the mobile robot according to the deviation information to optimize the position and posture information of the mobile robot, obtain updated target position and posture information, and reduce the error. It should be noted that the presence of certain perception deviation information of the mobile robot may be due to errors in the position and posture estimation, or it may be due to changes in the environment in which the mobile robot is located, resulting in errors in the position and posture information.
[0075] Specifically, the posture of the mobile robot is corrected through the perception deviation information of the mobile robot to obtain the target posture information, the position of the reference point is updated based on the target posture information and the robot camera information, and new reference points are added through the robot camera information. The map is constructed based on the new reference points to obtain updated target map information.
[0076] S250, generating a target display video according to the robot camera information and the target map information, and displaying the perception information of the mobile robot through the target display video.
[0077] The embodiment of the present invention can correct the perception map information through the robot camera data, ensuring that the mobile robot can maintain the accuracy of positioning in a dynamic and uncertain environment, thereby constructing accurate environmental map information, and then displaying accurate perception information, reducing the error of perception information, and improving the user experience.
[0078] Figure 3 This is a flowchart of another method for displaying perception information of a mobile robot provided by an embodiment of the present invention. The relationship between this embodiment and the above embodiment is a specific method for displaying perception information of a mobile robot. Figure 3 As shown, the method includes:
[0079] S310: Read robot camera data and robot sensor data from an external storage device of the mobile robot.
[0080] S320: Perform positioning and mapping based on the robot sensor data using a preset positioning and mapping method to obtain perception map information.
[0081] S330: Correct the perception map information according to the robot camera data to obtain target map information.
[0082] S340: Process the robot camera information in real-time video stream to obtain a video image to be displayed.
[0083] The video image to be displayed may be a video image that can be displayed in real time.
[0084] Optionally, since the robot camera information is video information collected by at least one camera, and in order to improve the accuracy of the mobile robot's perception of environmental information, there are too many features of the video information collected in real time, and it is necessary to process the video stream in real time, fuse the video information collected by each camera, simplify the features in the video information, and adjust the resolution and size of the video to obtain the video image to be displayed.
[0085] Optionally, the video image to be displayed can display the environment information observed by the mobile robot with a camera, display the shape of objects in the environment, the distance between the objects and the mobile robot, and select the display clarity according to the distance to ensure the usability of the video image to be displayed. Exemplarily, the video image to be displayed can simulate a fixed shot in an office environment, showing two tables and chairs in the distance, including some office supplies on the tables and chairs, and providing a basic view of the office.
[0086] Specifically, the robot camera information is processed into a real-time video stream to obtain a video image to be displayed.
[0087] S350: Visualize the target map information through a preset visualization tool to obtain point cloud information to be displayed.
[0088] The visualization tool may be a tool pre-set to visualize the SLAM map; the visualization tool may be used to directly convert the SLAM map into a visualized image, and the visualization tool may also provide color mapping of the SLAM map.
[0089] The point cloud information to be displayed may be a point cloud image that can be used to display target map information.
[0090] Specifically, the target map information is input into the visualization tool, and the target map information is converted by the visualization tool to obtain the point cloud information to be displayed.
[0091] S360: Generate the target display video according to the video image to be displayed and the point cloud information to be displayed.
[0092] Optionally, after obtaining the video image to be displayed and the point cloud information to be displayed, each frame of the video image to be displayed and each frame of the point cloud information to be displayed are fused to obtain a target display video, wherein the left half of each frame of the target display video is the video image to be displayed, and the right half is the point cloud information to be displayed.
[0093] Specifically, the target display video is generated by performing video fusion on the video image to be displayed and the point cloud information to be displayed.
[0094] Optionally, in another optional embodiment of the present invention, generating the target display video according to the video image to be displayed and the point cloud information to be displayed includes:
[0095] Acquire the working state information of the mobile robot; wherein the working state information includes power state information, signal connection information, posture state information and motion state information;
[0096] Obtaining battery status information and positioning quality information of the mobile robot;
[0097] The target display video is generated based on the video image to be displayed, the point cloud information to be displayed, the working status information, the battery status information and the positioning quality information.
[0098] Among them, the power status information may be the connection status of the mobile robot's power supply; exemplarily, the power status information may include a connected state and a disconnected state; the signal connection information may be the communication signal connection status of the mobile robot; exemplarily, the signal connection information may include having a communication signal and not having a communication signal; the posture state information may be the current target posture information of the mobile robot; the motion state information may be the current motion state of the mobile robot; exemplarily, the motion state information may be motion or still.
[0099] Among them, the battery status information can be a consumption trend chart of the available power of the mobile robot's own power supply; the positioning quality information can be a trend chart of the accuracy of the mobile robot's current positioning.
[0100] Specifically, the working status information, battery status information and positioning quality information of the mobile robot are obtained, the power status information, signal connection information, posture status information and motion status information of the working status information are displayed in the form of a pattern below the video image to be displayed, and the battery status information and positioning quality information are displayed below the point cloud information to be displayed, thereby generating a target display video. In the embodiment of the present invention, the upper left corner of the target display video also displays the real time corresponding to the target display video, and provides a video progress bar to facilitate the staff to review the target display video of other time periods.
[0101] When generating a target display video, the embodiment of the present invention synchronously displays the video image to be displayed shot by the mobile robot and the point cloud information to be displayed collected by the sensor, and adds the working status information, battery status information and positioning quality information of the mobile robot to the target display video. When displaying the perception information of the mobile robot, it also realizes a comprehensive display of the working status and the self status of the mobile robot, further improving the information richness of the target display video and effectively improving the user experience.
[0102] Figure 4 A schematic diagram of a display device for mobile robot perception information provided by an embodiment of the present invention. Figure 4 As shown, the device includes: an external data reading module 410, a perception module 420, a perception correction module 430 and a robot information display module 440; wherein,
[0103] The external data reading module 410 is used to read the robot camera data and the robot sensor data in the external storage device of the mobile robot;
[0104] A perception module 420, configured to perform positioning and mapping based on the robot sensor data by a preset positioning and mapping method to obtain perception map information;
[0105] A perception correction module 430, used to correct the perception map information according to the robot camera data to obtain target map information;
[0106] The robot information display module 440 is used to generate a target display video according to the robot camera information and the target map information, and display the perception information of the mobile robot through the target display video.
[0107] The technical solution of the embodiment of the present invention is to read the robot camera data and the robot sensor data in the external storage device of the mobile robot, store the camera data and the sensor data of the mobile robot through the external storage device, so as to read the data when using the relevant data, improve the efficiency of data reading, and provide a data basis for the display of the perception information of the mobile robot; perform positioning and mapping based on the robot sensor data through a preset positioning and mapping method to obtain perception map information. During the movement, the mobile robot will perform positioning and mapping based on the stored sensor data to identify the surrounding environment, and then determine the perception information to be displayed, thereby improving the information validity of the perception information display; correct the perception map information according to the robot camera data to obtain the target map information, and correct the perception map information through the robot camera data to improve the information accuracy of the perception information to be displayed; generate a target display video according to the robot camera information and the target map information, and display the perception information of the mobile robot through the target display video. By combining the robot's camera information and target map information to generate the corresponding display video, the corresponding display video can be effectively generated directly in the mobile robot, and then uploaded to the background server for display, which solves the technical problems of too high communication performance requirements and too high operating costs when the mobile robot displays perception data. The mobile robot side autonomously stores camera data and sensor data, and automatically processes the camera data and sensor data to obtain the display video corresponding to the perception data, and directly uploads the display video to the background server side without the need to transmit camera data and sensor data to the background server, thereby reducing the dependence of the mobile robot and the background server on the communication status, and when the background server needs camera data and sensor data, it can directly take out the external storage device and read the corresponding data, effectively improving the efficiency of data transmission and meeting customers' needs for reading perception data.
[0108] Optionally, the perception correction module is specifically used for:
[0109] Detecting a characteristic position deviation of the perceived map information according to the robot camera data, and determining the perceived deviation information;
[0110] The perceived map information is corrected according to the perceived deviation information to determine the target map information.
[0111] Optionally, the perception correction module is further used to:
[0112] Acquiring position information of the mobile robot;
[0113] Predicting features of the mobile robot according to the perception map information and the position and posture information to determine predicted feature information;
[0114] The predicted feature information is subjected to deviation detection according to the robot camera data to obtain the perceived deviation information.
[0115] Optionally, the perception correction module is further used to:
[0116] Performing frame extraction on the robot camera data to determine video frame information;
[0117] Performing feature detection on the video frame information to determine real feature information;
[0118] Deviation calculation is performed based on the real feature information and the predicted feature information to determine the perception deviation information.
[0119] Optionally, the perception correction module is further used to:
[0120] Correcting the position and posture information according to the perception deviation information to determine the target position and posture information;
[0121] The perception map information is corrected according to the target posture information and the robot camera data to determine the target map information.
[0122] Optionally, the robot information display module is specifically used to:
[0123] Processing the robot camera information in real-time video stream to obtain a video image to be displayed;
[0124] Visualizing the target map information through a preset visualization tool to obtain point cloud information to be displayed;
[0125] The target display video is generated according to the video image to be displayed and the point cloud information to be displayed.
[0126] Optionally, the robot information display module is further used to:
[0127] Acquire the working state information of the mobile robot; wherein the working state information includes power state information, signal connection information, posture state information and motion state information;
[0128] Obtaining battery status information and positioning quality information of the mobile robot;
[0129] The target display video is generated based on the video image to be displayed, the point cloud information to be displayed, the working status information, the battery status information and the positioning quality information.
[0130] The device for displaying perception information of a mobile robot provided in an embodiment of the present invention can execute the method for displaying perception information of a mobile robot provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.
[0131] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their modes are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0132] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0133] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer grid such as the Internet and / or various telecommunication grids.
[0134] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for displaying perceived information of a mobile robot.
[0135] In some embodiments, the method for displaying the perceived information of the mobile robot may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for displaying the perceived information of the mobile robot described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for displaying the perceived information of the mobile robot in any other appropriate manner (e.g., by means of firmware).
[0136] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0137] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the modes / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0138] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0139] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0140] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a grid browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication grid). Examples of communication grids include: a local area network (LAN), a wide area network (WAN), a blockchain grid, and the Internet.
[0141] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication grid. The client and server relationship is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0143] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the method for displaying perception information of a mobile robot provided in any embodiment of the present invention are implemented. The method includes:
[0144] Reading robot camera data and robot sensor data in an external storage device of the mobile robot;
[0145] Performing positioning and mapping based on the robot sensor data through a preset positioning and mapping method to obtain perception map information;
[0146] Correcting the perception map information according to the robot camera data to obtain target map information;
[0147] A target display video is generated according to the robot camera information and the target map information, and the perception information of the mobile robot is displayed through the target display video.
[0148] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.
[0149] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0150] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0151] Computer program code for performing the operation of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of grid, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0152] It should be understood by those skilled in the art that the above modules or steps of the present invention can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a grid composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.
[0153] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0154] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for displaying perception information of a mobile robot, characterized in that: include: Reading robot camera data and robot sensor data in an external storage device of the mobile robot; Performing positioning and mapping based on the robot sensor data through a preset positioning and mapping method to obtain perception map information; Correcting the perception map information according to the robot camera data to obtain target map information; A target display video is generated according to the robot camera information and the target map information, and the perception information of the mobile robot is displayed through the target display video.
2. The method according to claim 1, characterized in that The step of correcting the perceived map information according to the robot camera data to obtain target map information includes: Detecting a characteristic position deviation of the perceived map information according to the robot camera data, and determining the perceived deviation information; The perceived map information is corrected according to the perceived deviation information to determine the target map information.
3. The method according to claim 2, characterized in that The detecting the feature position deviation of the perceived map information according to the robot camera data and determining the perceived deviation information includes: Acquiring position information of the mobile robot; Predicting features of the mobile robot according to the perception map information and the position and posture information to determine predicted feature information; The predicted feature information is subjected to deviation detection according to the robot camera data to obtain the perceived deviation information.
4. The method according to claim 3, characterized in that: The performing deviation detection on the predicted feature information according to the robot camera data to obtain the perception deviation information includes: Performing frame extraction on the robot camera data to determine video frame information; Performing feature detection on the video frame information to determine real feature information; Deviation calculation is performed based on the real feature information and the predicted feature information to determine the perception deviation information.
5. The method according to claim 4, characterized in that The step of correcting the perceived map information according to the perceived deviation information to determine the target map information includes: Correcting the position and posture information according to the perception deviation information to determine the target position and posture information; The perception map information is corrected according to the target posture information and the robot camera data to determine the target map information.
6. The method according to claim 1, characterized in that The generating of the target display video according to the robot camera information and the target map information includes: Processing the robot camera information in real-time video stream to obtain a video image to be displayed; Visualizing the target map information through a preset visualization tool to obtain point cloud information to be displayed; The target display video is generated according to the video image to be displayed and the point cloud information to be displayed.
7. The method according to claim 6, characterized in that The generating the target display video according to the video image to be displayed and the point cloud information to be displayed includes: Acquire the working state information of the mobile robot; wherein the working state information includes power state information, signal connection information, posture state information and motion state information; Acquiring battery status information and positioning quality information of the mobile robot; The target display video is generated based on the video image to be displayed, the point cloud information to be displayed, the working status information, the battery status information and the positioning quality information.
8. A display device for mobile robot perception information, characterized in that: include: An external data reading module, used for reading robot camera data and robot sensor data in an external storage device of the mobile robot; A perception module, used to perform positioning and mapping based on the robot sensor data by a preset positioning and mapping method to obtain perception map information; A perception correction module, used to correct the perception map information according to the robot camera data to obtain target map information; The robot information display module is used to generate a target display video according to the robot camera information and the target map information, and display the perception information of the mobile robot through the target display video.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for displaying perception information of a mobile robot according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for displaying perception information of a mobile robot according to any one of claims 1 to 7 when executed.
