Sensor Data Processing Method, System, and Readable Storage Medium

Abstract

The present application discloses a sensor data processing method, system, mobile robot and readable storage medium. In one embodiment, the sensor data processing method collects data of the environment around the robot through multiple sensors, respectively performs obstacle detection on the collected environmental data, then generates a local obstacle distribution map according to the obstacle detection results, and further sends the local obstacle distribution maps generated by the multiple sensors to the processor of the robot. The processor fuses the received multiple local obstacle distribution maps into a global obstacle distribution map, thereby realizing the processing of the collected data on multiple sensors, and then sending the processed data to the processor, reducing the amount of data transmitted between the multiple sensors and the processor, reducing the pressure on the transmission bandwidth, improving the processing efficiency of the processor, and reducing the consumption of computing resources of the processor.

Description

Technical Field

[0001] This application relates to the field of robot technology, and more specifically, to a method for processing sensor data, a system including a robot and sensors, a mobile robot, and a readable storage medium. Background Art

[0002] With the development of sensor technology and artificial intelligence technology, robot products have become increasingly popular. Robots usually use a variety of sensors such as lidar, optical cameras, depth cameras, ultrasonic sensors, etc. to collect external environment data. The data collected by multiple sensors is uniformly transmitted to the main processor of the robot, and the main processor performs data processing and decision control of the robot.

[0003] With the increase in the number of sensors and the enhancement of data acquisition capabilities, transmitting all the data collected by the sensors to the main processor of the robot for processing has caused an increasing pressure on the transmission network bandwidth and the computing power of the main processor. For example, a depth camera can obtain various data such as high-definition color pictures, infrared pictures, point cloud data, and point cloud confidence information. Transmitting all these data to the main processor will consume a large amount of bandwidth and affect the processing frame rate. Summary of the Invention

[0004] In view of the above-mentioned disadvantages of the related art, the purpose of this application is to provide a method for processing sensor data, a system including a robot and sensors, a mobile robot, and a readable storage medium, which is used to solve the problem of large transmission bandwidth and computing resource pressure caused by the fact that all the data collected by sensors in the prior art is processed by the main processor.

[0005] To achieve the above object and other related objects, the first aspect disclosed in this application provides a method for processing sensor data for a system including a robot and sensors, including the following steps: collecting data of the environment around the robot through multiple sensors; the multiple sensors performing obstacle detection on the collected environment data to generate local data; the multiple sensors sending the locally generated data to a processor, where the processor is the main processor of the robot; the processor processing the received multiple local data into a global obstacle distribution map.

[0006] The second aspect disclosed in this application provides a system including a robot and sensors, including: multiple sensors for collecting data of the environment around the robot, each sensor performing obstacle detection on the collected environment data to generate local data and sending it to a processor; a memory for storing instructions executed by one or more processors of the robot; and a processor, which is the main processor of the robot, for processing the received multiple local data into a global obstacle distribution map.

[0007] A third aspect of the present application provides a mobile robot, including: a plurality of sensors for collecting data of the environment around the robot, each of the sensors performing obstacle detection on the collected environmental data to generate local data and sending it to a processor; a mobile device for performing a moving operation; a storage device for storing at least one program; and a processing device connected to the plurality of sensors, the mobile device, and the storage device for executing the at least one program to execute the sensor data processing method as described in the first aspect above.

[0008] A fourth aspect of the present application provides a readable storage medium storing at least one computer program, and when the computer program is run by a processor, it controls the device where the storage medium is located to execute the sensor data processing method as described in the first aspect above.

[0009] In summary, in an embodiment disclosed in the present application, the sensor data processing method for a system including a robot and sensors collects data of the environment around the robot through a plurality of sensors, respectively performs obstacle detection on the collected environmental data, then generates local data according to the obstacle detection results, and further sends the local data generated by the plurality of sensors to the processor of the robot. The processor fuses the received plurality of local data into a global obstacle distribution map, thereby realizing the processing of the collected data on the plurality of sensors, and then sending the processed data to the processor, reducing the amount of data transmitted between the plurality of sensors and the processor, reducing the pressure on the transmission bandwidth, improving the processing efficiency of the processor, and reducing the consumption of computing resources of the processor.

[0010] Those skilled in the art can easily understand other aspects and advantages of the present application from the following detailed description. Only the exemplary embodiments of the present application are shown and described in the following detailed description. As those skilled in the art will recognize, the content of the present application enables those skilled in the art to make changes to the disclosed specific embodiments without departing from the spirit and scope of the invention involved in the present application. Accordingly, the descriptions in the drawings and the specification of the present application are merely exemplary and not restrictive. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The specific features of the invention involved in the present application are shown in the appended claims. The features and advantages of the invention involved in the present application can be better understood by referring to the exemplary embodiments and the drawings described in detail below. A brief description of the drawings is as follows:

[0012] Figure 1 It shows a schematic diagram of the scenario where the robot constructs an obstacle distribution map in an embodiment of the present application.

[0013] Figure 2Shown is a schematic diagram of the hardware structure of the robot in an embodiment of the present application.

[0014] Figure 3 Shown is a flowchart of the sensor data processing method of the present application in an embodiment.

[0015] Figure 4 Shown is a schematic diagram of the local obstacle distribution maps respectively generated by two sensors according to the obstacle detection results in an embodiment of the present application.

[0016] Figure 5 Shown is a schematic diagram of the processor processing the acquired local obstacle distribution map in an embodiment of the present application.

[0017] Figure 6 Shown is a flowchart of the sensor data processing method of the present application in another embodiment.

[0018] Figure 7 Shown is a flowchart of the method for the processor of the robot to adjust the data acquisition mode of the sensor in an embodiment.

[0019] Figure 8 Shown is a principle block diagram of the internal processor of the sensor in an embodiment of the present application.

[0020] Figure 9 Shown is a system block diagram of the system including the robot and the sensor in an embodiment of the present application.

[0021] Figure 10 Shown is a system block diagram of the system including the robot and the sensor in another embodiment of the present application.

[0022] Figure 11 Shown is a block diagram of the mobile robot of the present application in an embodiment. Detailed implementation manners

[0023] The following specific embodiments illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in this specification. In the following description, reference is made to the accompanying drawings, which describe several embodiments of the present application. It should be understood that other embodiments may also be used, and changes in the composition of modules or units, electrical, and operations may be made without departing from the spirit and scope of the present disclosure. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is only defined by the published claims. The terms used herein are only for describing specific embodiments and are not intended to limit the present application.

[0024] It will be appreciated that, as used herein, the term "module" may refer to or include an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) that executes one or more software or firmware programs and / or memory, combinational logic circuitry, and / or other suitable hardware components that provide the described functionality, or may be a part of such hardware components.

[0025] It will be appreciated that in the embodiments of the present application, the processor may be a microprocessor, a digital signal processor, a microcontroller, etc., and / or any combination thereof. According to another aspect, the processor may be a single-core processor, a multi-core processor, etc., and / or any combination thereof.

[0026] It will be appreciated that the sensor data processing method of the present application is applicable to robots that process data from multiple sensors.

[0027] The embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0028] The present application provides a sensor data processing method for a system including a robot and sensors. In the following embodiments of the present application, the robot in the system including the robot and sensors is a mobile robot, and the mobile robot refers to an autonomous mobile device having the ability to build a map in a physical space, including but not limited to: drones, industrial robots, home companion mobile devices, medical mobile devices, household cleaning robots, commercial cleaning robots, intelligent vehicles, and patrol robots, etc., one or more of them. For example, it is a service robot (such as a cleaning robot, a patrol robot, or a robot for delivering food / items) applied in a commercial scenario to perform a certain task or a home robot (such as a floor cleaning robot or a companion robot, etc.) applied in a home scenario to perform cleaning or entertainment tasks.

[0029] The physical space refers to the actual three-dimensional space where the mobile robot is located, which can be described by abstract data constructed in the space coordinate system. For example, the physical space includes but is not limited to a home residence, public places (such as an office, a shopping mall, a hospital, an underground parking lot, and a bank), etc. For a mobile robot, the physical space generally refers to an indoor space, that is, the space has boundaries in the length, width, and height directions, etc. Particularly includes physical spaces with characteristics such as a large space range and a high scene repetition degree, such as a shopping mall and a waiting hall.

[0030] In a system including a robot and sensors, the sensors include, for example, laser sensors, ultrasonic sensors, infrared sensors, optical cameras (such as monocular cameras or binocular cameras), depth cameras (such as ToF sensors), millimeter-wave radar sensors, etc.; among them, for example, a laser sensor can determine its distance relative to an obstacle based on the time difference between the emission of a laser beam and the reception of the laser beam; another example is that an ultrasonic sensor can determine the distance of a mobile robot relative to an obstacle based on the vibration signal of the sound wave it emits being rebounded by the obstacle; yet another example is that a binocular imaging device can determine the distance of a mobile robot relative to an obstacle based on the images captured by its two cameras using the triangulation principle; still another example is that the infrared light projector of a ToF (Time of Flight) sensor projects infrared light outward, the infrared light is reflected after encountering the obstacle to be measured, and is received by the receiving module, and by recording the time from the emission to the reception of the infrared light, the depth information of the illuminated obstacle is calculated.

[0031] The obstacles include, but are not limited to, physical space partitions (the physical space partitions include, but are not limited to, one or more of door bodies, floor-to-ceiling windows, screens, walls, columns, and row-connected access gates, etc.), tables, chairs, cabinets, stairs, escalators, and scattered single roadblocks (such as flower pots), human bodies, etc.

[0032] In an example of the present application, a system including a robot and sensors refers to a robot system with multiple sensors provided on the robot body. Specifically, according to the functional requirements of the mobile robot, multiple sensors are arranged on the robot body. Please refer to Figure 1 , which shows a schematic diagram of the scenario where a robot constructs an obstacle distribution map in an embodiment of the present application. For example, as shown in the present application Figure 1 , on the robot 100, 4 sensors 401, 402, 403, and 404 are provided, and the 4 sensors continuously detect the surrounding environment during the working state of the robot 100; in an actual implementation manner, the positions of the sensors 401, 402, 403, and 404 on the robot 100 body will be set at corresponding positions according to the functions of each sensor. For example, a fisheye camera or a laser sensor is set on the top of the robot body, a binocular vision camera is set on the front side of the robot, ultrasonic sensors are set on the left and right sides or one side of the robot, and a laser sensor or a ToF sensor is set on the front or rear side of the robot, etc.

[0033] In another example of the present application, a system including a robot and sensors refers to a wireless communication system formed by arranging sensors at positions outside the robot body. For example, in the working environment of the robot, such as in physical spaces like the walkways / aisles where the robot often cruises, on walls or columns, etc. Moreover, the sensors are arranged on other electronic devices independent of the mobile robot, such as sensors on user terminal devices (such as smartphones, tablets, surveillance cameras, smart screens, etc.). For example, by binding a smartphone to the robot, the sensors of the smartphone can exchange data with the robot. In these cases, the system is composed of the robot and externally arranged sensors, and the data transmission between the robot and the external sensors is carried out through wireless communication methods, which may include but are not limited to wireless communication methods such as Wireless Fidelity (Wi-Fi) and Bluetooth (Blue Tooth).

[0034] The sensor data processing method of the present application collects data on the surrounding environment of the robot through multiple sensors, so that the multiple sensors perform obstacle detection on the collected environmental data to generate local data. Then, the multiple sensors send the locally generated data to the processor, and the processor processes the received multiple local data into a global obstacle distribution map. The method provided by the present application collects data on the surrounding environment of the robot through multiple sensors, respectively performs obstacle detection on the collected environmental data, generates local data according to the obstacle detection results, and then each sensor sends the locally generated data to the processor of the robot. The processor fuses the received multiple local data into a global obstacle distribution map, thereby realizing the processing of the collected data on multiple sensors and then sending the processed data to the processor, reducing the amount of data transmitted between the multiple sensors and the processor, reducing the pressure on the transmission bandwidth, improving the processing efficiency of the processor, and reducing the consumption of computing resources of the processor.

[0035] Please refer to Figure 2 , which shows a schematic diagram of the hardware structure of the robot in an embodiment of the present application. As shown in the figure, in this embodiment, the robot 100 includes a processor 300, depth cameras 401 to 404, a memory 160, and a motion component (mobile device) 170, etc.

[0036] In an embodiment, the processor 300 can be used to read and execute computer-readable instructions. In a specific implementation, the processor 300 mainly includes a controller, an arithmetic unit, and registers. Among them, the controller is mainly responsible for instruction decoding and sending control signals for operations corresponding to the instructions. The arithmetic unit is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations, etc., and can also perform address operations and conversions. The registers are mainly responsible for storing register operands and intermediate operation results temporarily stored during the execution of instructions. In a specific implementation, the hardware architecture of the processor 300 can be an application-specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, or an NP architecture, etc.

[0037] The processor 300 may include one or more processing units. For example, the processor 300 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0038] In some embodiments of the present application, the processor 300 can be used to receive point cloud data, image data, and local obstacle distribution map data sent by the depth cameras 401 to 404, fuse these data to obtain a complete two-dimensional obstacle distribution map, and then control the robot 100 to complete specific tasks such as item delivery, room cleaning / washing, etc. according to the two-dimensional obstacle distribution map.

[0039] It can be understood that in an embodiment where the depth cameras 401 to 404 are arranged outside the body of the robot 100, the processor 300 can receive the point cloud data, image data, and local obstacle distribution map data sent by the depth cameras 401 to 404 through wireless communication.

[0040] In an embodiment, the depth cameras 401 to 404 are used to periodically collect image information of the environment around the robot 100 when the robot 100 is moving. The way the depth cameras 401 to 404 collect image information is similar to that of ordinary cameras. An external object generates an optical image through a lens and projects it onto a photosensitive element, which can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing, and the DSP converts the digital image signal into a standard image signal in formats such as RGB and YUV.

[0041] The depth cameras 401 to 404 are also used to obtain the distance between obstacles in the environment and the depth cameras. The distance between the two can be used to construct an obstacle distribution map for the robot 100 to use.

[0042] In some embodiments of the present application, the depth cameras 401 to 404 further include an internal processor, which is used to preprocess the data collected by the depth cameras, such as denoising, coordinate transformation, clustering analysis, obstacle detection, target recognition, etc., and then transmit the processed data to the processor 300.

[0043] In an embodiment, the internal processor in the depth cameras 401 to 404 may also include one or more processing units, such as an application processor (AP), a modulation and demodulation processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. For example, the NPU is a neural-network (NN) computing processor, which can quickly process input information by referring to the biological neural network structure, such as referring to the transmission mode between human brain neurons, and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the robot 100 can be realized, such as image recognition, obstacle detection, object recognition, etc.

[0044] In an embodiment, the memory 160 is coupled to the processor 300 and is used to store various software programs and / or multiple sets of instructions. Specifically, the memory 160 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 160 may store an operating system, such as an embedded operating system like uCOS, VxWorks, RTLinux, etc. The memory 160 may also store a communication program, which may be used to communicate with a smart terminal, one or more servers, or additional devices.

[0045] In some embodiments of the present application, the memory 160 may be used to store global obstacle distribution map data, in which the positions and sizes of obstacles are marked. The robot 100 may use the stored global obstacle distribution map data to control its own movement and avoid collisions with obstacles.

[0046] In an embodiment, the moving component 170 is used to control the movement of the robot 100 according to instructions issued by the processor 300. The moving component 170 may include multiple components related to movement, such as motors, drive shafts, wheels, etc. In some embodiments of the present application, the moving component 170 is used to implement various movement forms of the robot 100, such as forward movement, backward movement, left movement, right movement, bow movement, etc.

[0047] The moving component 170 is used to perform a controlled movement operation. In an actual implementation manner, the moving component 170 may include a walking mechanism and a driving mechanism. Among them, the walking mechanism may be disposed at the bottom of the mobile robot, and the driving mechanism is built into the housing of the mobile robot. Further, the walking mechanism may adopt a walking wheel mode. In one implementation manner, the walking mechanism may, for example, include at least two omnidirectional walking wheels, and the at least two omnidirectional walking wheels are used to achieve movements such as forward, backward, turning, and rotation. In other implementation manners, the walking mechanism may, for example, include a combination of multiple straight walking wheels and at least one auxiliary steering wheel. Among them, in the case where the at least one auxiliary steering wheel does not participate, the multiple straight walking wheels are mainly used for forward and backward movement, and in the case where the at least one auxiliary steering wheel participates and cooperates with the multiple straight walking wheels, movements such as turning and rotation can be achieved. The driving mechanism may, for example, be a driving motor, and the driving motor is used to drive the walking wheels in the walking mechanism to achieve movement. In a specific implementation, the driving motor may, for example, be a reversible driving motor, and a speed change mechanism may also be provided between the driving motor and the axle of the walking wheel.

[0048] It can be understood that Figure 2The schematic structure does not specifically limit the robot 100. In some other embodiments of the present application, the robot 100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure can be implemented by hardware, software, or a combination of software and hardware.

[0049] The following takes Figure 1 the sensor of the robot 100 exemplified in

[0050] the depth camera (for example, a ToF sensor) and the obstacle 200 being a table as an example to describe the present application in more detail.

[0051] At this time, the depth camera 402 checks whether its direction is opposite to the forward direction of the robot 100. The checking method can be determined by processing the data it collects, or can be determined by pre-setting, that is, preset external parameters. If the checking result is completely opposite, it controls itself to stop data collection and at the same time pauses transmitting data to the processor 300 of the robot 100.

[0052] Similarly, the depth cameras 403 and 404 check whether they are not in the forward direction of the robot 100. The checking method can be determined by processing the data they collect, or can be determined by pre-setting, that is, preset external parameters. If the checking result is that they are not in the forward direction of the robot 100, they can control themselves to reduce the amount of data collected, for example, reducing the data collection frequency, reducing the resolution of the collected data, etc., and at the same time can also reduce the amount of data sent to the robot processor 300, for example, only sending data containing obstacle information to the robot processor 300.

[0053] After the depth camera 401 processes the collected data, it transmits the processed data to the processor 300 of the robot 100, and the processor 300 of the robot 100 performs subsequent processing such as data fusion. For example, a depth camera can obtain local two-dimensional obstacle map data through processing of point cloud data, and then send part of the two-dimensional obstacle map data to the robot processor 300. The robot processor 300 receives the local two-dimensional obstacle map data sent by multiple depth cameras and fuses the multiple local two-dimensional obstacle map data into a complete global two-dimensional obstacle distribution map.

[0054] The following combines the above Figure 2 shown structure and combines Figure 3 to describe the sensor data processing method of the present application; please refer to Figure 3 , which shows a flowchart of the sensor data processing method of the present application in an embodiment. As shown in the figure, the sensor data processing method includes the following steps:

[0055] Step S201, data of the environment around the robot collected by multiple sensors; in this embodiment, the depth cameras 401 to 404 collect image data of the environment around the robot 100. As mentioned above, the depth cameras 401 to 404 can be set on the robot 100 or in the working environment of the robot, and are used to regularly collect the environmental image data around the robot 100. In this embodiment, when the depth cameras 401 to 404 are set on the robot 100, the installation positions of the depth cameras on the robot 100 can be evenly distributed along the body circumference of the robot 100, and the image acquisition areas corresponding to each depth camera can have a certain overlap, so as to avoid blind spots in image data acquisition.

[0056] It can be understood that only 4 depth cameras for collecting the environmental data around the robot 100 are shown in the embodiment of the present application, but the number of depth cameras set on the robot 100 in the embodiment of the present application does not constitute a limitation on the number of sensors that can be set on the robot 100. The number of sensors set on the robot 100 can be more than 4, and the multiple sensors set on the robot 100 are not limited to the same type of sensor, and can be multiple sensors that collect the environmental data around the robot through different working principles. For the convenience of description, in the following embodiments of the present application, the multiple sensors are temporarily taken as depth cameras for illustration.

[0057] In this embodiment, the environmental image data collected by the depth cameras 401 to 404 can be of various types, including but not limited to: image data, infrared image data, point cloud data, point cloud confidence, etc. Here, the image data can support multiple resolutions, from low resolution to high resolution. The image data can be a color image or a black-and-white image, etc. The infrared image data is the image data obtained by measuring the heat radiated by an object. Compared with the visible light image data, the resolution, contrast, signal-to-noise ratio, visual effect, etc. of the infrared image data are all poor. The point cloud data is a set of sampled points with spatial coordinates obtained by the depth cameras 401 to 404. Since the quantity is large and dense, it is called a "point cloud". The sampled points include rich information, including but not limited to: three-dimensional coordinates (X, Y, Z), color, classification value, intensity value, timestamp, etc.

[0058] Step S202, the multiple sensors perform obstacle detection on the collected environmental data to generate local data; in the embodiment where the multiple sensors perform obstacle detection on the environmental data collected by the depth cameras, the depth cameras 401 to 404 respectively perform preprocessing on the collected point cloud data. Here, the number of sampled points in the point cloud data collected by the depth cameras 401 to 404 is too large and there is a lot of noise, so it is necessary to perform preprocessing on the collected point cloud data. In the embodiments of the present application, the preprocessing of the point cloud data can include but not limited to: point cloud denoising, point cloud simplification, point cloud registration, point cloud hole filling, etc. By preprocessing the above point cloud data, the noise and outliers in the point cloud data can be effectively removed, and the point cloud data can be simplified while maintaining geometric features, providing a robust data basis for subsequent point cloud data processing.

[0059] Specifically, the depth cameras 401 to 404 use an internal processor to complete the preprocessing of the point cloud data, and the internal processor completes the point cloud preprocessing by executing instructions related to preprocessing.

[0060] In one embodiment, the preprocessing performed by each of the sensors using its internal processor further includes that each sensor converts the coordinate system in the local data generated by it from the sensor coordinate system to the robot coordinate system according to a preset or acquired conversion relationship. In an embodiment where the multiple sensors are depth cameras disposed on the robot body, when performing obstacle detection on the environmental data collected by the depth cameras, after preprocessing the point cloud data, the depth cameras can further perform coordinate transformation on the processed point cloud data, that is, the depth cameras convert the point cloud data from the depth camera coordinate system to the robot coordinate system according to the conversion relationship determined by preset external parameters such as the position and angle of the depth cameras disposed on the robot body; through this coordinate system transformation, the point cloud data provided by the depth cameras enables the robot 100 to subsequently construct a global obstacle distribution map centered on the robot 100 body based on the point cloud data.

[0061] In an embodiment where the multiple sensors are depth cameras disposed outside the robot body, when performing obstacle detection on the environmental data collected by the depth cameras, after preprocessing the point cloud data, the depth cameras can further perform coordinate transformation on the processed point cloud data. In this embodiment, the conversion relationship between the sensor coordinate system and the robot coordinate system can be obtained through information obtained by the robot (such as the robot's positioning information, moving distance information, angle and pose information, etc.), for example, the robot obtains the conversion relationship between the sensor coordinate system and the robot coordinate system through the information provided by its own SLAM or VSLAM system, or inertial navigation module, or odometer module, etc.

[0062] In some embodiments of the present application, the preprocessing performed by each of the sensors using its internal processor can also perform point cloud data segmentation. Since each frame of point cloud data contains a large number of ground points, removing the point cloud on the ground is convenient for subsequent obstacle detection.

[0063] In some embodiments of the present application, the preprocessing performed by each of the sensors using its internal processor can also perform clustering analysis on the point cloud data, classifying the point cloud data into different types of point cloud data sets, and the same point cloud data set has similar or the same attributes. Here, usually an unsupervised clustering algorithm is used to form multiple point cloud data sets, and each point cloud data set represents a possible obstacle.

[0064] In an embodiment of performing step S202, the multiple sensors perform obstacle detection on the collected environmental data. In this embodiment, the depth cameras 401 to 404 respectively detect obstacles in the point cloud data. Specifically, the environmental perception by the depth camera can be divided into two levels. The low-level perception is also called obstacle detection, which only needs to detect that there is an obstacle ahead. The high-level perception can be regarded as target recognition, which requires further classification of the obstacle information. Obstacle detection refers to extracting potential obstacle objects from the point cloud data and obtaining their azimuth, size, shape, orientation and other information, which is generally described by adding a bounding box or a polygon. The so-called target recognition is to find the point cloud block with the highest similarity to the target point cloud in the point cloud data according to the standard target point cloud or the standard point cloud feature description vector. If the similarity meets the corresponding threshold condition, the point cloud block is recognized as the target point cloud.

[0065] In an embodiment of performing step S202, the multiple sensors perform obstacle detection on the collected environmental data and generate a local obstacle distribution map according to the obstacle detection result.

[0066] In an embodiment, the multiple sensors perform obstacle detection on the collected environmental data to obtain obstacle data, and construct a top view through the grid method to obtain local two-dimensional obstacle map data; in this embodiment, the obstacle detection result obtained after the depth camera detects obstacles in the point cloud data is the obstacle point cloud with a bounding box, and the bounding box is a polygon border. Projecting the obtained obstacle point cloud onto the image plane can obtain a two-dimensional obstacle distribution map. Specifically, the obstacle point cloud can be used to construct a top view through the grid method to obtain a local obstacle distribution map. Here, each depth camera can generate a corresponding local obstacle distribution map.

[0067] Please refer to Figure 4 , which shows a schematic diagram of the local obstacle distribution maps respectively generated by two sensors according to the obstacle detection results in an embodiment of the present application. In the embodiment shown in the figure, Figure 1Taking the depth camera 403 and the depth camera 404 as an example, when the robot 100 moves in a certain direction (as indicated by the arrow in the square representing the robot 100 in the figure), assuming that the depth camera 403 and the depth camera 404 perform obstacle detection on the collected environmental data at the moment of time stamp 0.5 s and respectively obtain the obstacle data of the obstacle O1 and the obstacle O2, the depth camera 403 and the depth camera 404 respectively construct a top view through the grid method to obtain two local two-dimensional obstacle map data (obtained by processing the point cloud data), and the time stamp information obtained, that is, 0.5 s, is carried in the map data. And based on the aforementioned conversion of the point cloud data from the depth camera coordinate system to the robot coordinate system in the depth camera 403 and the depth camera 404, the figures (a) and (b) in Figure 4 can be obtained respectively, where Figure 4 figure (a) in Figure 4 represents the local obstacle distribution map of the depth camera 403 at the current position;

[0068] In another embodiment, the multiple sensors perform obstacle detection on the collected environmental data to obtain obstacle data, and represent the type, coordinates and other information of the obstacles by using structured data, so as to obtain a local obstacle distribution map in a structured form.

[0069] In another embodiment of performing step S202, the step in which the multiple sensors perform obstacle detection on the collected environmental data to generate local data includes: generating local obstacle data according to the obstacle detection result. In this embodiment, each of the multiple sensors does not use its internal processor to construct its local obstacle distribution map, but each sensor generates the source data for constructing its local obstacle distribution map according to the obstacle detection result, so as to send the source data to the main processor of the robot, so that the main processor generates the local obstacle distribution map of each sensor at the current position after receiving the obstacle data sent by the multiple sensors. In this embodiment, still taking the depth camera as the sensor as an example, the obstacle detection is performed on the collected environmental data to generate point cloud data. It should be understood that the point cloud data is a set of sampling points carrying spatial coordinates and data of time stamps.

[0070] In this embodiment, each of the multiple sensors still uses its respective internal processor to perform the above-mentioned preprocessing. For example, the preprocessing of point cloud data may include, but is not limited to: point cloud denoising, point cloud simplification, point cloud registration, point cloud hole filling, etc.; the process of converting point cloud data from the depth camera coordinate system to the robot coordinate system; the process of segmenting and removing the point cloud on the ground; or / and the process of performing clustering analysis on the point cloud data, etc.

[0071] Step S203, the multiple sensors send the local data generated by each of them to the processor, where the processor is the main processor of the robot; in this embodiment, the main processor is, for example, the processor 300 described above with respect to Figure 2 the one described in.

[0072] In one embodiment, before the multiple sensors send the local data generated by each of them to the processor, the preprocessing performed by each sensor using its internal processor further includes that the step of the multiple sensors sending the local data generated by each of them to the processor also includes dividing the local data to be generated by each sensor into partial local data with different priorities according to a preset rule. Specifically, the conditions of the preset rule include whether the local data generated by the sensor contains an obstacle and the distance between the obstacle and the robot.

[0073] Still taking Figure 1 or Figure 2 the depth cameras 401 to 404 shown in as an example, the depth cameras 401 to 404 respectively set different priorities for partial point cloud data in the point cloud data obtained by each of them. Here, taking the depth camera 401 as an example, the point cloud data of the depth camera 401 is divided into multiple partial point cloud data by its internal processor. Different partial point cloud data may or may not contain an obstacle. Among them, the distance between the obstacle and the robot in the partial point cloud data containing an obstacle is not necessarily the same. In this embodiment, different priorities can be set for the partial point cloud data according to whether the partial point cloud data contains an obstacle and the distance between the obstacle and the robot. For example, partial point cloud data A does not contain an obstacle, partial point cloud data B contains an obstacle, the distance between the obstacle and the robot is 3 meters, and partial point cloud data C contains an obstacle, the distance between the obstacle and the robot is 7 meters. Then, the priority of partial point cloud data A can be set to 3, the priority of partial point cloud data B can be set to 1, and the priority of partial point cloud data C can be set to 2. The smaller the priority value, the higher the priority. That is, the order of priorities from high to low is partial point cloud data B, partial point cloud data C, and partial point cloud data A.

[0074] The step in which the multiple sensors send the locally generated data to the processor further includes that each sensor sends the locally generated data to the processor according to its preset priority. In an embodiment taking the depth camera 401 as an example, the depth camera 401 first sends a part of the point cloud data B to the main processor of the robot according to the determined priority order, then sends a part of the point cloud data C to the main processor of the robot, and finally sends a part of the point cloud data A to the main processor of the robot. Here, the priority of the part of the point cloud data reflects the importance of this part of the point cloud data to the robot. The higher the priority, the more important it is. For example, the distance between the obstacle in the part of the point cloud data B and the robot is less than the distance between the obstacle in the part of the point cloud data C and the robot. Therefore, the part of the point cloud data B is sent to the processor 300 of the robot first and is preferentially processed by the processor 300, so as to avoid the collision between the obstacle and the robot.

[0075] In some embodiments, the depth cameras 401 to 404 respectively send part of the point cloud data to the processor 300 according to the priority of the part of the point cloud data. Here, the part of the point cloud data with a higher priority is sent to the processor 300 first, and the part of the point cloud data with a lower priority waits until the part of the point cloud data with a higher priority is sent and then sends to the processor 300 or does not send. For example, in the above example, the priority of the part of the point cloud data A is 3, the priority of the part of the point cloud data B is 1, and the priority of the part of the point cloud data C is 2. Then the part of the point cloud data B is sent first, the part of the point cloud data C is sent second, and the part of the point cloud data A is sent last or may not be sent.

[0076] In some embodiments of the present application, the depth cameras 401 to 404 may further include buffers for respectively buffering part of the point cloud data within a period of time, and then sending the buffered part of the point cloud data according to the corresponding priority.

[0077] In another embodiment, before the multiple sensors send the locally generated data to the processor, each sensor is also preset with a priority, and each sensor sends the local data to the processor of the robot according to its own preset priority.

[0078] In the example where the multiple sensors are independent of the robot body and are arranged at certain positions in the robot working environment, the conditions for presetting the priority of each sensor consider factors such as the position of each sensor on the current traveling route of the robot. For example, when the robot is running forward, the priority of the sensor located on the traveling route of the robot (such as the sensor set on the aisle or column in front of the walking direction of the robot) is higher than that of the sensors at other positions, and other positions such as the sensors at certain positions behind the robot.

[0079] In an example where the multiple sensors are disposed on the robot body, the condition for presetting the priority of each sensor is the position where the sensor is disposed on the robot body. For example, the priority of the sensor disposed on the front side of the robot body is higher than that of the sensors disposed on the left and right sides of the robot body, and the priority of the sensors disposed on the left and right sides of the robot body is higher than that of the sensors disposed on the rear side of the robot body. Taking Figure 1 the walking direction of the shown robot and the position where the depth camera is installed as an example, the preset priority for the depth camera 401 to send local data to the main processor of the robot is higher than that of the depth cameras 403 and 404, and the preset priority for the depth cameras 403 and 404 to send local data to the main processor of the robot is higher than that of the depth camera 402; or, when the robot walks along the wall or around the edge on the left side, the preset priority for the depth camera 403 to send local data to the main processor of the robot is higher than that of the depth camera 404; and when the robot walks along the wall or around the edge on the right side, the preset priority for the depth camera 404 to send local data to the main processor of the robot is higher than that of the depth camera 403.

[0080] As described in the above embodiments, in step S203, the multiple sensors are sent to the processor of the robot in the order from high to low according to their own priorities or the priorities of some of the local data produced by themselves.

[0081] In an example where the multiple sensors are disposed on the robot body, each of the sensors has a communication module to send local data to the robot by means of wired transmission. For example, each of the depth cameras 401 to 404 has a communication module, and the communication module is used to implement two-way communication with the processor 300. Specifically, communication can be performed through, for example, a network port, a USB interface, a serial port, a Controller Area Network (CAN) bus, etc.

[0082] In an example where the multiple sensors are independent of the robot body and are disposed at certain positions in the robot working environment, each of the sensors has a wireless communication module for sending local data to the robot by means of wireless transmission.

[0083] Step S204, the processor processes the received multiple local data into a global obstacle distribution map.

[0084] In one embodiment, that is, in the above embodiment where the multiple sensors in step S202 detect obstacles in the collected environmental data and generate a local obstacle distribution map according to the obstacle detection result, in step S204, the processor fuses the received multiple local obstacle distribution maps into a global obstacle distribution map at the current position.

[0085] In another embodiment, that is, in the embodiment where each of the multiple sensors described in step S202 above does not use its internal processor to construct its local obstacle distribution map, but each sensor generates source data for constructing its local obstacle distribution map according to the detection result of the obstacle, the step S204 of the processor processing the received multiple pieces of local data into a global obstacle distribution map includes: the main processor receives the local obstacle data sent by the multiple sensors to generate a local obstacle distribution map of each sensor at the current position; and fuses and processes the multiple local obstacle distribution maps to generate a global obstacle distribution map at the current position.

[0086] In an embodiment, the processor fuses and processes the received multiple pieces of local data into a global obstacle distribution map at the current position of the robot. Still taking the Figure 1 or Figure 2 embodiment shown in as an example, the processor 300 receives the local obstacle distribution maps sent by the depth cameras 401 to 404 respectively, and fuses the received multiple local obstacle distribution maps into a complete global obstacle distribution map. In actual applications, the processor 300 of the robot can use a variety of fusion algorithms for the fusion of multiple local obstacle distribution maps. For example, in one embodiment, the processor fuses and processes the received multiple pieces of local data into a global obstacle distribution map using the probability occupancy grid map algorithm.

[0087] In addition, the processor 300 can use partial point cloud data from the depth cameras 401 to 404 respectively to correct the obstacle information in the local obstacle distribution map, obtain more accurate obstacle information, and then update the obstacle information in the global obstacle distribution map.

[0088] In one embodiment, the processor of the robot calculates the displacement information of the robot at the current timestamp based on the timestamps carried in the local data generated by each sensor, and updates the obstacle information in the local data according to the displacement information; superimposes the updated local data of the multiple sensors to form a global obstacle distribution map at the current position. As Figure 4In the illustrated embodiment, the timestamps carried in the local data generated by each of the sensors. For example, when the processor of the robot receives local data sent by a sensor, the processor of the robot calculates the displacement information of the robot at the current timestamp based on the timestamp carried in the local data generated by the sensor, and displaces the positions of the obstacles in the local obstacle distribution map to the calculated positions. That is, the main processor updates the local obstacle distribution map, and then superimposes the local obstacle distribution maps of multiple sensors to form the global obstacle distribution map of the robot at the current position.

[0089] Please refer to Figure 5 , which shows a schematic diagram of the processor in an embodiment of the present application processing the obtained local obstacle distribution map. In Figure 5 it continues with Figure 4 the local obstacle distribution maps obtained by the depth camera 403 and the depth camera 404 shown in Figure 5Figures (c) and (d) in []. At this time, since the depth cameras 403 and 404 have converted the coordinate system of the local data (the local two-dimensional obstacle map in this embodiment) from the sensor coordinate system to the robot coordinate system in their respective internal processors; moreover, the local two-dimensional obstacle maps of the depth cameras 403 and 404 have the same timestamp (1.2 s). In other words, the local two-dimensional obstacle maps of the depth cameras 403 and 404 have the same timestamp (1.2 s) and the same coordinate system (the robot coordinate system). The main processor superimposes Figures (c) and (d) in the figure under the robot coordinate system, that is, places Figures (c) and (d) in the same dimension (the time dimension and the space dimension represented by the coordinate system) for superimposition to obtain Figure (e). Figure (e) is the global obstacle distribution map at the current position after the fusion processing of the robot 100.

[0090] It should be understood that the above is only an example with two sensors, the depth cameras 403 and 404, and two timestamps (0.5 s and 1.2 s). In an actual application scenario, when the robot is in the working state, the multiple sensors and the main processor of the robot will continuously perform the above process, that is, the robot will obtain the global obstacle distribution map at each position. As the robot traverses the entire working scenario, the obstacle distribution map of the entire working scenario is finally formed.

[0091] The processor 300 controls the movement of the robot 100 according to the global obstacle distribution map and the robot task. Here, the processor 300 has obtained the global obstacle distribution maps at different positions and the obstacle distribution map of the entire working scenario, that is, it can determine the movement path of the robot 100 according to the task to be completed by the robot 100 currently, such as destination navigation, item delivery, and covering cleaning, and control the current movement of the robot 100 according to the movement path. For example, control the robot 100 to move forward, move left, move right, etc., so that the robot 100 can complete the established task.

[0092] Please refer to Figure 6 , which shows the flowchart of the sensor data processing method of the present application in another embodiment. As shown in the figure, in combination with the above Figure 2 shown structure and specific scenario, the sensor data processing method includes the following steps:

[0093] Step S301, the depth cameras 401 to 404 collect image data of the environment around the robot 100.

[0094] Step S302, the depth cameras 401 to 404 respectively preprocess the collected point cloud data.

[0095] Step S303: The depth cameras 401 to 404 respectively detect obstacles in the point cloud data.

[0096] Step S304: The depth cameras 401 to 404 generate local obstacle distribution maps according to the obstacle detection results.

[0097] Step S305: The depth cameras 401 to 404 respectively set different priorities for some of the point cloud data in the point cloud data.

[0098] Step S306: The depth cameras 401 to 404 respectively send the local obstacle distribution maps to the processor 300 and send some of the point cloud data according to the priorities.

[0099] Step S307: The processor 300 fuses multiple local obstacle distribution maps to obtain a global obstacle distribution map.

[0100] Step S308: The processor 300 controls the movement of the robot 100 according to the global obstacle distribution map and the robot task.

[0101] In the above embodiments, the sensor data processing method for a system including a robot and sensors collects data on the environment around the robot through multiple sensors, respectively performs obstacle detection on the collected environmental data, then generates local obstacle distribution maps according to the obstacle detection results, and further sends the local obstacle distribution maps generated by the multiple sensors to the processor of the robot. The processor fuses the received multiple local obstacle distribution maps into a global obstacle distribution map, thereby realizing the processing of the collected data on multiple sensors and then sending the processed data to the processor, reducing the amount of data transmitted between the multiple sensors and the processor, reducing the pressure on the transmission bandwidth, improving the processing efficiency of the processor, and reducing the consumption of computing resources of the processor.

[0102] In one embodiment, the sensor data processing method of the present application further includes the step of the processor adjusting the working modes of one or more of the sensors according to the working state of the robot; the working modes include data acquisition modes or / and data sending modes. In this embodiment, adjusting the working modes of one or more of the sensors includes adjusting the data acquisition frequency, adjusting the data resolution, or adjusting the data sending frequency.

[0103] Please refer to Figure 7 , which shows a flowchart of the method for the processor of the robot to adjust the data acquisition mode of the sensor in one embodiment. As shown in the figure, the scheme for the processor 300 to adjust the data acquisition modes of the depth cameras 401 and 402 includes:

[0104] Step S401: The processor 300 determines the depth camera for which the data acquisition method needs to be adjusted according to the working state of the robot 100. Here, the working state of the robot 100 may include, but is not limited to: going straight, turning left, turning right, "zigzag" walking, walking along the wall, etc. The "zigzag" walking here is a working state in the cleaning task of the robot 100, which is used to achieve sequential area cleaning. Walking along the wall is also a working state in the cleaning task, which is used to achieve cleaning or mopping along the wall, and can be divided into cleaning along the left wall and cleaning along the right wall.

[0105] In some embodiments of the present application, according to different working states of the robot 100, the data acquisition methods of different depth cameras can be adjusted. Here, among different working states of the robot 100, the importance of multiple depth cameras is different. The robot 100 has a higher demand for sending data from depth cameras with high importance and a lower demand for sending data from depth cameras with low importance. Therefore, the processor 300 of the robot 100 can choose to receive more data sent by depth cameras with high importance and less data sent by depth cameras with low importance. The following takes the depth camera 401 as the camera in front of the moving direction of the robot 100 and the depth camera 402 as the camera behind the moving direction of the robot 100 as an example for illustration.

[0106] When the working state of the robot 100 is going straight, the depth camera 401 and the depth camera 402 are respectively used to collect data in front of and behind the moving direction of the robot 100. The importance of the data collected by the depth camera 401 is high, and the importance of the data collected by the depth camera 402 is low. Therefore, the processor 300 determines the depth camera 401 and the depth camera 402 as the depth cameras for which the data acquisition method needs to be adjusted.

[0107] Step S402: The processor 300 generates control instructions for the depth cameras for which the data acquisition method needs to be adjusted. Here, the processor 300 generates corresponding control instructions for the depth cameras 401 and 402 for which the data acquisition method needs to be adjusted respectively. The control instructions are used to indicate how to adjust the data acquisition method of the depth cameras. The data acquisition method may include, but is not limited to: one or more of data acquisition frequency, data resolution, and data transmission frequency. For example, the control instructions generated by the processor 300 for the depth camera 401 include increasing the data resolution, increasing the data acquisition frequency, or / and the data transmission frequency, etc., and the control instructions generated for the depth camera 402 include decreasing the data resolution, decreasing the data acquisition frequency, or / and the data transmission frequency, etc.

[0108] Step S403: The processor 300 sends the control instructions for adjusting the data acquisition method to the depth cameras 401 and 402.

[0109] In steps S404 and S405, the depth cameras 401 and 402 respectively adjust the data acquisition method according to the control instructions. Here, the control instruction received by the depth camera 401 adjusts the data acquisition method in the direction of quality improvement, and the control instruction received by the depth camera 402 adjusts the data acquisition method in the direction of quality reduction.

[0110] In steps S406 and S407, the depth cameras 401 and 402 respectively send data according to the adjusted data acquisition method. Here, the current data acquisition method of the depth camera 401 has improvements in quality, acquisition frequency, and transmission frequency compared to the original data acquisition method, and the current data acquisition method of the depth camera 402 has reductions in quality, acquisition frequency, and transmission frequency compared to the original data acquisition method.

[0111] Through the above steps, the processor 300 can concentrate computing resources on processing more important data sent by the depth camera, thereby effectively utilizing the network bandwidth between the processor 300 and the depth camera and the computing resources of the processor 300.

[0112] Please refer to Figure 8 , which shows a schematic block diagram of the internal processor of the sensor described in this application in an embodiment. As shown in the figure, the depth camera 401 has a computing and processing device, such as a System on Chip (SoC), etc. In this embodiment, similar components have the same reference numerals. Additionally, the dashed box is an optional feature of a more advanced SoC. In Figure 8 , the SoC 1000 includes: an interconnect unit 1050, which is coupled to the processor 1010; a system agent unit 1070; a bus controller unit 1080; an integrated memory controller unit 1040; one or a group of co-processors 1020, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a Static Random Access Memory (SRAM) unit 1030; and a Direct Memory Access (DMA) unit 1060. In one embodiment, the co-processor 1020 includes a dedicated processor, such as a network or communication processor, a compression engine, a GPGPU, a high-throughput MIC processor, or an embedded processor, etc.

[0113] On the other hand, this application also provides a system including a robot and sensors, where the system includes multiple sensors, a memory, and a processor. In this application, both the memory and the processor are the memory and the processor of the computing device provided on the robot body.

[0114] The multiple sensors are used to collect data of the environment around the robot, and each of the sensors performs obstacle detection on the collected environmental data to generate local data and send it to the processor; in an embodiment, the sensors may include, for example, a laser sensor, an ultrasonic sensor, an infrared sensor, an optical camera (such as a monocular camera or a binocular camera), a depth camera (such as a ToF sensor), a millimeter-wave radar sensor, etc.

[0115] The memory is used to store instructions executed by one or more processors of the robot; in an embodiment, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. In some embodiments, the one or more memories may further include a memory remote from one or more processors, such as a network-attached memory accessed via an RF circuit or an external port and a communication network, where the communication network may be the Internet, one or more intranets, a local area network, a wide area network, a storage area network, etc., or a suitable combination thereof. A memory controller may control access to the memory by other components of the device, such as a CPU and a peripheral interface.

[0116] The processor is the main processor of the robot and is used to process the received multiple local data into a global obstacle distribution map. In an embodiment, the processor 300 may be used to read and execute computer-readable instructions. In a specific implementation, the processor 300 may mainly include a controller, an arithmetic unit, and registers. Among them, the controller is mainly responsible for instruction decoding and issuing a control signal for the operation corresponding to the instruction. The arithmetic unit is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations, etc., and may also perform address operations and conversions. The registers are mainly responsible for storing register operands and intermediate operation results temporarily stored during the execution of instructions, etc. In a specific implementation, the hardware architecture of the processor 300 may be an application-specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, or an NP architecture, etc.

[0117] The processor 300 may include one or more processing units. For example, the processor 300 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0118] The processor may be configured to receive the point cloud data, image data, and local obstacle distribution map data sent by the multiple sensors, fuse these data to obtain a complete two-dimensional obstacle distribution map, and then control the robot to complete specific tasks such as item delivery, room cleaning / washing, etc. according to the two-dimensional obstacle distribution map.

[0119] Please refer to Figure 9 , which shows a system block diagram of a system including a robot and sensors in an embodiment of the present application. As shown in the figure, in this embodiment, the system including a robot and sensors refers to a robot system with multiple sensors disposed on the robot body. Specifically, according to the functional requirements of the mobile robot, multiple sensors 401, 402, 403, and 404 are disposed on the robot body 400. The computing device 405 in the robot body 400 includes the memory 406 and the processor 407. For example, 4 sensors such as 4 sensors 401, 402, 403, and 404 are disposed on the illustrated robot body 400 to continuously detect the surrounding environment; in an actual implementation manner, the positions of the sensors disposed on the robot body will be set at corresponding positions according to the functions of the respective sensors. For example, a fisheye camera or a laser sensor is disposed on the top of the robot body, a binocular vision camera is disposed on the front side of the robot, ultrasonic sensors are disposed on the left and right sides or one side of the robot, and a laser sensor or a ToF sensor is disposed on the front or rear side of the robot, etc.

[0120] Please refer to Figure 10, which shows the system block diagram of the system including a robot and sensors in another embodiment of the present application. As shown in the figure, in this embodiment, the system including a robot and sensors refers to a wireless communication system formed by sensors 401, 402, 403, and 404 being arranged at positions outside the robot body 400. The computing device 405 in the robot body 400 includes the memory 406 and the processor 407. For example, in the working environment of the robot, for example, the sensors 401, 402, 403, and 404 are arranged in physical spaces such as the walkways / passages where the robot often cruises, on walls or columns, etc. Moreover, the sensors 401, 402, 403, and 404 are sensors arranged on other electronic devices independent of the mobile robot, such as sensors on user terminal devices (such as smartphones, tablets, surveillance cameras, smart screens, etc.). For example, by binding a smartphone to the robot, the sensors of the smartphone can exchange data with the robot. In these cases, the system consists of a robot and sensors arranged externally, and the data transmission between the robot and the external sensors is carried out through wireless communication methods, which can include but are not limited to wireless communication methods such as Wireless Fidelity (Wi-Fi) and Bluetooth (Blue Tooth).

[0121] The present application further provides a mobile robot in another aspect. Please refer to Figure 11 , which shows the block diagram of the mobile robot of the present application in an embodiment. As shown in the figure, the mobile robot of the present application includes multiple sensors 401, 402, 403, and 404, a mobile device 408, a storage device 406, and a processing device 407.

[0122] The multiple sensors 401, 402, 403, and 404 are arranged on the robot body 400 to collect data of the environment around the robot. Each of these sensors performs obstacle detection on the collected environmental data to generate local data and sends it to the processor; in the embodiment, the multiple sensors arranged on the robot body, for example, include a laser sensor, an ultrasonic sensor, an infrared sensor, an optical camera (such as a monocular camera or a binocular camera), a depth camera (such as a ToF sensor), a millimeter-wave radar sensor, etc.

[0123] The mobile device 408 is used to perform mobile operations; in the embodiment, the mobile device 408 is used to control the movement of the robot according to the instructions issued by the processor. The mobile device may include multiple components related to movement, such as motors, drive shafts, wheels, etc. In some embodiments of the present application, the mobile device 408 is used to implement various movement forms of the robot, such as forward movement, backward movement, left movement, right movement, arcuate movement, etc. to complete specific tasks such as item delivery, room cleaning / washing, etc.

[0124] The computing device 405 of the robot includes a storage device 406 and a processing device 407. Among them, the storage device 406 is used to store at least one program; the processing device 407 is connected to the multiple sensors 401, 402, 403, and 404, the mobile device 408, and the storage device 406, and is used to execute the at least one program to execute at least one of the embodiments described above for the sensor data processing method, such as Figure 3 any of the embodiments described in any of the corresponding embodiments; or execute at least one of the embodiments described above for the sensor data processing method, such as Figure 6 or Figure 7 any of the embodiments described in any of the corresponding embodiments.

[0125] On the other hand, this application also provides a computer-readable storage medium storing a computer program. When the computer program is executed, the device where the storage medium is located implements at least one of the embodiments described above for the sensor data processing method, such as Figure 3 any of the embodiments described in any of the corresponding embodiments; or when the computer program is executed, the device where the storage medium is located implements at least one of the embodiments described above for the sensor data processing method, such as Figure 6 or Figure 7 any of the embodiments described in any of the corresponding embodiments.

[0126] If the above function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application.

[0127] In the embodiments provided in the present application, the computer-readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disc storage device, a magnetic disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if the instructions are sent from a website, a server, or other remote source using coaxial cables, fiber optic cables, twisted pairs, digital subscriber lines (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cables, fiber optic cables, twisted pairs, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended to refer to non-transient, tangible storage media. As used in the application, magnetic disks and optical discs include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where magnetic disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0128] In one or more exemplary aspects, the functions described by the computer program of the method of the present application can be implemented in a manner of hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored or transmitted as one or more instructions or codes onto a computer-readable medium. The steps of the method or algorithm disclosed in the present application can be embodied by a processor-executable software module, where the processor-executable software module can be located on a tangible, non-transitory computer-readable and writable storage medium. The tangible, non-transitory computer-readable and writable storage medium can be any available medium that can be accessed by a computer.

[0129] The flowcharts and block diagrams in the above-mentioned accompanying drawings of the present application illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Based on this, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0130] In summary, in one of the above-disclosed embodiments of the present application, the sensor data processing method for a system including a robot and sensors collects data on the environment around the robot through multiple sensors, respectively performs obstacle detection on the collected environmental data, generates local data based on the obstacle detection results, further sends the local data generated by the multiple sensors to the processor of the robot, and the processor fuses the received multiple local data into a global obstacle distribution map, thereby realizing the processing of the collected data on multiple sensors, and then sending the processed data to the processor, reducing the amount of data transmitted between the multiple sensors and the processor, reducing the pressure on the transmission bandwidth, improving the processing efficiency of the processor, and reducing the consumption of computing resources of the processor.

[0131] According to the descriptions of the above examples, the present application provides multiple embodiments as follows:

[0132] 1. A sensor data processing method for a system including a robot and sensors, wherein it includes:

[0133] Collect data on the environment around the robot through multiple sensors;

[0134] The multiple sensors perform obstacle detection on the collected environmental data and generate a local obstacle distribution map based on the obstacle detection results;

[0135] The multiple sensors send the respective generated local obstacle distribution maps to a processor, where the processor is the main processor of the robot;

[0136] The processor fuses the received multiple local obstacle distribution maps into a global obstacle distribution map.

[0137] 2. The method according to Embodiment 1, wherein the sensor is a depth camera, and is disposed on the robot or in the working environment of the robot, and the depth camera is used to collect point cloud data of the environment around the robot.

[0138] 3. The method according to Embodiment 2, wherein the method further comprises:

[0139] The depth camera performs coordinate conversion on the point cloud data to convert the point cloud data from the depth camera coordinate system to the robot coordinate system.

[0140] 4. The method according to Embodiment 1, wherein the method further comprises:

[0141] The depth camera divides the point cloud data into multiple partial point cloud data, and sets corresponding priorities for the multiple partial point cloud data.

[0142] 5. The method according to Embodiment 4, wherein setting corresponding priorities for the multiple partial point cloud data comprises:

[0143] The depth camera determines the priorities corresponding to the partial point cloud data according to whether the partial point cloud data contains obstacles and the distance between the obstacles and the robot.

[0144] 6. The method according to Embodiment 4, wherein after setting corresponding priorities for the multiple partial point cloud data, it further comprises:

[0145] The depth camera sends the partial point cloud data to the processor according to the priorities corresponding to the partial point cloud data.

[0146] 7. The method according to Embodiment 2, wherein the method further comprises:

[0147] The processor determines the depth camera for which the data acquisition method needs to be adjusted according to the working state of the robot;

[0148] The processor controls the determined depth camera to adjust the data acquisition method;

[0149] The determined depth camera sends data to the processor according to the adjusted data acquisition method.

[0150] 8. The method according to Embodiment 7, wherein the data acquisition method comprises one or more of the following combinations: data acquisition frequency, data resolution, data sending frequency.

[0151] 9. A system comprising a robot and a sensor, wherein it comprises:

[0152] Multiple sensors, configured to collect data of the environment around the robot, perform obstacle detection on the collected environmental data, generate a local obstacle distribution map based on the obstacle detection result, and then send the generated local obstacle distribution map to a processor;

[0153] A memory, configured to store instructions executed by one or more processors of the robot, and

[0154] A processor, which is one of the processors of the robot, configured to fuse the received multiple local obstacle distribution maps into a global obstacle distribution map.

[0155] 10. A readable storage medium, wherein instructions are stored on the readable storage medium, and when the instructions are executed on a system including a robot and sensors, the system is caused to execute the sensor data processing method according to any one of Embodiments 1-8.

[0156] The above embodiments are only illustrative of the principles and effects of the present application, and are not intended to limit the present application. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed in the present application should still be covered by the claims of the present application.

Claims

1. A method for processing sensor data, used in a system including a robot and a sensor, characterized in that, Including the following steps: Collecting data of the environment around the robot through multiple sensors; The multiple sensors performing obstacle detection on the collected environmental data to generate local data; The multiple sensors sending the locally generated data to a processor, where the processor is the main processor of the robot; wherein, the step of the multiple sensors sending the locally generated data to the processor further includes dividing the locally generated data of each sensor into partial local data with different priorities according to a preset rule and sending the partial local data to the processor in the order from high to low according to the priority of the partial local data, and the conditions of the preset rule include whether the locally generated data of the sensor contains an obstacle and the distance between the obstacle and the robot; The processor processing the received multiple pieces of local data into a global obstacle distribution map; wherein, the step of the processor processing the received multiple pieces of local data into a global obstacle distribution map includes inferring the displacement information of the robot at the current timestamp based on the timestamps carried in the locally generated data of each sensor, and updating the obstacle information in the local data according to the displacement information; superimposing the updated local data of the multiple sensors to form a global obstacle distribution map of the current position.

2. The sensor data processing method according to claim 1, wherein The step of the multiple sensors performing obstacle detection on the collected environmental data to generate local data includes: generating a local obstacle distribution map according to the obstacle detection result.

3. The sensor data processing method according to claim 2, wherein The step of generating a local obstacle distribution map according to the obstacle detection result includes: the multiple sensors performing obstacle detection on the collected environmental data to obtain obstacle data, and constructing a top view through a grid method to obtain local two-dimensional obstacle map data.

4. The sensor data processing method according to claim 2, wherein The step of generating a local obstacle distribution map according to the obstacle detection result includes: using structured data to represent the type and coordinate information of the obstacle to obtain a local obstacle distribution map in a structured form.

5. The sensor data processing method according to claim 2, characterized in that The processor fusing the received multiple local obstacle distribution maps into a global obstacle distribution map of the current position.

6. The sensor data processing method according to claim 1, wherein The step of the multiple sensors performing obstacle detection on the collected environmental data to generate local data includes: generating local obstacle data according to the obstacle detection result.

7. The sensor data processing method according to claim 6, characterized in that, The step of the processor processing the received multiple pieces of local data into a global obstacle distribution map includes: the main processor receiving the local obstacle data sent by the multiple sensors to generate a local obstacle distribution map of each sensor at the current position; and performing a fusion process on the multiple local obstacle distribution maps to generate a global obstacle distribution map of the current position.

8. The sensor data processing method according to claim 1, wherein The sensor is one or more of a laser sensor, an ultrasonic sensor, an infrared sensor, an optical camera, a depth camera, and a millimeter-wave radar sensor.

9. The sensor data processing method according to claim 1, wherein The sensor is arranged on the robot or in the working environment of the robot for collecting local data of the environment around the robot.

10. The sensor data processing method according to claim 1, wherein In the step where the multiple sensors perform obstacle detection on the collected environmental data to generate local data, it further includes: each of the sensors converts the coordinate system in the local data generated by it from the sensor coordinate system to the robot coordinate system according to a preset or acquired conversion relationship.

11. The sensor data processing method according to claim 1, wherein In the step where the multiple sensors send the local data generated by them to the processor, it further includes that each of the sensors sends the local data generated by it to the processor according to its preset priority.

12. The sensor data processing method according to claim 11, wherein The conditions for presetting the priority of each of the sensors include the position of each of the sensors in the working environment where the robot is located or the position where the sensor is set on the robot body.

13. The sensor data processing method according to claim 1, wherein It further includes the step where the processor adjusts the working modes of one or more of the sensors according to the working state of the robot; the working modes include data acquisition modes or / and data sending modes.

14. The sensor data processing method according to claim 1, characterized in that Adjusting the working modes of the one or more sensors includes adjusting the data acquisition frequency, adjusting the data resolution, or adjusting the data sending frequency.

15. The sensor data processing method according to claim 1, wherein The processor fuses and processes the received multiple pieces of local data into a global obstacle distribution map using the probability occupancy grid map algorithm.

16. A system comprising a robot and a sensor, characterized in that, It includes: Multiple sensors, which are used to collect data of the environment around the robot. Each of the sensors performs obstacle detection on the collected environmental data to generate local data and sends it to the processor; among them, the way for the multiple sensors to send the local data generated by them to the processor further includes dividing the local data generated by each of the sensors into partial local data with different priorities according to a preset rule, and sending the partial local data to the processor in the order from high to low according to the priority of the partial local data. The conditions of the preset rule include whether the local data generated by the sensor contains obstacles and the distance between the obstacles and the robot. A memory, which is used to store instructions executed by one or more processors of the robot; and A processor, which is the main processor of the robot and is used to process the received multiple pieces of local data into a global obstacle distribution map; among them, the way for the processor to process the received multiple pieces of local data into a global obstacle distribution map includes inferring the displacement information of the robot at the current timestamp according to the timestamps carried in the local data generated by each of the sensors, and updating the obstacle information in the local data according to the displacement information; superimposing the updated local data of the multiple sensors to form a global obstacle distribution map at the current position.

17. A mobile robot, characterized in that, It includes: Multiple sensors, which are used to collect data of the environment around the robot. Each of the sensors performs obstacle detection on the collected environmental data to generate local data and sends it to the processor; A mobile device, which is used to perform a moving operation; A storage device, which is used to store at least one program; A processing device, which is connected to the multiple sensors, the mobile device, and the storage device, and is used to execute the at least one program to perform the sensor data processing method as described in any one of claims 1-15.

18. The mobile robot according to claim 17, wherein The mobile robot includes a commercial cleaning robot.

19. A readable storage medium, characterized in that, A storage medium stores at least one computer program which, when run by a processor, controls a device where the storage medium is located to execute the sensor data processing method according to any one of claims 1 to 15.

Images