Robot semantic map construction method, device and system
By combining real-time data of the robot terminal and video data for semantic annotation, the problems of high visual capabilities of robot hardware and poor semantic map quality in the existing technology are solved, and high-accurate semantic map construction is achieved, reducing hardware cost and difficulty .
Patent Information
- Application Number
- CN202410833593.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-11-08
AI Technical Summary
The existing robot semantic map construction methods rely on high-cost and difficult hardware vision sensors, resulting in high hardware requirements and poor semantic map quality, which cannot meet the functional requirements of complex scenarios.
By obtaining real-time data when the robot terminal performs a scan task in the target space and video data captured by the monitoring system, combining real-time position and posture data, semantic annotation is performed to build a semantic map, reducing dependence on the robot hardware visual ability and improving semantics The accuracy of the map.
The construction of semantic maps that do not rely on the hardware visual capabilities of the robot terminal is realized, which reduces the hardware cost and difficulty, improves the accuracy and quality of semantic maps, and meets the functional needs of complex scenarios.
Smart Images

Figure CN118913269A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of map construction technology, and in particular to a method, device and system for constructing a robot semantic map. Background Art
[0002] From the perspective of mobility, robots can be divided into non-mobile robots and mobile robots. In recent years, with the development of artificial intelligence, mobile robots have been used more and more widely in people's daily lives, and their application sites have become more and more diverse. From commercial buildings, shopping malls and supermarkets to hospitals, outdoor public places, subway stations, etc., they can all become areas where mobile robots can move. Mobile robots usually have autonomous mobility functions. The premise of autonomous mobility is autonomous navigation. Autonomous navigation means that mobile robots can autonomously plan their movement routes according to specific algorithms and rules to avoid obstacles or move along specific routes. Map construction is a key link for robots to realize functions such as autonomous navigation. At present, most of the robotics field uses raster mapping to construct two-dimensional raster maps as a basis for positioning, navigation, and task execution. However, the two-dimensional raster map itself does not have semantic information and cannot effectively support more complex scene function requirements.
[0003] Based on this, some existing semantic map construction methods rely on the visual sensors carried by the robot, such as laser scanners, radars, and depth cameras, to collect environmental information and perform semantic analysis to build semantic maps. However, these methods have high requirements for the robot hardware, which increases the cost and difficulty of robot manufacturing. In addition, the perception range of the robot sensors is limited, and the collected data may be sparse or incomplete, which affects the accuracy of object recognition and semantic annotation, resulting in poor quality of the constructed semantic map. Summary of the invention
[0004] In this embodiment, a robot semantic map construction method, device and system are provided to solve the problems of semantic map construction cost and poor quality in related technologies.
[0005] In a first aspect, a robot semantic map construction method is provided in this embodiment. The method is applicable to a robot semantic map construction system, which includes a robot terminal, a robot platform and a monitoring system. The method is applied to the robot platform and includes:
[0006] Acquire real-time data generated when the robot terminal performs a scanning task in the target space; the real-time data includes a grid map of the target space, real-time position data and real-time posture data of the robot terminal;
[0007] Obtaining visual information generated based on raw video data in the monitoring system; the raw video data is video data obtained by the monitoring system shooting the target space when the robot terminal performs the image scanning task; the visual information includes a correlation relationship between environmental information of the target space and position information of the robot terminal;
[0008] Based on the real-time data and the visual information, the grid map is semantically annotated to obtain a semantic map.
[0009] In some embodiments, the real-time data generated when the robot terminal performs a scanning task in the target space is obtained; the real-time data includes a grid map of the target space, real-time position data and real-time posture data of the robot terminal, including:
[0010] Controlling the robot terminal to perform a scanning task;
[0011] After the image scanning task is completed, a grid map of the target space generated by the robot terminal, as well as real-time position data and real-time posture data of the robot terminal in a preset time period are obtained from the robot terminal.
[0012] In some of the embodiments, the monitoring system includes a monitoring platform and a fixed-position monitoring device;
[0013] Wherein, the visual information generated based on the original video data in the monitoring system is obtained; the original video data is the video data obtained by the monitoring system shooting the target space when the robot terminal performs the scanning task; the visual information includes the correlation relationship between the environmental information of the target space and the position information of the robot terminal, including:
[0014] When the robot terminal starts to perform the image scanning task, controlling the monitoring device to capture raw video data in the target space;
[0015] The monitoring platform is controlled to analyze and process the original video data to obtain the environmental information of the target space and the position information of the robot terminal at each moment; and the correlation between the environmental information and the position information is analyzed to obtain visual information.
[0016] In some embodiments, the number of the monitoring devices is greater than or equal to 2, and each of the monitoring devices is arranged at a different position of the target space;
[0017] The controlling the monitoring platform to analyze and process the raw video data to obtain the environment information of the target space and the position information of the robot terminal at each moment includes:
[0018] Controlling the monitoring platform to analyze the image information at each sampling moment in the original video data;
[0019] The image information is subjected to target detection and target recognition processing to obtain the environmental information of the target space and the position information of the robot terminal of each monitoring device at each moment.
[0020] In some of the embodiments, the environmental information includes: each identified target object located in the target space, a confidence rate of the target object, and the coordinates of the target object.
[0021] In some embodiments, the step of semantically annotating the grid map based on the real-time data and the visual information to obtain a semantic map includes:
[0022] Based on the real-time data and the visual information, semantic annotation is performed on occupied grids and unknown grids in the grid map to obtain a semantic map.
[0023] In a second aspect, a robot semantic map construction device is provided in this embodiment, and the device is applicable to a robot semantic map construction system, the system comprising a robot terminal, a robot platform and a monitoring system, and the device is applied to the robot platform, comprising:
[0024] An initial map data acquisition module, used to acquire real-time data generated when the robot terminal performs a map scanning task in the target space; the real-time data includes a grid map of the target space, real-time position data and real-time posture data of the robot terminal;
[0025] a visual information acquisition module, configured to acquire visual information generated in the monitoring system based on raw video data; the raw video data being video data obtained by the monitoring system when the robot terminal performs the image scanning task and photographing the target space; the visual information including the correlation between the environmental information of the target space and the position information of the robot terminal;
[0026] The semantic annotation module is used to perform semantic annotation on the grid map based on the real-time data and the visual information to obtain a semantic map.
[0027] In a third aspect, the present application also provides a robot semantic map construction system, the system comprising: a robot terminal, a robot platform and a monitoring system;
[0028] The robot terminal and the monitoring system are respectively connected to the robot platform for communication;
[0029] The robot terminal is used to perform a scanning task in the target space;
[0030] The monitoring system is used to collect raw video data in the target space when the robot terminal performs the image scanning task, and generate visual information based on the raw video data;
[0031] The robot platform is used to execute the steps of the robot semantic map construction method described in the first aspect.
[0032] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the robot semantic map construction method described in the first aspect is implemented.
[0033] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the robot semantic map construction method described in the first aspect is implemented.
[0034] Compared with the related art, the robot semantic map construction method, device and system provided in this embodiment obtains the real-time data generated by the robot terminal when performing the image scanning task in the target space through the robot platform; the real-time data includes the grid map of the target space, the real-time position data and real-time posture data of the robot terminal; obtains the visual information generated based on the original video data in the monitoring system; the original video data is the video data obtained by the monitoring system when the robot terminal performs the image scanning task, and the visual information includes the correlation between the environmental information of the target space and the position information of the robot terminal; based on the real-time data and the visual information, the grid map is semantically annotated to obtain a semantic map. The problem of high requirements for robot hardware and poor quality of construction results when constructing semantic maps is solved, and the hardware visual capabilities of the robot terminal itself are not relied on. By linking the monitoring system in a specific space, the accuracy of semantic map construction is improved.
[0035] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0037] Figure 1A hardware structure block diagram of a terminal of a robot semantic map construction method in one embodiment;
[0038] Figure 2 A structural block diagram of a robot semantic map construction system in one embodiment;
[0039] Figure 3 A schematic diagram of a process of constructing a robot semantic map in one embodiment;
[0040] Figure 4 A schematic diagram of a robot terminal constructing a grid map in one embodiment;
[0041] Figure 5 is a schematic diagram of a labeled semantic map in one embodiment;
[0042] Figure 6 A schematic diagram of a robot terminal performing a scanning task in a target space in a preferred embodiment;
[0043] Figure 7 A schematic diagram of video images of different monitoring devices at different times in a preferred embodiment;
[0044] Figure 8 A schematic diagram of annotated before and after maps in a preferred embodiment;
[0045] Fig. 9 It is a structural block diagram of a robot semantic map building device in one embodiment.
[0046] Figure numerals: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 210, robot terminal; 220, robot platform; 230, monitoring system; 231, monitoring device; 232, monitoring platform; 91, initial map data acquisition module; 92, visual information acquisition module; 93, semantic annotation module. DETAILED DESCRIPTION
[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.
[0048] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.
[0049] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal of the robot semantic map construction method of this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0050] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the robot semantic map construction method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0051] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.
[0052] In this embodiment, a robot semantic map construction system is provided. Figure 2 As shown, the system includes: a robot terminal 210, a robot platform 220 and a monitoring system 230. The robot terminal 210 and the monitoring system 230 are respectively connected to the robot platform 220 for communication.
[0053] The robot terminal 210 is used to perform a map scanning task in the target space. The robot terminal 210 obtains space map data through its own laser radar sensor, performs the map scanning task, and constructs a grid map.
[0054] The monitoring system 230 is used to collect raw video data in the target space and generate visual information based on the raw video data when the robot terminal 210 performs a scanning task. The monitoring system 230 includes a monitoring device 231 and a monitoring platform 232. The monitoring device 231 is used to collect raw video data in the target space, and the monitoring platform 232 is used to generate visual information based on the raw video data. At least one monitoring device 231 is set at a fixed position in the target space. In the target space where a camera has been installed for security monitoring and other purposes, there is no need to install a new monitoring device 231. The existing camera can be used as the monitoring device 231 for video data collection.
[0055] The robot platform 220 is used to execute the steps of the robot semantic map construction method in any of the following embodiments.
[0056] In this embodiment, the mobile robot terminal 210 and the monitoring system 230 are linked to obtain visual semantic features in the space for the robot terminal 210, and a semantic map of the space is constructed synchronously. A method for constructing a semantic map can be provided for the robot terminal 210 that has been put into use but has limited hardware capabilities, so as to perform more complex scene function requirements. For example, existing sweeping robots and disinfection robots usually perform cleaning and disinfection according to the global map and do not have the ability to perform tasks at fixed points. According to the semantic map, tasks can be performed according to instruction sets such as "clean the ground near the table" and "please go to inspection area C for disinfection".
[0057] In this embodiment, a method for constructing a robot semantic map is provided, and the method is applicable to Figure 2 The robot semantic map construction system shown includes a robot terminal 210, a robot platform 220 and a monitoring system 230. Figure 3 is a flow chart of a robot semantic map construction method executed by the robot platform 220 side, such as Figure 3 As shown, the process includes the following steps:
[0058] Step S310, acquiring real-time data generated when the robot terminal 210 performs a scanning task in the target space; the real-time data includes a grid map of the target space, real-time position data and real-time posture data of the robot terminal 210.
[0059] Specifically, the robot terminal 210 moves in the target space, obtains spatial map data through the laser radar sensor carried by itself, and constructs a grid map. In addition, the robot terminal 210 uploads the real-time position data and real-time posture data obtained during the map construction process in real time. The robot terminal 210 can directly send the real-time data to the robot platform 220, or upload the real-time data to the server for storage, and the robot platform 220 obtains the real-time data from the server.
[0060] See also Figure 4 , the grid map can be expressed as MAP = (O, F, U), where O represents occupied grid information, F represents free grid information, and U represents unknown grid information. The real-time position data is the center coordinate set CR of the robot terminal 210 at each time node within the preset period T, with the lower left corner of the grid map MAP as the origin. k The r shown in the figure t1 、r t2 and r t3are the identifiers of the positions of the robot terminal 210 at time t1, t2 and t3 respectively. The real-time posture data is the posture data set ATT of the robot terminal 210 at each time node within the T period with the robot terminal 210 as the coordinate system. k .
[0061] Step S320, obtaining visual information generated in the monitoring system 230 based on the original video data; the original video data is the video data obtained by the monitoring system 230 when the robot terminal 210 performs a scanning task and captures the target space; the visual information includes the correlation between the environmental information of the target space and the position information of the robot terminal 210.
[0062] Specifically, the monitoring system 230 includes at least one monitoring camera, which can record the screen when the robot terminal 210 performs the scanning task in the target space to obtain video data. For the video screens collected by each monitoring camera at each time, target detection and object recognition are performed to analyze the position coordinate set of the robot terminal 210 at different times in the screen, the type and position of the objects placed in the target space, and other information. It is also possible to identify the idle area in the target space where the robot terminal 210 can pass, divide the idle area into different sub-areas, and obtain environmental information based on the information such as the various objects or idle areas that are identified.
[0063] Step S330, semantically annotating the grid map based on the real-time data and visual information to obtain a semantic map.
[0064] For details, see Figure 5 , the grids in the grid map can be labeled according to actual needs. For example, in the application scenario of an indoor office, the unknown grid where the file cabinet in the office is located is labeled as a file cabinet; in the application scenario of a subway station waiting platform, the passage area in front of the platform door is divided into two areas, and the idle grids are marked as the boarding area and the alighting area on the grid map. The real-time data is the relevant information of the robot terminal 210 in the target space from the perspective of the two-dimensional grid map, and the visual information is the relevant information of the robot terminal 210 in the target space from the perspective of the monitoring system 230; the two perspectives are merged, and the visual capabilities of the monitoring system 230 are used to give semantics to the two-dimensional grid map.
[0065] In this embodiment, the real-time data generated when the robot terminal 210 performs a scanning task in the target space is obtained through the robot platform 220; the real-time data includes the grid map of the target space, the real-time position data and the real-time posture data of the robot terminal 210; the visual information generated based on the original video data in the monitoring system 230 is obtained; the original video data is the video data obtained by the monitoring system 230 when the robot terminal 210 performs the scanning task, and the visual information includes the correlation between the environmental information of the target space and the position information of the robot terminal 210; based on the real-time data and the visual information, the grid map is semantically annotated to obtain a semantic map. The problem of high requirements for robot hardware and poor quality of construction results when constructing semantic maps is solved, and the hardware visual capabilities of the robot terminal 210 itself are not relied on. By linking the monitoring system 230 in a specific space, the accuracy of semantic map construction is improved.
[0066] In some embodiments, based on the above step S310, real-time data generated when the robot terminal 210 performs a scanning task in the target space is obtained; the real-time data includes a grid map of the target space, real-time position data and real-time posture data of the robot terminal 210, and specifically includes the following steps:
[0067] Step S311, controlling the robot terminal 210 to perform a scanning task.
[0068] Step S312, after the scanning task is completed, obtain from the robot terminal 210 the grid map of the target space generated by the robot terminal 210, as well as the real-time position data and real-time posture data of the robot terminal 210 in a preset time period.
[0069] Specifically, the robot terminal 210 obtains spatial map data through a conventional laser radar sensor and constructs a grid map. The robot terminal 210 does not need visual sensors such as binocular / depth cameras, which reduces the hardware requirements for the robot terminal 210 when generating semantic maps and saves the cost of upgrading robot hardware. The preset time period can be a complete time period for executing the scanning task, or it can be a certain end time intercepted from the time of executing the scanning task based on a preset duration as the preset time period.
[0070] In this embodiment, by communicating with the robot terminal 210, real-time data for generating semantics is efficiently acquired.
[0071] In some of the embodiments, the monitoring system 230 includes a monitoring platform 232 and a fixed-position monitoring device 231 .
[0072] Wherein, based on the above step S320, the visual information generated based on the original video data in the monitoring system 230 is obtained; the original video data is the video data obtained by the monitoring system 230 shooting the target space when the robot terminal 210 performs the scanning task; the visual information includes the correlation relationship between the environmental information of the target space and the position information of the robot terminal 210, and specifically includes the following steps:
[0073] Step S321, when the robot terminal 210 starts to perform the scanning task, the monitoring device 231 is controlled to capture the original video data in the target space.
[0074] Step S322, control the monitoring platform 232 to analyze and process the original video data to obtain the environmental information of the target space and the position information of the robot terminal 210 at each moment; and analyze the correlation between the environmental information and the position information to obtain visual information.
[0075] Specifically, the monitoring device 231 is preferably fixed at a high place, higher than the height of the target object to be identified, so as to capture the target space from a bird's-eye view and obtain a wider field of view. The video cycle recorded by the monitoring device 231 should include the execution time of the acquired real-time data of the robot terminal 210. The environmental information includes information such as the position and confidence rate of the identified object in the target space. The identified object can be a placed object or an idle space.
[0076] In this embodiment, visual information used to generate semantics is acquired by communicating with the monitoring system 230 .
[0077] In some of the embodiments, the number of monitoring devices 231 is greater than or equal to 2, and each monitoring device 231 is set at a different position in the target space.
[0078] Specifically, when the target space is large or the spatial position is specially blocked, multiple monitoring devices 231 can be set to shoot from different perspectives to achieve full coverage. Or for a small space, the target object can be identified from various perspectives to obtain more accurate positioning information.
[0079] Wherein, based on the above step S322, the control monitoring platform 232 analyzes and processes the original video data to obtain the environmental information of the target space and the position information of the robot terminal 210 at each moment, which specifically includes the following steps:
[0080] Step S401, controlling the monitoring platform 232 to analyze the image information at the sampling moment in each original video data.
[0081] Step S402 , performing target detection and target recognition processing on the image information, and obtaining the environmental information of the target space of each monitoring device 231 at each moment and the position information of the robot terminal 210 .
[0082] Specifically, each time in step S402 here needs to correspond to the time included in the real-time data.
[0083] In this embodiment, a multi-view monitoring device 231 is provided to obtain environmental information and position information under multiple viewpoints, which helps to improve the accuracy of positioning each object in the space, thereby improving the quality of the semantic map.
[0084] In some of the embodiments, the environmental information includes: each identified target object located in the target space, as well as the confidence rate of the target object and the coordinates of the target object.
[0085] Specifically, according to the actual needs of use, only the target objects placed in the target space can be annotated. When analyzing environmental information, the confidence rate of the target object and the coordinates of the target object can be calculated. The confidence rate is used to indicate the degree of credibility of the target object predicted in the video screen. In addition, a confidence rate threshold can be set. Only when the confidence rate of the prediction result is higher than the threshold, it will be considered as a valid target object prediction result, thereby filtering out some inaccurate data to improve the accuracy and reliability of the annotated semantic map.
[0086] The coordinates of the target object are based on a coordinate system established with the lower left corner of the video data screen as the origin. Correspondingly, the position information of the robot terminal 210 analyzed by the same monitoring device 231 is also calculated using the same coordinate system as the target object, so as to establish an association relationship between the target object and the robot terminal 210.
[0087] In this embodiment, the environmental information including the confidence rate of the target object and the coordinates of the target object is helpful for accurately generating the semantic information.
[0088] In some embodiments, based on step S330, semantic annotation is performed on the grid map based on the real-time data and the visual information to obtain a semantic map, which specifically includes the following steps:
[0089] Step S331 , based on the real-time data and visual information, semantic annotation is performed on the occupied grids and unknown grids in the grid map to obtain a semantic map.
[0090] In this embodiment, by performing semantic analysis on the target object, the meanings of occupied grids and unknown grids can be expanded, and semantic information can be added to the two-dimensional grid map.
[0091] See also Figure 6 , Figure 7 and Figure 8 , this embodiment is described and illustrated through a preferred embodiment.
[0092] Figure 6 FIG. 1 is a schematic diagram of a robot terminal performing a scanning task in a target space in a preferred embodiment. Figure 6 As shown, the target space includes several objects a. Figure 7 The figure shows that in the preferred embodiment, monitoring devices m1 and m2 are in the same period t1 to t k Schematic diagram of the video screen at the moment. Figure 8 It is a schematic diagram of annotated before and after maps in a preferred embodiment.
[0093] 1. Construct mathematical models for each component in the space, including:
[0094] 1) Spatial information:
[0095] a) Robot terminal in space: r.
[0096] b) The set of monitoring devices in the space: M = {m1,m2,m3…m i}.
[0097] c) The two-dimensional coordinate set of the monitoring device at a fixed position projected on the top view of the space with the lower left corner as the origin: CM i .
[0098] d) A set of objects in space: A = {a1, a2, a3…a i}.
[0099] e) The scanning time period of the robot terminal r and the k time nodes included: T = {t1, t2, t3…t i}.
[0100] 2) Information collected and analyzed by monitoring devices and monitoring device platforms:
[0101] a) The original data set collected by the monitoring device set within T time: D.
[0102] b) Analyze the original data set D obtained by the monitoring equipment set within the scanning cycle T, and obtain the visual information set D'={d1,d2,d3…d i}.
[0103] Among them, d i Indicates that the monitoring device m i The visual data analyzed from the acquired video data, specifically, d i ={c i 1,c i 2,c i 3…ci k}.
[0104] Among them, c i k =f(A i K ,P i k ,CA i k ,cr i k ).
[0105] Among them, A i k It represents the set of objects in the video image of monitoring device i at the kth time node obtained through target detection, object recognition and related analysis:
[0106]
[0107] P i k It indicates the confidence rate of each object in the picture at the kth time node obtained by monitoring device i through target detection, object recognition and related analysis; CA i k represents the coordinate set of the object at the kth time node of monitoring device i, with the lower left corner of the video screen as the origin of the coordinate system; cr i k represents the center coordinate set of the robot terminal r in the video screen at the kth time node of the monitoring device i, with the lower left corner of the video screen as the origin of the coordinate system; f represents c i k The relationship between each data.
[0108] 3) Robot terminal collects information:
[0109] a) Map information: Grid map MAP = (O, F, U), where O represents occupied grid information, F represents idle grid information, and U represents unknown grid information.
[0110] b) Taking the lower left corner of the grid map as the origin, the center coordinate set of the robot r at each time node in the T period: CR k .
[0111] c) The robot posture data set at each time point in the T period with the robot terminal as the coordinate system: ATT k .
[0112] 2. Overall process:
[0113] 1) The robot terminal r performs the image scanning task in the T time period.
[0114] 2) The robot platform obtains the real-time data of the robot terminal within the T period and synchronizes the real-time data to the robot platform; the real-time data includes: the grid map MAP = (O, F, U) constructed by the robot terminal r, the real-time coordinate data and the real-time posture data.
[0115] 3) Each monitoring device in the monitoring device set obtains a video information set D within a period T.
[0116] 4) Video information set D is synchronously sent to the monitoring platform.
[0117] 5) The monitoring platform analyzes the video information set D, see Figure 7 By analyzing the image information of each time node within the T period, that is, the image frame obtained by any monitoring device in the space at a certain moment, the following data information is obtained: a) the set of target objects existing in the image at that moment; b) the confidence rate corresponding to each target object; c) the coordinate set of each target object; d) the coordinate information of the robot.
[0118] 6) The monitoring platform synchronizes the visual information set D' to the robot platform.
[0119] 7) The robot platform semantically annotates the grid map based on the real-time data and visual information set D', see Figure 8 Specifically, the occupied grids and unknown grids in the grid map are marked to obtain the semantic map MAP semantic =(O',F,U').
[0120] This preferred embodiment is suitable for spatial scenarios with mobile robot terminal devices and fixed-position monitoring devices, such as home sweeping robot scenarios, shopping mall and hospital disinfection robot scenarios, and Internet of Vehicles scenarios in parks or closed roads. Compared with the prior art, the advantages are: it does not depend on whether the robot terminal itself has a visual sensor, which can reduce the hardware cost of the robot itself. For devices that do not have visual sensors that are put into operation, it can become an optional solution for building semantic maps. This preferred embodiment obtains more spatial information by linking other devices with visual capabilities in the space, which can fill the limitations of the robot itself in obtaining information and build semantic maps at low cost and high efficiency.
[0121] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0122] In this embodiment, a robot semantic map construction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
[0123] Fig. 9 is a structural block diagram of the robot semantic map construction device of this embodiment. Fig. 9 As shown, the device is suitable for a robot semantic map construction system, which includes a robot terminal, a robot platform and a monitoring system. The device is applied to the robot platform, and the device includes:
[0124] The initial map data acquisition module 91 is used to acquire real-time data generated when the robot terminal performs a map scanning task in the target space; the real-time data includes a grid map of the target space, real-time position data and real-time posture data of the robot terminal.
[0125] The visual information acquisition module 92 is used to acquire visual information generated in the monitoring system based on the original video data; the original video data is the video data obtained by the monitoring system when the robot terminal performs a scanning task and captures the target space; the visual information includes the correlation between the environmental information of the target space and the position information of the robot terminal.
[0126] The semantic annotation module 93 is used to semantically annotate the raster map based on the real-time data and visual information to obtain a semantic map.
[0127] In some of the embodiments, the initial map data acquisition module 91 is also used to control the robot terminal to perform a map scanning task; after the map scanning task is completed, the grid map of the target space generated by the robot terminal, as well as the real-time position data and real-time posture data of the robot terminal in a preset time period are obtained from the robot terminal.
[0128] In some of the embodiments, the monitoring system includes a monitoring platform and fixed-position monitoring equipment.
[0129] Among them, the visual information generated based on the original video data in the monitoring system is obtained; the original video data is the video data obtained by the monitoring system shooting the target space when the robot terminal performs the scanning task; the visual information includes the correlation between the environmental information of the target space and the position information of the robot terminal, including:
[0130] When the robot terminal starts to perform the scanning task, the monitoring equipment is controlled to capture the original video data in the target space; the monitoring platform is controlled to analyze and process the original video data to obtain the environmental information of the target space and the position information of the robot terminal at each moment; and the correlation between the environmental information and the position information is analyzed to obtain visual information.
[0131] In some of the embodiments, the number of monitoring devices is greater than or equal to 2, and each monitoring device is arranged at a different position in the target space.
[0132] Among them, the control monitoring platform analyzes and processes the original video data to obtain the environmental information of the target space and the location information of the robot terminal at each moment, including:
[0133] The control monitoring platform analyzes the image information at the sampling moment in each raw video data; performs target detection and target recognition processing on the image information to obtain the environmental information of the target space of each monitoring device at each moment and the position information of the robot terminal.
[0134] In some of the embodiments, the environmental information includes: each identified target object located in the target space, as well as the confidence rate of the target object and the coordinates of the target object.
[0135] In some of the embodiments, the semantic annotation module 93 is further used to perform semantic annotation on occupied grids and unknown grids in the grid map based on real-time data and visual information to obtain a semantic map.
[0136] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0137] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.
[0138] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.
[0139] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.
[0140] In addition, in combination with the robot semantic map construction method provided in the above embodiments, a storage medium can also be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the robot semantic map construction methods in the above embodiments is implemented.
[0141] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.
[0142] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.
[0143] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.
[0144] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.
Claims
1. A method for constructing a robot semantic map, characterized in that: The method is applicable to a robot semantic map construction system, which includes a robot terminal, a robot platform and a monitoring system. The method is applied to the robot platform and includes: Acquire real-time data generated when the robot terminal performs a scanning task in the target space; the real-time data includes a grid map of the target space, real-time position data and real-time posture data of the robot terminal; Obtaining visual information generated based on raw video data in the monitoring system; the raw video data is video data obtained by the monitoring system shooting the target space when the robot terminal performs the image scanning task; the visual information includes a correlation relationship between environmental information of the target space and position information of the robot terminal; Based on the real-time data and the visual information, the grid map is semantically annotated to obtain a semantic map.
2. The robot semantic map construction method according to claim 1, characterized in that: The real-time data generated when the robot terminal performs a scanning task in the target space is obtained; the real-time data includes a grid map of the target space, real-time position data and real-time posture data of the robot terminal, including: Controlling the robot terminal to perform a scanning task; After the image scanning task is completed, a grid map of the target space generated by the robot terminal, as well as real-time position data and real-time posture data of the robot terminal in a preset time period are obtained from the robot terminal.
3. The robot semantic map construction method according to claim 1, characterized in that: The monitoring system includes a monitoring platform and fixed-position monitoring equipment; Wherein, the visual information generated based on the original video data in the monitoring system is obtained; the original video data is the video data obtained by the monitoring system shooting the target space when the robot terminal performs the scanning task; the visual information includes the correlation relationship between the environmental information of the target space and the position information of the robot terminal, including: When the robot terminal starts to perform the image scanning task, controlling the monitoring device to capture raw video data in the target space; The monitoring platform is controlled to analyze and process the original video data to obtain the environmental information of the target space and the position information of the robot terminal at each moment; and the correlation between the environmental information and the position information is analyzed to obtain visual information.
4. The robot semantic map construction method according to claim 3, characterized in that: The number of the monitoring devices is greater than or equal to 2, and each of the monitoring devices is arranged at a different position of the target space; The controlling the monitoring platform to analyze and process the raw video data to obtain the environment information of the target space and the position information of the robot terminal at each moment includes: Controlling the monitoring platform to analyze the image information at each sampling moment in the original video data; The image information is subjected to target detection and target recognition processing to obtain the environmental information of the target space and the position information of the robot terminal of each monitoring device at each moment.
5. The robot semantic map construction method according to claim 1, characterized in that: The environmental information includes: each identified target object located in the target space, a confidence rate of the target object, and the coordinates of the target object.
6. The robot semantic map construction method according to claim 1, characterized in that: The step of semantically annotating the grid map based on the real-time data and the visual information to obtain a semantic map includes: Based on the real-time data and the visual information, semantic annotation is performed on occupied grids and unknown grids in the grid map to obtain a semantic map.
7. A robot semantic map construction device, characterized in that: The device is applicable to a robot semantic map construction system, which includes a robot terminal, a robot platform and a monitoring system. The device is applied to the robot platform and includes: An initial map data acquisition module, used to acquire real-time data generated when the robot terminal performs a map scanning task in the target space; the real-time data includes a grid map of the target space, real-time position data and real-time posture data of the robot terminal; a visual information acquisition module, configured to acquire visual information generated in the monitoring system based on raw video data; the raw video data being video data obtained by the monitoring system when the robot terminal performs the image scanning task and photographing the target space; the visual information including the correlation between the environmental information of the target space and the position information of the robot terminal; The semantic annotation module is used to perform semantic annotation on the grid map based on the real-time data and the visual information to obtain a semantic map.
8. A robot semantic map construction system, characterized in that: The system comprises: a robot terminal, a robot platform and a monitoring system; The robot terminal and the monitoring system are respectively connected to the robot platform for communication; The robot terminal is used to perform a scanning task in the target space; The monitoring system is used to collect raw video data in the target space when the robot terminal performs the image scanning task, and generate visual information based on the raw video data; The robot platform is used to execute the steps of the robot semantic map construction method according to any one of claims 1 to 6.
9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the robot semantic map construction method according to any one of claims 1 to 6.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the robot semantic map construction method according to any one of claims 1 to 6 are implemented.
