Robot repositioning method and device, storage medium and electronic device

Abstract

The invention provides a robot repositioning method and device, a storage medium and an electronic device, and the method comprises the steps: carrying out the object detection through a target detection part, and obtaining the object information of a candidate object in a detection range of the target detection part; determining the candidate object of which the object type belongs to a preset type as a reference object, and obtaining object information of the reference object; according to the object information of the reference object, determining an area matched with the reference object from a target area map to obtain a candidate area; and repositioning the target robot in the candidate area by using target point cloud data acquired by a target acquisition component. By adopting the technical scheme, the problem of long repositioning time consumption caused by overlarge data volume required to be processed in a robot repositioning method in the related technology is solved.

Description

[0001] This application is a divisional application of application number 202210488235.4, filed on May 6, 2022, entitled "Robot Relocation Method and Apparatus, Storage Medium and Electronic Device". [Technical Field] This application relates to the field of robotics, and more specifically, to a robot relocation method and apparatus, a storage medium, and an electronic device. [Background Technology] Currently, when relocalizing a robot, the robot is generally relocalized by using detection components on the robot (such as lidar sensors) to search the area and matching the detected point cloud data with the area map.

[0002] However, the full-map matching method described above results in a long relocalization time due to the large amount of point cloud data contained within a single map region. Therefore, it is evident that robot relocalization methods in related technologies suffer from the problem of long relocalization times caused by the excessive amount of data to be processed. [Summary of the Invention] The purpose of this application is to provide a robot relocation method and apparatus, storage medium and electronic device, so as to at least solve the problem that the robot relocation method in the related art has a long relocation time due to the large amount of data to be processed.

[0003] The purpose of this application is to achieve the following technical solution: According to one aspect of the embodiments of this application, a robot relocalization method is provided, comprising: performing object detection by a target detection component to obtain object information of a reference object; determining a region matching the reference object from a target region map based on the object information of the reference object to obtain a candidate region; and relocalizing a target robot in the candidate region using target point cloud data collected by a target acquisition component.

[0004] In an exemplary embodiment, the step of detecting objects through a target detection component to obtain object information of a reference object includes: detecting objects through the target detection component to obtain object information of candidate objects located within the detection range of the target detection component; determining candidate objects whose object type belongs to a preset type as the reference object, and obtaining object information of the reference object.

[0005] In an exemplary embodiment, the step of detecting an object through a target detection component to obtain object information of a reference object includes: acquiring an image of the detection area of ​​the target detection component to obtain an image to be identified; inputting the image to be identified into a target object recognition model to obtain the object type of the reference object, wherein the object information of the reference object includes the object type of the reference object.

[0006] In an exemplary embodiment, the step of inputting the image to be identified into a target object recognition model to obtain the object type of the reference object includes: inputting the image to be identified into the feature extraction layer of the target object recognition model to obtain the semantic features contained in the image to be identified output by the feature extraction layer; and inputting the semantic features contained in the image to be identified into the fully connected layer of the target object recognition model to obtain the object type of the reference object.

[0007] In one exemplary embodiment, the object information of the reference object includes at least one of object type and object location; the step of determining a region matching the reference object from a target area map based on the object information of the reference object to obtain a candidate region includes: marking a plurality of candidate objects from a preset number of objects in the target area map based on the object type of the reference object, wherein the object type of each candidate object is the same as the object type of the reference object; filtering out matching objects from the plurality of candidate objects based on the object location of the reference object; and determining the region in the target area map containing the matching object as the candidate region.

[0008] In an exemplary embodiment, the step of filtering matching objects from the plurality of candidate objects based on the object position of the reference object includes: when there are multiple reference objects, determining the object type with the fewest corresponding candidate objects among the object types of the multiple reference objects to obtain a target object type, wherein the target object type corresponds to a target reference object among the multiple reference objects; determining the relative position between other reference objects and the target reference object to obtain target object features, wherein the other reference objects are reference objects other than the target reference object, and the target object features are used to describe the object type of the other reference objects and the relative position between the other reference objects and the target reference object; and sequentially searching for candidate objects of the target object type according to the target object features to obtain a plurality of matching objects, wherein the plurality of matching objects includes target candidate objects of the target object type, and candidate objects whose object type and relative position with the target candidate object are consistent with the target object features.

[0009] In an exemplary embodiment, the candidate region includes multiple candidate grid cells that allow the target robot to enter; the relocalization of the target robot within the candidate region using target point cloud data acquired by the target acquisition component includes: projecting the target point cloud data onto a two-dimensional grid map to obtain grid data to be matched, wherein the grid data to be matched is grid data with the target grid cell as the origin; performing the following steps on each candidate grid cell to obtain a matching degree corresponding to each candidate grid cell, wherein, when performing the following steps, each candidate grid cell is the current grid cell: determining reference grid data corresponding to the current grid cell, wherein the grid area occupied by the reference grid data is consistent with the grid area occupied by the grid data to be matched after the target grid cell is converted to the current grid cell; determining the matching degree between the reference grid data and the grid data to be matched as the matching degree corresponding to the current grid cell; and determining the position of the candidate grid cell with the highest matching degree in the candidate region as the position to which the target robot is relocalized.

[0010] According to another aspect of the embodiments of this application, a robot relocalization device is also provided, comprising: a detection unit, configured to perform object detection by a target detection component to obtain object information of a reference object; a determination unit, configured to determine a region matching the reference object from a target region map based on the object information of the reference object to obtain a candidate region; and a relocalization unit, configured to relocalize a target robot in the candidate region using target point cloud data collected by a target acquisition component.

[0011] In an exemplary embodiment, the detection unit includes: a detection module, configured to perform object detection through the target detection component to obtain object information of candidate objects located within the detection range of the target detection component; and a first determination module, configured to determine candidate objects whose object type belongs to a preset type as the reference object and obtain object information of the reference object.

[0012] In an exemplary embodiment, the detection unit includes: an acquisition module, configured to acquire an image of the detection area of ​​the detection component through the target detection component to obtain an image to be identified; and an input module, configured to input the image to be identified into a target object recognition model to obtain the object type of the reference object, wherein the object information of the reference object includes the object type of the reference object.

[0013] In an exemplary embodiment, the input module includes: a first input submodule, configured to input the image to be recognized into the feature extraction layer of the target object recognition model to obtain the semantic features contained in the image to be recognized output by the feature extraction layer; and a second input submodule, configured to input the semantic features contained in the image to be recognized into the fully connected layer of the target object recognition model to obtain the object type of the reference object.

[0014] In one exemplary embodiment, the object information of the reference object includes at least one of object type and object location; the determining unit includes: a marking module, configured to mark a plurality of candidate objects from preset objects in the target area map according to the object type of the reference object, wherein the object type of each candidate object is the same as the object type of the reference object; a filtering module, configured to filter out matching objects that match the reference object from the plurality of candidate objects according to the object location of the reference object; and a second determining module, configured to determine the area in the target area map containing the matching object as the candidate area.

[0015] In an exemplary embodiment, the filtering module includes: a first determining submodule, configured to, when there are multiple reference objects, determine the object type with the fewest corresponding candidate objects among the object types of the multiple reference objects to obtain a target object type, wherein the target object type corresponds to a target reference object among the multiple reference objects; a second determining submodule, configured to determine the relative position between other reference objects and the target reference object to obtain target object features, wherein the other reference objects are reference objects other than the target reference object, and the target object features are used to describe the object type of the other reference objects and the relative position between the other reference objects and the target reference object; and a searching submodule, configured to, according to the target object features, sequentially search for candidate objects of the target object type to obtain multiple matching objects, wherein the multiple matching objects include target candidate objects of the target object type, and candidate objects whose object type and relative position with the target candidate objects are consistent with the target object features.

[0016] In an exemplary embodiment, the candidate region includes multiple candidate grid cells that allow the target robot to enter; the relocalization unit includes: a projection module, used to project the target point cloud data onto a two-dimensional grid map to obtain grid data to be matched, wherein the grid data to be matched is grid data with the target grid cell as the origin; an execution module, used to perform the following steps on each candidate grid cell to obtain a matching degree corresponding to each candidate grid cell, wherein, when performing the following steps, each candidate grid cell is the current grid cell: determining reference grid data corresponding to the current grid cell, wherein the grid area occupied by the reference grid data is consistent with the grid area occupied by the grid data to be matched after the target grid cell is converted to the current grid cell; determining the matching degree between the reference grid data and the grid data to be matched as the matching degree corresponding to the current grid cell; and a third determination module, used to determine the position of the candidate grid cell with the highest matching degree in the candidate region as the position to which the target robot is relocalized.

[0017] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the test method of the above-described interface at runtime.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the test method of the interface described above through the computer program.

[0019] In this embodiment, the robot's approximate location is determined by first identifying surrounding objects, and then relocalization is performed to obtain the robot's precise location. Object detection is performed using a target detection component to obtain object information of a reference object. Based on this reference object information, a region matching the reference object is determined from the target area map, resulting in a candidate region. The target point cloud data collected by the target acquisition component is used to relocalize the target robot within the candidate region. Because the approximate location of the robot is determined by identifying surrounding objects before relocalization, the number of matching steps for relocalization is reduced, thereby reducing the amount of data required for robot relocalization. This achieves the technical effect of shortening the robot relocalization time and improving its efficiency, thus solving the problem of long relocalization times due to excessive data processing in related robot relocalization methods. [Attached Image Description] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the hardware environment of an optional robot relocation method according to an embodiment of this application; Figure 2 This is a flowchart illustrating an optional robot relocation method according to an embodiment of this application; Figure 3 This is a flowchart illustrating another optional robot relocation method according to an embodiment of this application; Figure 4 This is a structural block diagram of an optional robot relocation device according to an embodiment of this application; Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application.

Detailed Implementation Methods

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0023] According to one aspect of the embodiments of this application, a robot relocalization method is provided. Optionally, in this embodiment, the above-described robot relocalization method can be applied to, for example... Figure 1 The hardware environment shown consists of robot 102 and server 104. (As shown) Figure 1 As shown, server 104 is connected to robot 102 via a network and can be used to provide services to the robot or clients installed on the robot (such as distributing area maps, distributing tasks, etc.). A database can be set up on the server or independently of the server to provide data storage services for server 104.

[0024] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. Robot 102 may include, but is not limited to, one of the following: delivery robot, cleaning robot. The delivery robot may include, but is not limited to, one of the following: food delivery robot, item delivery robot. The cleaning robot may include, but is not limited to, one of the following: sweeping robot (referred to as sweeping machine), floor scrubbing robot (referred to as floor scrubbing machine), robot that integrates washing and mopping.

[0025] The robot relocation method of this application embodiment can be executed by server 104, robot 102, or jointly by server 104 and robot 102. Alternatively, the robot 102 can execute the robot relocation method of this application embodiment by a client installed on it.

[0026] Taking the robot relocation method in this embodiment as an example, which is executed by robot 102, Figure 2 This is a flowchart illustrating an optional robot relocalization method according to an embodiment of this application, as shown below. Figure 2 As shown, the process of this method may include the following steps: Step S202: Object detection is performed using the target detection component to obtain object information of the reference object.

[0027] The robot relocation method in this embodiment can be applied to scenarios where a robot needs to be relocated within a target area. The robot can be a robotic vacuum cleaner, a flying robot, or other types of robots. The relocation can be performed in an indoor environment, and is not limited thereto. The relocation can be performed after the robot has been picked up and placed back on the ground, or when its current position cannot be determined, or in other scenarios. The target area can be an area where the robot can move, such as a user's home, office, or factory workshop, and is not limited thereto in this embodiment.

[0028] In related technologies, when relocalizing a robot, the large amount of point cloud data in a region's map can lead to lengthy relocalization times and a poor user experience if the robot is directly relocalized. Furthermore, if multiple similar regions exist within the target area, the robot may misidentify them, causing relocalization failure.

[0029] To at least partially solve the above-mentioned technical problems, in this embodiment, the approximate position of the robot is determined by detecting objects in the surrounding environment, and then the robot is repositioned based on this position to obtain the precise position of the robot. This can reduce the number of matching operations for robot repositioning and shorten the time required for robot repositioning.

[0030] Optionally, during relocalization of the target robot, the target robot (an example of robot 102 described above) can detect objects within its detection field of view using its target detection component to obtain object information of the reference object. The target detection component can be a perception sensor mounted on the target robot, which may include, but is not limited to, at least one of the following: LiDAR, depth camera, monocular camera, binocular stereo camera, etc. Here, the LiDAR can be an LDS (Laser Distance Sensor), and correspondingly, the target robot can be an LDS robot, etc.

[0031] The reference object can be an object located around the target robot. For example, the terminal can identify all objects within the detection range of the target detection component as reference objects. The number of reference objects can be one or more; the object information of the reference object can include one or more types of information about the reference object, such as the object type, object shape information, object position information, etc., which are not limited in this embodiment.

[0032] Step S204: Based on the object information of the reference object, determine the region that matches the reference object from the target area map to obtain the candidate region.

[0033] In this embodiment, the target robot can determine the region matching the reference object from the target area map based on the object information of the reference object. For example, the target area map may contain multiple regions, and the target robot can determine the region matching the object information of the reference object from its multiple regions to obtain candidate regions. Candidate regions can be regions containing the reference object, or regions whose relative positions match the relative positions of the reference object and the target robot. Here, the matching object is an object in the target area map that matches the reference object; it can be an object of the same object type as the reference object, or an object of the same object type, matching object shape, and the same relative position as the reference object.

[0034] Step S206: Use the target point cloud data collected by the target acquisition component to relocate the target robot within the candidate area.

[0035] After determining the candidate area where the target robot is located, the target robot can use the target acquisition component deployed on the target robot to collect point cloud data within the current environmental range, obtaining target point cloud data. Based on the collected target point cloud data, the target robot can relocalize within the candidate area to obtain its precise position on the target area map. The aforementioned target acquisition component can be the same as the target detection component, or it can be a different component; this embodiment does not impose any limitations on this.

[0036] Through steps S202 to S206, object detection is performed by the target detection component to obtain the object information of the reference object; based on the object information of the reference object, the region matching the reference object is determined from the target area map to obtain the candidate region; the target point cloud data collected by the target acquisition component is used to relocalize the target robot in the candidate region, which solves the problem that the robot relocalization method in related technologies has a long relocalization time due to the large amount of data to be processed, shortens the robot relocalization time, and improves the efficiency of robot relocalization.

[0037] In one exemplary embodiment, object detection is performed by a target detection component to obtain object information of a reference object, including: S11, Object detection is performed by the target detection component to obtain object information of candidate objects within the detection range of the target detection component; S12, identify candidate objects whose object type belongs to the preset type as reference objects, and obtain the object information of the reference objects.

[0038] In this embodiment, in order to improve the accuracy of relocation, the target robot can perform object detection on all objects within the field of view of the target detection component to obtain object information of candidate objects. The object information of the candidate objects includes the object type of each candidate object.

[0039] Optionally, the target robot can detect each candidate object within its field of view using a target detection component, obtaining object information for each candidate object. This object information may include the object type of each candidate object. There can be one or more methods to determine the object type of each candidate object. For example, the object type can be identified based on the image features of each candidate object, or it can be identified using a neural network model or other types of recognition models. This embodiment does not limit this approach.

[0040] The identified objects can be infrequently moved items, such as large household appliances or immovable objects like integrated stoves, or frequently moved objects, such as trash cans and brooms. To avoid the impact of object position changes on relocalization, an object type can be set as the reference object based on whether it is an infrequently moved object; that is, a preset type. The target robot can identify candidate objects whose object type belongs to the preset type as reference objects and obtain the object information of the reference objects.

[0041] For example, the identified candidate objects may include: refrigerator, integrated stove, trash can, broom. If the preset type includes refrigerator and integrated stove, but does not include trash can and broom, then refrigerator and integrated stove can be identified as reference objects, while trash can and broom can not be identified as reference objects.

[0042] In this embodiment, objects used for relocation are selected from the detected objects based on a preset type, which reduces the impact of object movement on the accuracy of relocation and improves the accuracy and efficiency of robot relocation.

[0043] In one exemplary embodiment, object detection is performed by a target detection component to obtain object information of a reference object, including: S21, The target detection component acquires an image of the detection area of ​​the detection component to obtain the image to be identified; S22, input the image to be recognized into the target object recognition model to obtain the object type of the reference object, wherein the object information of the reference object includes the object type of the reference object.

[0044] In this embodiment, a target object recognition model can be used for object detection to obtain object information of a reference object. The target robot can use a target detection component to acquire images of objects within its detection area (i.e., the field of view) to obtain an image to be recognized. The target detection component can be a wide-angle image acquisition device, which can capture all objects within the detection area in a single image, thus obtaining a single image to be recognized. Alternatively, the target detection component can be a regular image acquisition device, which cannot capture all objects within the detection area in a single image, thus obtaining multiple images to be recognized.

[0045] The target robot can input the image to be recognized into the target object recognition model. The target object recognition model can parse the image to be recognized, determine the object type of the reference object contained in the image, and obtain the object type of the reference object. The target robot can obtain the object type of the reference object output by the target object recognition model. In addition, the target object recognition model can also recognize other object information such as the shape of the reference object, but this is not limited in this embodiment.

[0046] Here, the target object recognition model can be obtained by training an initial object recognition model using a set of training images labeled with corresponding object types. The model training device can acquire a set of training images labeled with corresponding object types, use the set of training image samples as training samples to train the initial object recognition model, and adjust the model parameters of the initial object recognition model based on the label (the corresponding object type) of each training image sample and the model output of the initial object recognition model. After multiple iterations, a trained target object recognition model can be obtained.

[0047] The trained target object recognition model can be deployed on the target robot or on the server. The deployed target object recognition model can then be used to identify the image to be recognized and determine the object type of the reference object.

[0048] When the image to be recognized is a single image, it can be directly input into the target object recognition model to obtain the object type of the reference object output by the model. When the image to be recognized contains multiple images, each image can be input into the target object recognition model separately to obtain the object type of the reference object corresponding to each image output by the model. The object types corresponding to each image are then deduplicated to obtain the final object type of the reference object.

[0049] In this embodiment, an object recognition model is used to identify the image to be recognized in order to obtain the object type of the reference object, which can improve the efficiency and convenience of object detection.

[0050] In one exemplary embodiment, the image to be recognized is input into a target object recognition model to obtain the object type of a reference object, including: S31, Input the image to be recognized into the feature extraction layer of the target object recognition model to obtain the semantic features contained in the image to be recognized output by the feature extraction layer; S32, input the semantic features contained in the image to be recognized into the fully connected layer of the target object recognition model to obtain the object type of the reference object.

[0051] In this embodiment, the target recognition model can be an AI (Artificial Intelligence) recognition model, which may include a feature extraction layer and a fully connected layer. Inputting the image to be recognized into the target recognition model can be achieved by: inputting the image to be recognized into the feature extraction layer of the target object recognition model; extracting features from the edges, corners, regions, etc., of the image to obtain the semantic features contained in the image. The feature extraction layer can then input the semantic features contained in the image to be recognized into the fully connected layer of the target object recognition model. The target robot can then obtain the semantic features contained in the image output by the feature extraction layer.

[0052] The fully connected layers of the target object recognition model can integrate class-discriminative semantic features from the image to be recognized, and map these features to corresponding object types using a classification function, thus obtaining the object type of the reference object. The target robot can then obtain the object type of the reference object output by the target object recognition model.

[0053] In this embodiment, by extracting and integrating the semantic features contained in the image to be recognized, the object type of the reference object can be obtained, which can improve the convenience and accuracy of object recognition.

[0054] In an exemplary embodiment, the object information of the reference object may include at least one of object type and object position. The object position is the position of the reference object in the robot coordinate system, or it may be the relative position to a reference object that serves as the origin, or it may be other types of positions; this embodiment does not limit this. Correspondingly, based on the object information of the reference object, regions matching the reference object are determined from the target area map to obtain candidate regions, including: S41, Based on the object type of each reference object in the reference object, mark multiple candidate objects from the preset objects in the target area map, wherein the object type of each candidate object is the same as the object type of the reference object. S42, Based on the object position of the reference object, filter out matching objects from the candidate objects that match the reference object; S43, the regions in the target area map that contain matching objects are identified as candidate regions.

[0055] The constructed target area map can be marked with the object type and position of each of the multiple preset objects contained within the target area. The object type and position of the preset objects can be automatically marked by the target robot, other robots, or the server when constructing the target area map, or they can be manually marked by the user, or marked in other ways. This embodiment does not limit this.

[0056] In this embodiment, the object information of the reference object includes the object type of the reference object. The target robot can mark candidate objects from preset objects in the target area map based on the object type of the reference object, wherein the object type of the marked candidate object is at least the same as the object type of one reference object. The marking process involves: marking preset objects of the same object type as each reference object according to its object type, thus obtaining candidate objects.

[0057] After the candidate objects are marked, the target robot can filter out matching objects from the candidate objects based on the object position of the reference object.

[0058] The target robot can identify regions in the target area map that contain matching objects as candidate regions. Optionally, the target robot can partition the target area map to obtain multiple regions, each containing a subset of preset objects. The target robot can then identify the region containing the matching object from these multiple regions and designate that region as a candidate region.

[0059] Optionally, a target reference object is selected from a set of reference objects, which is a preset object that is only present once in the target area map; the partition where the target reference object is located in multiple partitions is determined as a candidate area, or other methods can be used to determine the candidate area based on the target reference object, which are not limited here.

[0060] This embodiment uses the object type of a preset object in the area map to determine candidate areas, thus improving the accuracy and rationality of the candidate area determination.

[0061] In one exemplary embodiment, selecting matching objects from a plurality of candidate objects based on the object position of the reference object includes: S51, when there are multiple reference objects, determine the object type with the fewest corresponding candidate objects among the multiple reference object types to obtain the target object type, wherein the target object type corresponds to the target reference object among the multiple reference objects. S52, determine the relative positions between other reference objects and the target reference object, and obtain the target object features. Here, other reference objects are reference objects other than the target reference object, and the target object features are used to describe the object type of other reference objects and the relative positions between other reference objects and the target reference object. S53. Based on the characteristics of the target object, candidate objects of the target object type are searched sequentially to obtain multiple matching objects. Among them, the multiple matching objects include target candidate objects of the target object type, object types, and candidate objects whose relative positions to the target candidate objects are consistent with the characteristics of the target object.

[0062] In this embodiment, when there are multiple reference objects, the target robot can determine the target object type from the object types of the multiple reference objects. The target object type can be the object type with the fewest corresponding candidate objects among the object types of the multiple reference objects, such as objects that are not frequently moved in a room (e.g., gas stove, standing air conditioner, etc.). Here, the target object type corresponds to the target reference object among the multiple reference objects.

[0063] After determining the type of the target object, the target robot can determine the relative positions of other reference objects and the target reference object based on the positions of multiple reference objects, thus obtaining the target object's features. The target reference object refers to the reference object of the target object type, while the other reference objects are reference objects other than the target reference object. The target robot can use the target object features to describe the object types of other reference objects and the relative positions of other reference objects and the target reference object.

[0064] The target robot can sequentially search for candidate objects of the target object type according to the characteristics of the target object, and obtain multiple matching objects that match multiple reference objects. The multiple matching objects can include target candidate objects of the target object type, and candidate objects whose object type and relative position to the target candidate objects are consistent with the characteristics of the target object.

[0065] This embodiment uses objects of a type with fewer candidate objects to determine the relative positions of other types of objects, thereby identifying multiple matching objects, which can improve the accuracy and convenience of object matching.

[0066] In one exemplary embodiment, the candidate region includes a plurality of candidate grid cells that the target robot is allowed to enter, and may also include grid cells that the target robot is not allowed to enter. The plurality of candidate grid cells may be determined based on the passability value of the grid cells, or may be determined by other means, which is not limited in this embodiment.

[0067] Correspondingly, the target robot is relocalized within the candidate region using the target point cloud data acquired by the target acquisition component, including: S61, Project the target point cloud data onto a two-dimensional raster map to obtain raster data to be matched, wherein the raster data to be matched is raster data with the target raster cell as the origin; S62, perform the following steps for each candidate raster cell to obtain the matching degree corresponding to each candidate raster cell, wherein each candidate raster cell is the current raster cell when performing the following steps: determine the reference raster data corresponding to the current raster cell, wherein the raster area occupied by the reference raster data is consistent with the raster area occupied by the raster data to be matched after the target raster cell is transformed into the current raster cell; determine the matching degree between the reference raster data and the raster data to be matched as the matching degree corresponding to the current raster cell; S63, determine the position of the candidate grid cell with the highest matching degree in the candidate region as the position to which the target robot is relocated.

[0068] In this embodiment, the target area map can be a two-dimensional grid map. After determining the candidate area, the target robot can project the target point cloud data onto the two-dimensional grid map to obtain the grid data to be matched. The grid data to be matched is the grid data with the target grid cell as the origin, and the target grid cell can be the grid cell where the target robot is currently located.

[0069] After obtaining the grid data to be matched, the target robot can perform the following steps with each candidate grid cell in the candidate region as the current grid cell to obtain the matching degree corresponding to each candidate grid cell: Step 1: Determine the reference raster data corresponding to the current raster cell.

[0070] The target robot can determine the reference grid data corresponding to the current grid cell. Here, the grid area occupied by the reference grid data is the same size as the grid area occupied by the target grid cell after the target grid cell is converted to the current grid cell.

[0071] Step 2: Determine the matching degree between the reference raster data and the raster data to be matched as the matching degree corresponding to the current raster cell.

[0072] The target robot can match the reference grid data with the grid data to be matched to obtain the matching degree between the reference grid data and the grid data to be matched. The matching degree is used to indicate the degree of overlap between the reference grid data and the grid data to be matched (the values ​​on the same grid cells are the same or basically consistent). The higher the degree of overlap, the greater the probability that the target robot is in the target area map.

[0073] After obtaining the matching degree corresponding to each candidate grid cell, the target robot can determine the position of the candidate grid cell with the highest matching degree in the candidate region as the position to which the target robot has relocated. Optionally, the target robot can also determine the highest matching degree among the matching degrees corresponding to each candidate grid cell, and if the highest matching degree is greater than or equal to the target matching degree threshold, determine the position of the candidate grid cell with the highest matching degree in the candidate region as the position to which the target robot has relocated.

[0074] In this embodiment, point cloud data is mapped onto a two-dimensional grid map, and the robot is relocalized based on the degree of matching between grid data, which can improve the accuracy and efficiency of robot relocalization.

[0075] The robot relocalization method in this embodiment will be explained below with reference to an optional example. In this optional example, the candidate region is the estimated location, the target region map is the stored map, and the object information includes object type and location information.

[0076] This optional example provides a relocalization scheme based on camera AI semantic information assistance, such as... Figure 3 As shown, the robot relocalization method in this optional example may include the following steps: In step S302, the robot uses AI to identify and store indoor object information, including object type and location, during the last mapping process.

[0077] When the robot was building the map last time, it could use the camera to perform AI recognition on the indoor images to obtain the type and location of each indoor object. The robot can then store this type and location information in the map.

[0078] In step S304, the machine rotates once in place to identify indoor object information in the current environment, matches it with object information on the stored map, and obtains the estimated location.

[0079] The robot rotates once in place, using its camera to perform AI recognition of the current environment and obtain information about objects in the room. The robot can perform AI recognition of objects that are not frequently moved, such as large household appliances (e.g., integrated stoves, standing air conditioners), and match them with object information on a stored map to obtain the robot's estimated position on the map.

[0080] Step S306: Perform fine relocalization using LDS point cloud.

[0081] After obtaining the estimated position of the robot on the map, the robot can be relocalized using LDS point cloud to obtain a more precise position on the map.

[0082] This example first uses AI to identify specific objects in the current environment and matches them with object information in a stored map to obtain the robot's estimated position. Then, the robot is relocalized to obtain its precise position. This reduces the time required for robot relocalization and improves the speed and accuracy of robot relocalization in indoor environments.

[0083] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0085] According to another aspect of the embodiments of this application, a robot relocation apparatus for implementing the above-described robot relocation method is also provided. Figure 4 This is a structural block diagram of an optional robot relocation device according to an embodiment of this application, such as... Figure 4 As shown, the device may include: The detection unit 402 is used to detect objects through the target detection component and obtain object information of the reference object; The determining unit 404, connected to the detection unit 402, is used to determine the region matching the reference object from the target area map based on the object information of the reference object, and obtain the candidate region. The relocation unit 406, connected to the determination unit 404, is used to relocate the target robot within the candidate area using the target point cloud data collected by the target acquisition component.

[0086] It should be noted that the detection unit 402 in this embodiment can be used to perform the above step S202, the determination unit 404 in this embodiment can be used to perform the above step S204, and the relocation unit 406 in this embodiment can be used to perform the above step S206.

[0087] Through the above modules, object detection is performed by the target detection component to obtain the object information of the reference object; based on the object information of the reference object, the region matching the reference object is determined from the target area map to obtain the candidate region; the target point cloud data collected by the target acquisition component is used to relocalize the target robot in the candidate region, which solves the problem that the robot relocalization method in related technologies has a long relocalization time due to the large amount of data to be processed, shortens the robot relocalization time, and improves the efficiency of robot relocalization.

[0088] In one exemplary embodiment, the detection unit includes: The detection module is used to detect objects through the target detection component and obtain object information of candidate objects within the detection range of the target detection component. The first determination module is used to determine candidate objects whose object type belongs to a preset type as reference objects and obtain the object information of the reference objects.

[0089] In one exemplary embodiment, the detection unit includes: The acquisition module is used to acquire images of the detection area of ​​the target detection component through the target detection component to obtain the image to be identified; The input module is used to input the image to be recognized into the target object recognition model to obtain the object type of the reference object, wherein the object information of the reference object includes the object type of the reference object.

[0090] In one exemplary embodiment, the input module includes: The first input submodule is used to input the image to be recognized into the feature extraction layer of the target object recognition model to obtain the semantic features contained in the image to be recognized output by the feature extraction layer. The second input submodule is used to input the semantic features contained in the image to be recognized into the fully connected layer of the target object recognition model to obtain the object type of the reference object.

[0091] In one exemplary embodiment, the object information of the reference object includes at least one of object type and object location; The defined unit includes: The marking module is used to mark multiple candidate objects from preset objects in the target area map based on the object type of each reference object in the reference object, wherein the object type of each candidate object is the same as the object type of the reference object. The filtering module is used to filter out matching objects from multiple candidate objects based on the object position of the reference object. The second determination module is used to determine the areas in the target area map that contain matching objects as candidate areas.

[0092] In one exemplary embodiment, the filtering module includes: The first determination submodule is used to determine the object type with the fewest corresponding candidate objects among the multiple reference objects when there are multiple reference objects, and obtain the target object type, wherein the target object type corresponds to the target reference object among the multiple reference objects. The second determination submodule is used to determine the relative position between other reference objects and the target reference object, and to obtain the target object features. The other reference objects are reference objects other than the target reference object. The target object features are used to describe the object type of the other reference objects and the relative position between the other reference objects and the target reference object. The search submodule is used to sequentially search for candidate objects of the target object type according to the target object characteristics, and obtain multiple matching objects. Among them, the multiple matching objects include target candidate objects of the target object type, object types, and candidate objects whose relative positions to the target candidate objects are consistent with the target object characteristics.

[0093] In one exemplary embodiment, the candidate region includes multiple candidate grid cells that allow the target robot to enter; the relocation unit includes: The projection module is used to project the target point cloud data onto a two-dimensional raster map to obtain raster data to be matched, wherein the raster data to be matched is raster data with the target raster cell as the origin; The execution module performs the following steps for each candidate raster cell to obtain the matching degree corresponding to each candidate raster cell. During the execution of these steps, each candidate raster cell is the current raster cell: determining reference raster data corresponding to the current raster cell, wherein the raster area occupied by the reference raster data is consistent with the raster area occupied by the raster data to be matched after transforming the target raster cell to the current raster cell; and determining the matching degree between the reference raster data and the raster data to be matched as the matching degree corresponding to the current raster cell. The third determination module is used to determine the position of the candidate grid cell with the highest matching degree in the candidate region as the position to which the target robot is relocated.

[0094] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0095] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium can be used to execute program code for any of the robot relocation methods described above in the embodiments of this application.

[0096] Optionally, in this embodiment, the storage medium may be located on at least one of the network devices in the network shown in the above embodiment.

[0097] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: S1, Object detection is performed by the target detection component to obtain object information of the reference object; S2, Based on the object information of the reference object, determine the region that matches the reference object from the target area map to obtain the candidate region; S3, using the target point cloud data collected by the target acquisition component to relocate the target robot within the candidate area.

[0098] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0099] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0100] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described robot relocation method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0101] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application, such as... Figure 5 As shown, it includes a processor 502, a communication interface 504, a memory 506, and a communication bus 508. The processor 502, communication interface 504, and memory 506 communicate with each other via the communication bus 508. Memory 506 is used to store computer programs; When processor 502 executes a computer program stored in memory 506, it performs the following steps: S1, Object detection is performed by the target detection component to obtain object information of the reference object; S2, Based on the object information of the reference object, determine the region that matches the reference object from the target area map to obtain the candidate region; S3, using the target point cloud data collected by the target acquisition component to relocate the target robot within the candidate area.

[0102] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 5 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned electronic device and other devices.

[0103] The aforementioned memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0104] As an example, the memory 506 described above may include, but is not limited to, the detection unit 402, the determination unit 404, and the repositioning unit 406 from the control device of the aforementioned device. Furthermore, it may include, but is not limited to, other module units from the control device of the aforementioned device, which will not be elaborated upon in this example.

[0105] The processor mentioned above can be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0106] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0107] Those skilled in the art will understand that Figure 5The structure shown is for illustrative purposes only. The device that implements the above robot relocation method can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet Devices (MID), PAD, etc.

[0108] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0109] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0110] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0111] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0114] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0115] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A robot relocalization method, characterized in that, include: Object detection is performed using a target detection component to obtain object information of candidate objects within the detection range of the target detection component; Candidate objects whose object type belongs to a preset type are identified as reference objects, and the object information of the reference objects is obtained. The preset type is set based on whether the object is an object that does not move frequently. Based on the object information of the reference object, a region matching the reference object is determined from the target area map to obtain a candidate region, wherein the candidate region is determined from multiple regions obtained by partitioning the target area map; The target robot is repositioned within the candidate area using the target point cloud data acquired by the target acquisition component.

2. The method as described in claim 1, characterized in that, The process of detecting objects using a target detection component to obtain object information of a reference object includes: The target detection component acquires an image of the detection area of ​​the detection component to obtain the image to be identified. The image to be identified is input into the target object recognition model to obtain the object type of the reference object. The object information of the reference object includes the object type of the reference object. The target object recognition model is deployed on the target robot or server.

3. The method as described in claim 2, characterized in that, The step of inputting the image to be identified into the target object recognition model to obtain the object type of the reference object includes: The image to be identified is input into the feature extraction layer of the target object recognition model to obtain the semantic features contained in the image to be identified output by the feature extraction layer; The semantic features contained in the image to be identified are input into the fully connected layer of the target object recognition model to obtain the object type of the reference object.

4. The method as described in claim 3, characterized in that, The image to be identified is input into the target object recognition model, and the object type of the reference object is obtained, including: If the image to be identified is a single image, then the single image is directly input into the target object recognition model to obtain the object type of the reference object output by the target object recognition model; If the image to be identified contains multiple images, then each of the multiple images is input into the target object recognition model to obtain the object type of the reference object corresponding to each image, as output by the target object recognition model. The object types of the reference objects corresponding to each image are deduplicated to obtain the object type of the reference object.

5. The method as described in claim 1, characterized in that, The step of determining the region matching the reference object from the target area map based on the object information of the reference object to obtain candidate regions includes: Select a target reference object from the reference objects, wherein the target reference object is a preset object that is only present once in the target area map; and The partition containing the target reference object in the multiple regions obtained from the partitioning is determined as the candidate region.

6. The method as described in claim 1, characterized in that, The object information of the reference object includes at least one of object type and object location; The step of determining the region matching the reference object from the target area map based on the object information of the reference object to obtain candidate regions includes: Based on the object type of the reference object, multiple candidate objects are marked from preset objects in the target area map, wherein the object type of each candidate object is the same as the object type of the reference object. Based on the object position of the reference object, a matching object that matches the reference object is selected from the plurality of candidate objects; The regions in the target area map that contain the matching object are identified as the candidate regions.

7. The method as described in claim 6, characterized in that, The step of filtering out matching objects from the plurality of candidate objects based on the object position of the reference object includes: When there are multiple reference objects, the object type with the fewest corresponding candidate objects among the multiple reference object types is determined to obtain the target object type, wherein the target object type corresponds to the target reference object among the multiple reference objects; Determine the relative positions between other reference objects and the target reference object to obtain the target object features. The other reference objects are reference objects other than the target reference object. The target object features are used to describe the object type of the other reference objects and the relative positions between the other reference objects and the target reference object. Based on the characteristics of the target object, candidate objects of the target object type are searched sequentially to obtain multiple matching objects. The multiple matching objects include target candidate objects of the target object type, object types, and candidate objects whose relative positions to the target candidate objects are consistent with the characteristics of the target object.

8. The method according to any one of claims 1 to 7, characterized in that, The candidate region includes multiple candidate grid cells that the target robot is allowed to enter; the relocalization of the target robot within the candidate region using the target point cloud data acquired by the target acquisition component includes: The target point cloud data is projected onto a two-dimensional raster map to obtain raster data to be matched, wherein the raster data to be matched is raster data with the target raster cell as the origin; For each candidate raster cell, the following steps are performed to obtain the matching degree corresponding to each candidate raster cell, wherein each candidate raster cell is the current raster cell when performing the following steps: determining reference raster data corresponding to the current raster cell, wherein the raster area occupied by the reference raster data is consistent with the raster area occupied by the raster data to be matched after the target raster cell is transformed into the current raster cell; determining the matching degree between the reference raster data and the raster data to be matched as the matching degree corresponding to the current raster cell; The position of the candidate grid cell with the highest matching degree in the candidate region is determined as the repositioning position of the target robot.

9. A robot repositioning device, characterized in that, include: The detection unit is used to detect objects through the target detection component and obtain object information of candidate objects located within the detection range of the target detection component; And determine candidate objects whose object type belongs to a preset type as reference objects, and obtain the object information of the reference objects, wherein the preset type is set based on whether the object is an object that does not move frequently; The determining unit is configured to determine, based on the object information of the reference object, a region matching the reference object from a target area map to obtain a candidate region, wherein the candidate region is determined from multiple regions obtained by partitioning the target area map; The relocation unit is used to relocate the target robot within the candidate area using the target point cloud data acquired by the target acquisition component.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method of any one of claims 1 to 8.

11. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1 to 8 through the computer program.

Images