Method, apparatus, device, storage medium and program product for map construction

Abstract

Embodiments of the present disclosure provide a method, device, equipment, storage medium and program product for map construction. An example method comprises: based on a plurality of images collected in a scene, creating a plurality of nodes respectively corresponding to the plurality of images, the scene comprising a plurality of landmark elements, and a node in the plurality of nodes indicating at least one landmark element in the plurality of landmark elements identified from an image corresponding to the node and a pose for collecting the corresponding image. Based on the plurality of nodes, determining respective attributes of the plurality of landmark elements, an attribute of a landmark element in the plurality of landmark elements comprising pose information related to the landmark element, at least one node in the plurality of nodes indicating the landmark element, and functional description information for the landmark element. Based on the respective attributes of at least part of the plurality of landmark elements, constructing a map for the scene.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices, computer-readable storage media, and program products for map construction. Background Technology

[0002] In recent years, robotics technology has developed rapidly and has been widely used in various technical fields. For example, on factory production lines, robots can perform various tasks such as sorting and handling. To ensure the normal operation of robots, it is necessary to accurately construct maps related to the robot's working environment. Summary of the Invention

[0003] In a first aspect of this disclosure, a map construction method is provided. The method includes: creating multiple nodes corresponding to multiple images acquired in a scene, the scene including multiple landmark elements, and nodes indicating at least one landmark element identified from the image corresponding to that node and a pose used to acquire the corresponding image; determining corresponding attributes of the multiple landmark elements based on the multiple nodes, the attributes of the landmark elements including pose information related to the landmark element, at least one node in the multiple nodes indicating the landmark element, and functional description information for the landmark element; and generating a map for the scene based on the corresponding attributes of at least a portion of the landmark elements.

[0004] In a second aspect of this disclosure, an apparatus for map construction is provided. The apparatus includes: a creation module configured to create multiple nodes corresponding to multiple images acquired in a scene, the scene including multiple landmark elements, and nodes indicating at least one landmark element identified from the image corresponding to that node and a pose for acquiring the corresponding image; a determination module configured to determine corresponding attributes of the multiple landmark elements based on the multiple nodes, the attributes of the landmark elements including pose information associated with the landmark element, at least one node in the multiple nodes indicating the landmark element, and functional description information for the landmark element; and a construction module configured to construct a map for the scene based on the corresponding attributes of at least a portion of the landmark elements.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program executable by a processor to implement the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0011] Figure 2 An architecture diagram of an example system for constructing maps according to some embodiments of this disclosure is shown;

[0012] Figure 3 A flowchart illustrating an example process for map construction according to some embodiments of this disclosure is shown;

[0013] Figure 4 A schematic structural block diagram of an example apparatus for map building according to some embodiments of the present disclosure is shown; and

[0014] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0019] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0020] Example Environment

[0021] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1As shown, one or more robots are deployed in environment 100, such as a first robot 110-1, a second robot 110-2, and a third robot 110-3. The first robot 110-1, the second robot 110-2, and the third robot 110-3 can also be collectively referred to as robot 110 or individually. In the following description, an example embodiment of controlling the robot to perform tasks will be described using the first robot 110-1 as an example. Each robot 110 may be equipped with a control device to control its operation. For example, such as... Figure 1 As shown, control devices 120-1, 120-2, and 120-3 are respectively deployed at the first robot 110-1, the second robot 110-2, and the third robot 110-3, which can also be collectively referred to as control device 120 or individually. Figure 1 The number of robots 110 and control devices 120 shown is merely exemplary and not intended to be limiting. In some embodiments, only one robot may be deployed in environment 100, and control device 120 needs to control that robot to perform tasks. In some embodiments, multiple robots may be deployed in environment 100. In this case, multiple robots in environment 100 can be controlled simultaneously in real time, or only any one robot in environment 100 can be controlled.

[0022] Each robot 110 can also be equipped with sensing devices to obtain environmental information about the robot's location. For example... Figure 1 As shown, sensing devices 130-1, 130-2, and 130-3 are respectively deployed at the first robot 110-1, the second robot 110-2, and the third robot 110-3. They can also be collectively referred to as sensing device 130 or individually. Figure 1 The number of sensing devices 130 shown is merely exemplary and not intended to be limiting. For example, sensing device 130 may be a camera to capture images of the scene in which robot 110 is located. In some embodiments, the scene in which robot 110 is located may refer to the surrounding environment experienced by robot 110 within a certain time frame, which may include static objects, dynamic objects, etc. Static objects may include, for example, roads, pedestrians, obstacles, buildings, shelves, and other objects that remain stationary over a relatively long period of time. Dynamic objects may include, for example, people, animals, other movable objects, and other objects that may move in real time.

[0023] In embodiments of this disclosure, robot 110 can be used for various suitable purposes, such as a transport robot for delivering goods. In this case, control device 120 can be used to plan the movement path of robot 110. Robot 110 can, for example, perform parcel delivery services. In this case, control device 120 can obtain information such as the parcel's origin, destination, recipient, and recipient's contact information. As an example, robot 110 can operate in an indoor environment, such as a warehouse environment.

[0024] exist Figure 1 In environment 100, control device 120 can be any type of microcontroller, programmable logic controller, industrial computer, single-board computer, field-programmable gate array, digital signal processor, multi-core processor, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.

[0025] In some embodiments, the control device 120 can control the robot 110 based on environmental information collected by the sensing device 130 and control commands generated by itself. In some embodiments, the operation of the robot 110 can be controlled via a server 150. The server 150 can receive relevant data for various tasks and control the operation of the robot 110 remotely. The sensing device 130 collects environmental information of the scene in which the robot 110 is located and sends the environmental information directly to the server 150 or via the control device 120 to the server 150. The server 150 can create a scene-related map based on the acquired scene-related environmental information, thereby providing data support for controlling the robot 110 to perform tasks. The control device 120 collects data from the robot 110 and communicates with the server 150 via the network 140 to send the data to the server 150. The server 150 determines the control commands for the robot based on the acquired data and sends the control commands to the control device 120. The control device 120 controls the robot 110 based on the received control commands. Server 150 can be any type of computing system / server capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0026] In some embodiments, robot 110 can connect to and communicate with server 150 via network 140. Robot 110 can send relevant data on various tasks performed by robot 110 to server 150 in real time.

[0027] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Furthermore, although in Figure 1The illustration shows a robot, but embodiments of this disclosure can also be applied to other types of movable objects.

[0028] As mentioned earlier, maps need to be built for robots to control them in performing tasks. Currently, most maps require the construction of a global model related to the scene and semantic information. On the one hand, the semantic information typically only contains basic attributes such as location and category, lacking sufficient richness. Using this map to control the robot's tasks will affect the accuracy and efficiency of its task execution. On the other hand, the amount of data in the scene-related global model is excessive, requiring significant memory usage.

[0029] This disclosure proposes a scheme for map construction. In this scheme, multiple nodes are created, each corresponding to one of multiple images acquired in a scene. The scene includes multiple landmark elements, and each node indicates at least one of the landmark elements identified from the image corresponding to that node, along with the pose of the image used for acquisition. Based on the multiple nodes, corresponding attributes of the landmark elements are determined. The attributes of each landmark element include pose information related to the landmark element, at least one node indicating the landmark element, and functional description information for the landmark element. A map for the scene is constructed based on the corresponding attributes of at least a portion of the landmark elements.

[0030] In this way, embodiments of this disclosure can determine the corresponding attributes of multiple landmark elements included in a scene based on multiple images collected in the scene, thereby constructing a map for the scene. This reduces the amount of content required for the map. Simultaneously, the attributes of the landmark elements include functional description information for those elements. In embodiments of this disclosure, such a map represents the scene based on the functional characteristics of the landmark elements, and therefore can also be called a functional map. This functional map differs from the form of a general map. Utilizing the functional information of the landmark elements can improve the accuracy and efficiency of the robot in performing tasks.

[0031] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0032] Example System

[0033] Figure 2 An architecture diagram of an example system 200 for constructing maps according to some embodiments of this disclosure is shown. Figure 2 As shown, scene-related images can be acquired through sensing device 130. Figure 2As shown, the images acquired by the sensing device 130 include a first image 210-1, a second image 210-2, and a third image 210-3, which can be referred to individually or collectively as image 210. Figure 1 The number of images 210 shown is merely exemplary and not intended to be limiting. In some embodiments, the sensing device 130 may be a graphics sensor, a depth sensor, or a radar sensor. Accordingly, the image 210 may include RGB image data, depth image data, and radar data. For example, if the sensing device 130 is a graphics sensor, environmental information about the location of the target robot 110 (e.g., a landmark image, scene image, etc. at the current location) is acquired through the graphics sensor. In some embodiments, multiple sensing devices of the same type may be present.

[0034] In some embodiments, if multiple sets of the same type of sensing devices 130 are provided, in response to a change in the position of the target robot 110, the sensing devices 130 will simultaneously acquire multiple sets of images 210 of the same type related to the position of the target robot 110. For example, the sensing devices 130 may contain multiple image sensors. If movement of the target robot 110 in the scene is detected, each image sensor will acquire one or more RGB images. The same information may be present in each RGB image.

[0035] In some embodiments, server 150 can create multiple nodes corresponding to multiple images collected in the scene, each corresponding to one of the multiple images. For example... Figure 2 As shown, the nodes corresponding to multiple images include the first node 230-1, the second node 230-2, and the third node 230-3, which can be referred to individually or collectively as node 230. Figure 2 The number of nodes 230 shown is merely exemplary and is not intended to be a limitation. A scene includes multiple iconic elements. As used herein, the term "iconic element" can refer to an element in a scene that serves a certain degree of identification, such as objects, signs, etc. For example, in an office scene, iconic elements could include desks, sofas, room signs, etc.

[0036] A node in a set of multiple nodes indicates at least one of a plurality of landmark elements identified from the image corresponding to that node, and the pose used to acquire the corresponding image. For example, a node can be in the form of {id: 24000, pose_body: [1, 2, 0.05], image_path: [], task_instruction_goal{}}. image_path represents the storage path of the image corresponding to that node; pose_body represents the pose used for the corresponding image, where the x-coordinate representing the position is 1, the y-coordinate representing the position is 2, and the orientation representing the pose is 0.05; task_instruction_goal{} represents the landmark elements included in the image corresponding to that node.

[0037] First, the system receives marker description information for iconic elements in the scene, indicating a demand for multiple iconic elements. In some embodiments, the marker description information may include the corresponding types of the multiple iconic elements. For example, the marker description information may be "obtain a landmark element of the doorplate type," "obtain a landmark element of the sofa type," etc. In some embodiments, the marker description information may include descriptions of the corresponding visual characteristics of the multiple iconic elements. For example, the marker description information may include "obtain a landmark element of the pillar type," "obtain a landmark element that is white." In some embodiments, the marker description information may include a corresponding description of the environment in which the multiple iconic elements are located. For example, the marker description information may be "obtain a landmark element that does not contact other objects," etc. In some embodiments, the description of visual characteristics may indicate the visual effect of the iconic element and the surrounding environment of the iconic element. For example, the description of visual characteristics may include the color, shape, and environment of the required iconic element, etc.

[0038] In some embodiments, the marker description information can be determined based on the type of map to be constructed. For example, if the type of map to be determined is a sparse map, then objects with a smaller number in the scene are designated as marker elements, and the marker description information is determined based on the corresponding marker elements. In some embodiments, the marker description information can be determined based on user input. For example, if the user input is "build a map based on independent objects in the scene," then the marker description information could be "obtain marker elements in the scene that do not contact other objects."

[0039] In some embodiments, a second machine learning model 240 can identify multiple landmark elements from multiple images based on landmark description information. For example, the landmark description information can be provided to the second machine learning model 240 as part of a cue word to obtain the output of the second machine learning model 240. Subsequently, multiple landmark elements corresponding to the landmark description information are determined based on the output of the second machine learning model 240. In some embodiments, if the number of a certain type of landmark element among the identified landmark elements is too large, that type of landmark element will not be used to construct the map to ensure the accuracy of the map.

[0040] In some embodiments, server 150 determines the corresponding attributes of multiple marker elements based on multiple nodes. For example... Figure 2 As shown, the attributes of multiple iconic elements include attribute 250-1 of the first iconic element, attribute 250-2 of the second iconic element, and attribute 250-3 of the third iconic element, which can be referred to individually or collectively as attribute 250. Figure 2 The number of attributes 250 shown is merely exemplary and not intended to be a limitation. In some embodiments, for one of a plurality of iconic elements, at least one node among a plurality of nodes indicating the iconic element may first be determined. Subsequently, attributes of the iconic element are determined based on relevant information of the at least one node indicating the iconic element. In some embodiments, the attributes of the iconic element include pose information associated with the iconic element, at least one node among a plurality of nodes indicating the iconic element, and functional description information for the iconic element. In some embodiments, the functional description information indicates the type and purpose of the iconic element. For example, if the iconic element is an elevator, the functional description information may be the type of elevator (such as elevator power, model, etc.), the elevator's capabilities (e.g., the elevator can move between different floors), and the elevator's purpose (e.g., the elevator is used to transport robots to different floors).

[0041] In some embodiments, functional description information corresponding to multiple iconic elements can be determined using a machine learning model based on the type of the iconic elements. For example, for one of the multiple iconic elements, the type of the iconic element is first determined. Then, based on the type and task information related to the iconic element, cue word information is generated for the first machine learning model 220. The cue word information is provided to the first machine learning model 220 to obtain its output. Based on the output of the first machine learning model 220, functional description information is determined. In some embodiments, the task information relates to the task performed by the robot in the current scene. For example, the task information may include iconic elements that the robot needs to use or contact during the performance of the task, or the robot's interaction actions with the iconic elements, etc.

[0042] In some embodiments, a database comprising multiple functional descriptions can be constructed based on pre-determined functional description information corresponding to different types of iconic elements. For example, all iconic elements in a scene (e.g., doorplates, elevators, etc.) can be pre-determined. Then, functional description information corresponding to each iconic element is determined to construct the database. For instance, for a particular iconic element, a functional description for that iconic element can be determined using publicly available data (e.g., internet pages) or internal data (e.g., introductory documents for that iconic element) and added to the database. Based on the type of the iconic element, the functional description corresponding to that iconic element is determined from the database as a predetermined functional description for that iconic element. For example, if an iconic element is determined to be a doorplate, a predetermined functional description corresponding to "doorplate" is determined from the database. Based on the predetermined functional description, at least a portion of the functional description information for that iconic element is determined. In some embodiments, images can be processed based on a machine learning model to determine the types of iconic elements included in the image.

[0043] In some embodiments, to further improve the accuracy and efficiency of robot task execution, at least one sample instruction related to the task can be added to the attributes of the iconic element. For example, if the iconic element is an elevator, the sample instruction related to the iconic element could include "go to the elevator," "go to the elevator to go up or down," or "find the elevator to change floors," etc. In some embodiments, for a particular iconic element among multiple iconic elements, a task related to that iconic element is determined based on the type of that iconic element. For example, in performing the task related to that iconic element, the robot needs to contact or interact with that iconic element. Then, candidate instructions for the determined task are retrieved from a set of candidate instructions for multiple tasks as at least one sample instruction. In this way, the attribute content of the iconic elements of different robots in the constructed map can be enriched, facilitating better use of the map to control the robot to perform tasks.

[0044] In some embodiments, the first machine learning model 220 can be a large language model. For example, the attributes of the landmark element could be {"landmark":{"object":"elevator","description":"This is an anelevator, which can be used to travel between..."} The `different"},nodes":[0001,0002,0008,0009],"average_pose":[x,y,yaw]}` field represents the functional description information of the landmark element (e.g., the type or purpose of the landmark element). Specifically, the `object` field indicates the type of the landmark element, for example, an elevator; the `description` field indicates the purpose of the landmark element, for example, the landmark element can be used to move between different floors. The `nodes` field includes the nodes that indicate the landmark element. Specifically, in this example, the nodes indicating the landmark element are numbered "0001,0002,0008,0009". The `average_pose` field represents the pose information associated with the landmark element. In this example, the x-coordinate represents the position, the y-coordinate represents the position, and the yaw represents the pose.

[0045] In some embodiments, pose information associated with a landmark element can be determined based on the pose of the node indicating the landmark element. In some embodiments, a landmark element may appear in at least two nodes, and these at least two nodes may indicate the same entity or different entities. In some embodiments, multiple entities corresponding to a landmark element may exist in the scene; for example, multiple identical tables may exist in the scene. Therefore, the pose of an entity can be determined based on the pose indicated by the node indicating the same entity. For example, at least two nodes indicating a landmark element can be divided into one or more node sets based on the difference between the poses indicated by the at least two nodes indicating a landmark element. For example, for each of the at least two nodes indicating a landmark element, the distance between that node and other nodes is determined based on the pose of the image indicated by that node. In some embodiments, one or more node sets can be obtained by performing a clustering operation on the at least two nodes indicating a landmark element. The pose difference between the nodes included in each node set is less than a difference threshold.

[0046] In some embodiments, if at least two nodes indicating a certain marker element are grouped into a single node set, it indicates that only one entity of that marker element exists in the scene. In this case, the pose corresponding to the entity can be determined by the average pose of the at least two nodes indicating the marker element. If at least two nodes indicating a certain marker element are grouped into multiple node sets, it indicates that multiple entities of that marker element exist in the scene. In this case, for one of the at least two node sets, the average pose corresponding to that node set is determined based on the poses of the nodes included in that node set. Subsequently, based on the average poses determined for each of the at least two node sets, the pose information of multiple entities of the marker element existing in the scene is determined.

[0047] In some embodiments, server 150 may construct map 260 based on the corresponding attributes of multiple iconic elements. Exemplarily, the constructed map is a collection including multiple iconic elements. The map includes multiple iconic elements and their corresponding attributes. In some embodiments, map 260 may be in the form of a list. The first item in the list is the identifier of the iconic element (e.g., ID or name), and the second item in the list is the corresponding attribute of the iconic element. In some embodiments, map 260 may be in the form of a knowledge graph, where the multiple iconic elements are multiple nodes in the graph.

[0048] In some embodiments, map 260 can be constructed based solely on landmark elements that appear less than a threshold number in the scene. For example, a set of landmark elements can be determined from multiple landmark elements based on their respective attributes, where the number of nodes corresponding to each landmark element in the set is less than a threshold number. A map is then constructed based on the corresponding attributes of this set of landmark elements. This approach prevents the same landmark element from appearing too frequently in the map, improving the accuracy and reliability of tasks performed using the map.

[0049] Example process

[0050] Figure 3 A flowchart of an example process 300 for map construction according to some embodiments of the present disclosure is shown. Process 300 can be implemented at server 150. Reference is made below. Figure 1 To describe process 300.

[0051] like Figure 3 As shown in box 310, server 150 creates multiple nodes corresponding to multiple images respectively based on multiple images acquired in the scene. The scene includes multiple landmark elements, and the nodes in the multiple nodes indicate at least one landmark element among the multiple landmark elements identified from the image corresponding to the node and the pose used to acquire the corresponding image.

[0052] In some embodiments, multiple iconic elements are determined by: receiving icon description information indicating a need for multiple iconic elements; and, based on the icon description information, using a second machine learning model to identify multiple iconic elements from multiple images.

[0053] In some embodiments, the logo description information includes at least one of the following: the corresponding types of the multiple logo elements, a description of the corresponding visual characteristics of the multiple logo elements, or a corresponding description of the environment in which the multiple logo elements are located.

[0054] In some embodiments, multiple images are acquired using a robot.

[0055] In box 320, server 150 determines the corresponding attributes of multiple iconic elements based on multiple nodes. The attributes of the iconic elements include pose information related to the iconic element, at least one node in the multiple nodes that indicates the iconic element, and functional description information for the iconic element.

[0056] In some embodiments, the functional description information for the iconic element is determined by at least one of the following: obtaining at least a portion of the functional description information for the iconic element using a first machine learning model based on the type of the iconic element, or determining at least a portion of the functional description information for the iconic element based on a predetermined functional description corresponding to the type of the iconic element.

[0057] In some embodiments, obtaining functional description information using a first machine learning model includes: generating cue word information for the first machine learning model based on the type and task information related to the marker element; providing the cue word information to the first machine learning model to obtain the output of the first machine learning model; and determining functional description information based on the output of the first machine learning model.

[0058] In some embodiments, the attributes of a landmark element among a plurality of landmark elements also include at least one sample instruction for a task associated with that landmark element.

[0059] In some embodiments, at least one sample instruction is obtained by: determining a task related to the marker element based on the type of the marker element; and retrieving, from a set of candidate instructions for multiple tasks, a candidate instruction for the determined task as at least one sample instruction.

[0060] In some embodiments, the pose information associated with the iconic element is determined by: determining at least two nodes from a plurality of nodes that indicate the iconic element; and determining pose information based on the poses indicated by the at least two nodes respectively.

[0061] In some embodiments, determining pose information includes: dividing at least two nodes into at least two node sets based on the difference between the poses indicated by at least two nodes; determining an average pose value corresponding to a node set based on the nodes included in the node set; and determining pose information based on the average pose values ​​determined for each of the at least two node sets.

[0062] In box 330, server 150 constructs a scene-specific map based on the corresponding attributes of at least some of the multiple iconic elements.

[0063] In some embodiments, constructing a map for a scene includes: determining a set of iconic elements from a plurality of iconic elements based on corresponding attributes of the multiple iconic elements, wherein the number of nodes among the multiple nodes indicating the iconic elements in the set of iconic elements is less than a threshold number; and constructing a map based on the corresponding attributes of the set of iconic elements.

[0064] Example devices and equipment

[0065] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an example apparatus 400 for map building according to certain embodiments of the present disclosure is shown. Apparatus 400 may be implemented as or included in server 150. Various modules / components in apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0066] like Figure 4 As shown, the device 400 includes a creation module 410, configured to create multiple nodes corresponding to multiple images acquired in a scene. The scene includes multiple landmark elements, and each node in the multiple nodes indicates at least one landmark element identified from the image corresponding to that node, and the pose used to acquire the corresponding image. The device 400 also includes a determination module 420, configured to determine corresponding attributes of the multiple landmark elements based on the multiple nodes. The attributes of the landmark elements include pose information related to the landmark element, at least one node in the multiple nodes indicating the landmark element, and functional description information for the landmark element. The device 400 also includes a construction module 430, configured to construct a map for the scene based on the corresponding attributes of at least a portion of the multiple landmark elements.

[0067] In some embodiments, the creation module is further configured to receive marker description information indicating a need for multiple marker elements; and, based on the marker description information, to use a second machine learning model to identify multiple marker elements from multiple images.

[0068] In some embodiments, the logo description information includes at least one of the following: the corresponding types of the multiple logo elements, a description of the corresponding visual characteristics of the multiple logo elements, or a corresponding description of the environment in which the multiple logo elements are located.

[0069] In some embodiments, multiple images are acquired using a robot.

[0070] In some embodiments, the determining module is further configured to determine at least one node from a plurality of nodes that indicates the iconic element; determine at least one image from a plurality of images that corresponds to the at least one node; and generate visual description information based on the at least one image using a first machine learning model, the visual description information indicating the visual effect of the iconic element and the surrounding environment in which the iconic element is located.

[0071] In some embodiments, the determining module is further configured to obtain at least a portion of the functional description information for the iconic element from the type of the iconic element using a first machine learning model, or to determine at least a portion of the functional description information for the iconic element based on a predetermined functional description corresponding to the type of the iconic element.

[0072] In some embodiments, the determining module is further configured to generate cue word information for a first machine learning model based on the type and task information related to the marker element; provide the cue word information to the first machine learning model to obtain the output of the first machine learning model; and determine functional description information based on the output of the first machine learning model.

[0073] In some embodiments, the attributes of a landmark element among a plurality of landmark elements also include at least one sample instruction for a task associated with that landmark element.

[0074] In some embodiments, the determining module is further configured to determine a task associated with the marker element based on the type of the marker element; and to retrieve candidate instructions for the determined task from a set of candidate instructions for multiple tasks as at least one sample instruction.

[0075] In some embodiments, the determining module is further configured to determine at least two nodes from a plurality of nodes that indicate the iconic element; and to determine pose information based on the poses indicated by the at least two nodes respectively.

[0076] In some embodiments, the determining module is further configured to divide at least two nodes into at least two node sets based on the difference between the poses indicated by at least two nodes; for a node set in the at least two node sets, determine an average pose value corresponding to the node set based on the nodes included in the node set; and determine pose information based on the average pose values ​​determined for each of the at least two node sets.

[0077] In some embodiments, the building module is further configured to determine a set of iconic elements from multiple iconic elements based on corresponding attributes of multiple iconic elements, wherein the number of nodes among multiple nodes indicating iconic elements in the set of iconic elements is less than a threshold number; and to build a map based on the corresponding attributes of the set of iconic elements.

[0078] Figure 5 A block diagram of an electronic device 500 capable of implementing various embodiments of the present disclosure is shown. (See diagram for example.) Figure 5As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0079] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0080] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0081] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0082] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0083] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0084] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0085] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0086] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0088] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A map construction method, comprising: Based on multiple images captured in the scene, multiple nodes are created that correspond to the multiple images respectively. The scene includes multiple landmark elements, and each node in the multiple nodes indicates at least one landmark element identified from the image corresponding to the node and the pose used to capture the corresponding image. Based on the plurality of nodes, corresponding attributes of the plurality of marker elements are determined. The attributes of a marker element include pose information related to that marker element, at least one node among the plurality of nodes indicating that marker element, and functional description information for that marker element; and A map for the scene is constructed based on the corresponding attributes of at least some of the plurality of iconic elements.

2. The method of claim 1, wherein the functional description information for the iconic element is determined by at least one of the following: Based on the type of the iconic element, at least a portion of the functional description information for the iconic element is obtained using a first machine learning model, or Based on the predetermined functional description corresponding to the type of the iconic element, at least a portion of the functional description information for the iconic element is determined.

3. The method according to claim 2, wherein obtaining the functional description information using the first machine learning model includes: Based on the type and task information related to the iconic element, generate prompt word information for the first machine learning model; The prompt word information is provided to a first machine learning model to obtain the output of the first machine learning model; and The functional description information is determined based on the output of the first machine learning model.

4. The method of claim 1, wherein the attribute of the iconic element among the plurality of iconic elements further includes at least one sample instruction for a task related to the iconic element.

5. The method of claim 4, wherein at least one sample instruction is obtained by: Based on the type of the iconic element, determine the task associated with that iconic element; and From a set of candidate instructions for multiple tasks, a candidate instruction for the determined task is retrieved as the at least one sample instruction.

6. The method of claim 1, wherein the pose information associated with the iconic element is determined by: Determine at least two nodes from the plurality of nodes that indicate the identifying element; and The pose information is determined based on the poses indicated by the at least two nodes respectively.

7. The method according to claim 6, wherein determining the pose information includes: Based on the difference between the poses indicated by the at least two nodes, the at least two nodes are divided into at least two node sets; For a node set in the at least two node sets, determine the average pose value corresponding to the node set based on the nodes included in the node set; as well as The pose information is determined based on the average pose values ​​determined for the at least two node sets respectively.

8. The method of claim 7, wherein constructing the map for the scene comprises: Based on the corresponding attributes of the plurality of iconic elements, a set of iconic elements is determined from the plurality of iconic elements, wherein the number of nodes in the plurality of nodes indicating the iconic elements in the set of iconic elements is less than a threshold number. as well as The map is constructed based on the corresponding attributes of the set of iconic elements.

9. The method of claim 1, wherein the plurality of identifying elements are determined by: Receive marker description information, the marker description information indicating the need for the plurality of marker elements; and Based on the marker description information, a second machine learning model is used to identify the multiple iconic elements from the multiple images.

10. The method of claim 9, wherein the flag description information includes at least one of the following: The corresponding types of the multiple iconic elements, Description of the corresponding visual features of the aforementioned multiple iconic elements, or The corresponding description of the environment in which the aforementioned iconic elements are located.

11. The method of claim 1, wherein the plurality of images are acquired using a robot.

12. An apparatus for map building, comprising: The creation module is configured to create multiple nodes corresponding to multiple images acquired in a scene, the scene including multiple landmark elements, and the nodes in the multiple nodes indicate at least one landmark element identified from the multiple landmark elements corresponding to the node and the pose used to acquire the corresponding image; The determination module is configured to determine the corresponding attributes of the plurality of landmark elements based on the plurality of nodes. The attributes of the landmark elements include pose information related to the landmark element, at least one node among the plurality of nodes indicating the landmark element, and functional description information for the landmark element. as well as A building module is configured to build a map for the scene based on the corresponding attributes of at least a portion of the plurality of iconic elements.

13. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images