Map Construction Method for Embodied Robot Control System, Embodied Robot and Medium

Abstract

The present application discloses a method for constructing a map of an embodied robot control system, an embodied robot, and a medium. The method includes obtaining 3D point cloud data and a first environmental image collected when the embodied robot moves in a home environment; obtaining a second environmental image collected by a camera in the home environment; constructing a target three-dimensional home map based on the 3D point cloud data, the first environmental image, and the second environmental image; when it is determined that the pose of a target object has changed according to the second pose of the target object in the target three-dimensional home map at the current moment and the third pose of the target object in the target three-dimensional home map at the previous moment, recording the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time to obtain a target four-dimensional home map. This method realizes the generation of a four-dimensional home map based on the pose changes of an object in a three-dimensional home map at different times, which is beneficial to improving the intelligent control level of a home robot.

Description

Technical Field

[0001] This application belongs to the technical field of smart home, and particularly relates to a method for constructing a map of an embodied robot control system, an embodied robot, and a medium. Background Art

[0002] Before an embodied robot executes a home task, it usually detects environmental data of the home environment through sensors during movement and constructs a two-dimensional map based on the detected environmental data.

[0003] Currently, during the process of an embodied robot executing a home task, it can only execute the set home tasks issued by the user according to the two-dimensional map, and the degree of intelligent control of the embodied robot is relatively low. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method for constructing a map of an embodied robot control system, an embodied robot, and a medium to overcome the above problems in the prior art.

[0005] In a first aspect, embodiments of this application provide a method for constructing a map of an embodied robot control system, including: obtaining 3D point cloud data and a first environmental image collected when the embodied robot moves in a home environment; fusing the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map; obtaining second environmental data collected by a camera in the home environment, where the second environmental data at least includes a second environmental image; matching the second environmental image and the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map; determining a target three-dimensional home map according to the first pose, the second environmental image, and the initial three-dimensional home map; obtaining the second pose of a target object in the target three-dimensional home map at the current moment; determining whether the pose of the target object has changed according to the third pose of the target object in the target three-dimensional home map at the previous moment and the second pose; when it is determined that the pose of the target object has changed, record the pose change of the target object from the third pose to the second pose in the target three-dimensional home map according to time dynamics to obtain a target four-dimensional home map.

[0006] Wherein, in some optional embodiments, the determining a target three-dimensional home map according to the first pose, the second environmental image, and the initial three-dimensional home map includes: determining the pose offset amount of each object in the second environmental image relative to the camera; determining the target three-dimensional home map according to the pose offset amount, the first pose, and the initial three-dimensional home map.

[0007] Among them, in some alternative embodiments, determining the pose offset of each object in the second environmental image relative to the camera includes: obtaining the depth point cloud data of each object; determining the pose offset according to the depth point cloud data.

[0008] Among them, in some alternative embodiments, determining the pose offset according to the depth point cloud data includes: constructing an object coordinate system corresponding to each object based on the geometric features of each object; determining the origin of the object coordinate system; aligning the object coordinate system and the camera coordinate system of the camera according to the depth point cloud data to obtain a rotation matrix from the object coordinate system to the camera coordinate system; obtaining a translation vector according to the position from the origin of the object coordinate system to the camera coordinate system; determining the pose offset according to the rotation matrix and the translation vector.

[0009] Among them, in some alternative embodiments, the second environmental data further includes the object type of each object, and determining the target three-dimensional home map according to the pose offset, the first pose, and the initial three-dimensional home map includes: determining the fourth pose of each object in the initial three-dimensional home map according to the pose offset and the first pose; performing semantic annotation on each object in the initial three-dimensional home map according to the fourth pose and the object type to obtain the target three-dimensional home map.

[0010] Among them, in some alternative embodiments, after determining that the pose of the target object has changed and recording the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time to obtain a target four-dimensional home map, the map construction method of the embodied robot control system further includes: determining a target home task according to the second pose and the third pose; sending the target home task to one or more embodied robots associated with the target object, so that the one or more embodied robots execute the target home task.

[0011] Among them, in some alternative embodiments, after determining that the pose of the target object has changed and recording the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time to obtain a target four-dimensional home map, the map construction method of the embodied robot control system further includes: generating a current movement trajectory of the target object according to the second pose and the third pose; generating a target movement trajectory according to the historical movement trajectory and the current movement trajectory of the target object.

[0012] Among them, in some alternative embodiments, the camera includes a plurality of sub-cameras, the plurality of sub-cameras are installed at different positions in the home environment, the second environmental image includes a plurality of third environmental images, and each third environmental image corresponds to a sub-camera. The step of matching the second environmental image and the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map includes: matching each third environmental image with the first environmental image to determine a first pose of each sub-camera in the initial three-dimensional map, so as to obtain a plurality of first poses; the step of determining the target three-dimensional home map according to the first pose, the second environmental image and the initial three-dimensional home map includes: determining the target three-dimensional home map according to the plurality of first poses, the plurality of third environmental images and the initial three-dimensional home map.

[0013] In a second aspect, an embodiment of the present application provides a map construction device for an embodied robot control system. The map construction device for the embodied robot control system includes a first acquisition module, a fusion module, a second acquisition module, a matching module, a first determination module, a third acquisition module, a second determination module, and a recording module. The first acquisition module is configured to acquire 3D point cloud data and a first environmental image collected when the embodied robot moves in a home environment; the fusion module is configured to fuse the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map; the second acquisition module is configured to acquire second environmental data collected by a camera in the home environment, and the second environmental data at least includes a second environmental image; the matching module is configured to match the second environmental image and the first environmental image to obtain a first pose of the camera in the initial three-dimensional home map; the first determination module is configured to determine a target three-dimensional home map according to the first pose, the second environmental image and the initial three-dimensional home map; the third acquisition module is configured to acquire a second pose of a target object in the target three-dimensional home map at the current moment; the second determination module is configured to determine whether the pose of the target object has changed according to the third pose of the target object in the target three-dimensional home map at the previous moment and the second pose; the recording module is configured to, when it is determined that the pose of the target object has changed, record the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time to obtain a target four-dimensional home map.

[0014] In a third aspect, an embodiment of the present application provides an embodied robot, including a memory; one or more processors coupled to the memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more application programs are configured to execute the map construction method of the embodied robot control system provided in the first aspect as described above.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which program code is stored, and the program code can be called by a processor to execute the map construction method of the embodied robot control system provided in the first aspect as described above.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer device, causes the computer device to execute the map construction method of the embodied robot control system provided in the first aspect as described above.

[0017] The solution provided by the present application obtains 3D point cloud data and a first environmental image collected when the embodied robot moves in a home environment, fuses the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map, obtains second environmental data collected by a camera in the home environment, the second environmental data at least includes a second environmental image, matches the second environmental image and the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map, determines a target three-dimensional home map according to the first pose, the second environmental image and the initial three-dimensional home map, obtains the second pose of the target object in the target three-dimensional home map at the current moment, and determines whether the pose of the target object has changed according to the third pose of the target object in the target three-dimensional home map at the previous moment and the second pose, and when it is determined that the pose of the target object has changed, records the pose change of the target object from the third pose to the second pose in the target three-dimensional home map according to time dynamics to obtain a target four-dimensional home map, realizing the generation of a four-dimensional home map according to the pose changes of an object in a three-dimensional home map at different times, and can autonomously control the corresponding embodied robot to execute corresponding home tasks according to the four-dimensional home map, which is beneficial to improving the intelligent control degree of the embodied robot.

[0018] Furthermore, constructing a three-dimensional home map based on the data collected by the embodied robot and the data collected by the camera in the home environment can suppress the problem that the data collected due to the height limitation of the embodied robot is missing information, resulting in an incomplete three-dimensional home map constructed only based on the data collected by the embodied robot and unable to reflect the environmental information of the entire home environment, and improves the construction integrity of the three-dimensional home map. Description of the Drawings

[0019] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0020] Figure 1 Fig. 5 shows a schematic diagram of a scenario of the embodied robot control system provided by an embodiment of the present application.

[0021] Figure 2 Fig. 9 shows a schematic flowchart of a method for map construction of the embodied robot control system provided by an embodiment of the present application.

[0022] Figure 3 Fig. 13 shows another schematic flowchart of a method for map construction of the embodied robot control system provided by an embodiment of the present application.

[0023] Figure 4 Fig. 17 shows yet another schematic flowchart of a method for map construction of the embodied robot control system provided by an embodiment of the present application.

[0024] Figure 5 Fig. 21 shows a structural block diagram of a map construction device of the embodied robot control system provided by an embodiment of the present application.

[0025] Figure 6 Fig. 25 shows a functional block diagram of an embodied robot provided by an embodiment of the present application.

[0026] Figure 7 Fig. 29 shows a computer-readable storage medium for storing or carrying program codes for implementing the method for map construction of the embodied robot control system provided by an embodiment of the present application.

[0027] Figure 8 Fig. 33 shows a computer program product for storing or carrying program codes for implementing the method for map construction of the embodied robot control system provided by an embodiment of the present application. Detailed Embodiments

[0028] To make the invention objectives, features, and advantages of the present application more obvious and understandable, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described below are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0029] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their combinations.

[0030] It should also be understood that the terms used in the specification of this application are for the purpose of describing particular embodiments only and are not intended to limit this application. As used in the specification of this application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0031] It should be further understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0032] In addition, in the description of this application, the terms "first", "second", "third", etc. are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0033] Before performing household tasks, an embodied robot usually detects environmental data of the household environment through sensors during movement and constructs a two-dimensional map based on the detected environmental data.

[0034] Currently, during the process of performing household tasks, an embodied robot can only execute the set household tasks issued by the user according to the two-dimensional map, and the degree of intelligent control of the embodied robot is relatively low.

[0035] In view of the above problems, the map construction method, embodied robot and medium of the embodied robot control system provided by the embodiments of the present application obtain 3D point cloud data and a first environmental image collected when the embodied robot moves in a home environment, fuse the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map, obtain second environmental data collected by a camera in the home environment, the second environmental data at least includes a second environmental image, match the second environmental image with the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map, determine a target three-dimensional home map according to the first pose, the second environmental image and the initial three-dimensional home map, obtain the second pose of a target object in the target three-dimensional home map at the current moment, and determine whether the pose of the target object has changed according to the third pose of the target object in the target three-dimensional home map at the previous moment and the second pose. When it is determined that the pose of the target object has changed, record the pose change of the target object from the third pose to the second pose in the target three-dimensional home map according to time dynamics to obtain a target four-dimensional home map, realizing the generation of a four-dimensional home map according to the pose changes of an object in a three-dimensional home map at different moments, and the corresponding embodied robot can be autonomously controlled to execute corresponding home tasks according to the four-dimensional home map, which is beneficial to improving the intelligent control degree of the embodied robot.

[0036] Furthermore, constructing a three-dimensional home map based on the data collected by the embodied robot and the data collected by the camera in the home environment can suppress the problem that the data collected is missing information due to the height limitation of the embodied robot, resulting in an incomplete three-dimensional home map constructed only based on the data collected by the embodied robot and unable to reflect the environmental information of the entire home environment, and improving the construction integrity of the three-dimensional home map.

[0037] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application.

[0038] Please refer to Figure 1 , which shows a schematic diagram of an application scenario of the embodied robot control system provided by the embodiments of the present application. The embodied robot control system may include an embodied robot 100, a camera 200, an object 300, a control device 400, etc. The embodied robot 100, the camera 200, the object 300, and the control device 400 are all installed in a home environment.

[0039] Among them, the embodied robot 100 is a movable home robot, and the movable home robot can move in the home environment. For example, the movable home robot may include at least any one of a cleaning robot (such as a sweeping robot, a mopping robot, and a sweeping and mopping integrated robot), a robotic arm, and a humanoid robot, which is not limited here.

[0040] The camera 200 can be used to collect environmental data of the home environment. The camera 200 can include any one of a wide-angle camera, a macro camera, an ultra-wide-angle camera, a panoramic camera, a depth camera, a monocular camera, or a binocular camera, etc. The type of the camera 200 is not limited here and can be specifically set according to actual needs.

[0041] The object 300 can include at least any one of a water cup, a sofa, a refrigerator, an air conditioner, a table and chair, a TV, a trash can, etc. The definition is not made here.

[0042] The control device 400 can be connected to the embodied robot 100 and the camera 200 through a network and perform data interaction with the embodied robot 100 and the camera 200 through the network.

[0043] The control device 400 can include any one of a terminal device and a server, etc. The type of the control device 400 is not limited here and can be specifically set according to actual needs.

[0044] The terminal device can be a mobile terminal device (for example, any one of a mobile phone, a personal digital assistant (PDA), a tablet personal computer (Tablet PC), a notebook computer, a smart watch, a smart bracelet, or a wearable device, etc.), or a fixed terminal device (a smart gateway device, a desktop computer, a smart panel, etc.). The definition is not made here.

[0045] The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), big data, or artificial intelligence platforms, etc. The definition is not made here.

[0046] The network can be any one of a ZigBee network, a Bluetooth (BT) network, a Wireless Fidelity (Wi-Fi) network, a Thread network, a Long Range Radio (LoRa) network, a Low-Power Wide-Area Network (LPWAN), an infrared network, a Narrow Band Internet of Things (NB-IoT), a Controller Area Network (CAN), a Digital Living Network Alliance (DLNA) network, a Wide Area Network (WAN), a Local Area Network (LAN), a Metropolitan Area Network (MAN), or a Wireless Personal Area Network (WPAN), etc., which is not limited here.

[0047] In some embodiments, the camera 200 may include multiple sub-cameras. The multiple sub-cameras can be installed at different positions in the home environment, and each sub-camera can be used to collect environmental data for different areas in the home environment.

[0048] The control device 400 can be connected to the multiple sub-cameras through the network and perform data interaction with the multiple sub-cameras through the network.

[0049] In some embodiments, the embodied robot control system may further include fixed home robots. The fixed home robots are fixedly installed in the home environment. For example, the fixed home robots may include at least any one of a curtain robot, a switch robot, and an intelligent lock, etc., which is not limited here.

[0050] Please refer to Figure 2 , which shows a flowchart of a map construction method for an embodied robot control system provided by an embodiment of the present application. In a specific embodiment, the map construction method of the embodied robot control system can be applied to the control device 400 in the embodied robot control system as shown in Figure 1 , and can also be applied to the embodied robot 100 as shown in Figure 1 . Taking the application to the control device 400 as an example, the process shown in Figure 2 will be elaborated in detail. The map construction method of the embodied robot control system may include the following steps 101 to step 108.

[0051] Step 101: Obtain the 3D point cloud data and the first environmental image collected when the embodied robot moves in the home environment.

[0052] In the embodiment of the present application, when the user needs to construct a map of the embodied robot control system, a composition instruction can be sent to the control device. The control device receives and responds to the composition instruction, sends a first acquisition instruction to the embodied robot through the network. The embodied robot receives and responds to the first acquisition instruction, moves and travels in the home environment, and collects environmental data of the home environment to obtain 3D point cloud data and the first environmental image, and sends the 3D point cloud data and the first environmental image to the control device through the network. The control device receives the 3D point cloud data and the first environmental image returned by the embodied robot until the embodied robot traverses all rooms in the home environment.

[0053] Among them, the embodied robot can be provided with a robot sensor and a robot camera, and the robot sensor and the robot camera move as the embodied robot moves and travels in the home environment.

[0054] The embodied robot can control the robot sensor to collect the 3D point cloud data of the objects that can be detected when the embodied robot moves in the home environment in real time or at regular intervals, and control the robot camera to collect the first environmental image when the embodied robot moves in the home environment in real time or at regular intervals.

[0055] In an implementation manner, the robot camera can also be controlled to collect the first environmental image when the embodied robot moves in the home environment in real time or at regular intervals, and the 3D point cloud data of the objects that can be detected is obtained according to the first environmental image.

[0056] The robot sensor can include any one of a lidar or a vision sensor, etc., and the robot camera can include any one of an RGB camera, a depth camera or a stereo camera, etc., which is not limited here.

[0057] In some implementation manners, the control device can detect the user's operation. When it is determined according to the detected user operation that the user has input a composition instruction for composing the embodied robot control system, that is, a composition instruction for composing the embodied robot control system is received. For example, when the user needs to construct a map of the embodied robot control system, a touch operation can be performed on the operation panel of the control device. The control device responds to the user's touch operation, generates a corresponding touch signal, and analyzes the touch signal. When it is determined that the touch signal is a preset signal for characterizing the composition of the embodied robot control system, it is determined that a composition instruction for composing the embodied robot control system is received.

[0058] In some embodiments, the control device may be provided with a voice recognition module. When a user needs to construct a map of the embodied robot control system, the user can send voice information within the voice collection range of the voice recognition module. The voice recognition module collects the voice information sent by the user, performs voice recognition on the collected voice information, and determines, according to the recognition result of the voice recognition, that the recognition result contains keywords for indicating the construction of the embodied robot control system map, such as "construct a map of the embodied robot control system", or for example, "embodied robot control system" and "map construction", etc., then it is determined that a map construction instruction for the embodied robot control system is received.

[0059] As an example, the voice information sent by the user is: construct a map of the embodied robot control system. Then the recognition result of the voice recognition contains the keywords "embodied robot control system" and "map construction", and it is determined that a map construction instruction for the embodied robot control system is received.

[0060] In some embodiments, the embodied robot control system may further include a client. The client is connected to the control device through a network and performs data interaction with the control device through the network.

[0061] When a user needs to construct a map of the embodied robot control system, a map construction instruction can be sent to the client. The client receives and responds to the map construction instruction, forwards the map construction instruction to the control device through the network, and the control device receives the map construction instruction forwarded by the client.

[0062] Among them, the client can be any one of a mobile client (for example, a mobile phone client, a PDA client, a Tablet PC client, a laptop client, a smart watch client, a smart bracelet client or a wearable client, etc.) or a fixed client (for example, a desktop computer client, a smart panel client, etc.). The type of the client is not limited here and can be specifically set according to actual needs.

[0063] Step 102: Integrate the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map.

[0064] In the embodiments of the present application, the control device can integrate the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map with image features of the home environment.

[0065] Specifically, the control device can depict an initial three-dimensional space image of the home environment according to the 3D point cloud data, determine the matching feature points between the initial three-dimensional space image and the first environmental image, and integrate the initial three-dimensional space image and the first environmental image according to the matching feature points to obtain an initial three-dimensional home map.

[0066] Step 103: Obtain second environmental data collected by a camera in a home environment.

[0067] In an embodiment of the present application, the control device may send a second acquisition instruction to the camera via a network. The camera receives and responds to the second acquisition instruction, collects environmental data for the home environment to obtain second environmental data, and sends the second environmental data to the control device via the network. The control device receives the second environmental data returned by the camera. Among them, the second environmental data may at least include a second environmental image, and the second environmental image contains objects that the embodied robot cannot detect. For example, as Figure 1 shown, due to height limitations, the embodied robot 100 (a sweeping robot) can detect the table, but cannot detect the water cup on the table. The camera 200 installed on the ceiling in the home environment can capture a second environmental image containing the water cup.

[0068] In some embodiments, the second environmental image may include a plurality of third environmental images. The camera may include a plurality of sub-cameras, and the plurality of sub-cameras may be installed at different positions in the home environment (such as at each corner of the ceiling, near the window, above the door, etc.). Each sub-camera may be used to collect environmental data for different areas in the home environment to obtain a plurality of third environmental images.

[0069] The control device may send a second acquisition instruction to the plurality of sub-cameras via a network. Each sub-camera receives and responds to the second acquisition instruction, collects environmental data for the corresponding area covered by each sub-camera in the home environment to obtain a third environmental image for each sub-camera, and sends the third environmental image to the control device via the network. The control device receives a third environmental image returned by each sub-camera to obtain a plurality of third environmental images.

[0070] Step 104: Match the second environmental image and the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map.

[0071] In an embodiment of the present application, the control device may match the image features in the second environmental image and the image features in the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map.

[0072] Among them, the first pose may include a first position and a first orientation, and the first orientation may be used to characterize the direction of the camera in the initial three-dimensional home map.

[0073] In some embodiments, the camera may include a plurality of sub-cameras, and the plurality of sub-cameras may be installed at different positions in the home environment. The second environmental image may include a plurality of third environmental images, and each third environmental image may correspond to a sub-camera.

[0074] The control device can match each third environmental image with the first environmental image to determine a first pose of each sub-camera in the initial three-dimensional map, so as to obtain a plurality of first poses.

[0075] Step 105: Determine the target three-dimensional home map according to the first pose, the second environmental image, and the initial three-dimensional home map.

[0076] In the embodiment of the present application, the control device can determine the pose offset of each object in the second environmental image relative to the camera, and determine the target three-dimensional home map according to the pose offset, the first pose, and the initial three-dimensional home map; specifically, it can determine the pose offset of the objects that the embodied robot cannot detect in the second environmental image, and determine the target three-dimensional home map according to the pose offset, the first pose, and the initial three-dimensional home map.

[0077] Constructing the target three-dimensional home map according to the pose offset of the object relative to the camera and the first pose of the camera in the initial three-dimensional home map can supplement the pose information of the objects not in the initial three-dimensional home map, improving the integrity of the target three-dimensional home map.

[0078] Among them, the pose offset can be used to represent the pose (i.e., position and direction) of each object relative to the camera in the second environmental image.

[0079] Regarding the process of the above control device determining the pose offset of each object in the second environmental image relative to the camera, in some embodiments, the control device can obtain the depth point cloud data of each object in the second environmental image and determine the pose offset according to the depth point cloud data, calculating the pose offset of each object according to the depth point cloud data, improving the calculation accuracy of the pose offset.

[0080] Among them, the control device can perform object recognition on the second environmental image and estimate the distance between the object and the camera according to the depth estimation algorithm to obtain the depth point cloud data. Optionally, the target detection algorithm (such as YOLO, SSD, Faster R-CNN, etc.) can be first used to identify and generate the bounding box of the object in the second environmental image, then obtain the 3D point cloud data of the scene corresponding to the camera, further project the 3D point cloud data onto the second environmental image, and according to the bounding box of the object, screen out the pixel points located within the bounding box of the object in the projected second environmental image, and then extract the 3D point cloud data corresponding to these pixel points, that is, obtain the 3D point cloud data of the object.

[0081] The control device can construct an object coordinate system corresponding to each object based on the geometric features of each object, determine the origin of the object coordinate system, and align the object coordinate system with the camera coordinate system of the camera according to the depth point cloud data to obtain the rotation matrix from the object coordinate system to the camera coordinate system. Then, according to the position from the origin of the object coordinate system to the camera coordinate system, the translation vector can be obtained, and the pose offset can be determined based on the rotation matrix and the translation vector. By aligning and translating the object coordinate system based on the depth point cloud data to obtain the pose offset, the accuracy of the pose offset is improved. The geometric features can include shape information for characterizing the three-dimensional features of the object, and position and contour information for characterizing the two-dimensional features of the object, etc.

[0082] In some embodiments, the second environmental data may further include the object types of each object. The control device can determine the fourth pose of each object in the initial three-dimensional home map according to the pose offset and the first pose, and perform semantic annotation on each object in the initial three-dimensional home map according to the fourth pose and the object type to obtain the target three-dimensional home map. Semantically annotating the objects in the three-dimensional home map based on the pose and type of the objects to obtain the three-dimensional home map enriches the semantic information of the three-dimensional home map and is beneficial to improving the control accuracy of controlling the embodied robot according to the three-dimensional home map.

[0083] Among them, the control device can calculate the sum of the pose offset and the first pose to obtain the fourth pose; the control device can extract image features from the second environmental image, input the extracted features into a classification model (such as ResNet, Inception, etc.) to identify the type of the object, and then use a large language model (such as GPT-3) to generate a semantic interpretation of the recognition result, and let the large language model generate natural language, so as to perform semantic annotation on the object to obtain a target three-dimensional home map with semantic annotation information for each object.

[0084] The fourth pose may include a fourth position and a fourth orientation, and the fourth orientation can be used to characterize the direction of the object in the target three-dimensional home map.

[0085] The object type may include at least any one of a first type for characterizing a water cup, a second type for characterizing a sofa, a third type for characterizing a refrigerator, a fourth type for characterizing an air conditioner, a fifth type for characterizing a table and chair, a sixth type for characterizing a television, and a seventh type for characterizing a trash can, etc., which is not limited here.

[0086] In some embodiments, the camera may include multiple sub-cameras, and the multiple sub-cameras may be installed at different positions in the home environment. The second environmental image may include multiple third environmental images, and each third environmental image may correspond to a sub-camera.

[0087] The control device can determine the target 3D home map based on multiple first poses, multiple third environmental images, and the initial 3D home map.

[0088] Among them, the target 3D home map can be constructed according to the pose offset of the same object relative to multiple sub-cameras and the multiple first poses of the multiple sub-cameras in the initial 3D home map, improving the accuracy of the target 3D home map. Since the multiple sub-cameras are installed at different positions in the home environment, the regions corresponding to the third environmental images collected by each sub-camera are different. Therefore, the target 3D home map can also be constructed according to the different objects in the different third environmental images collected by the multiple sub-cameras, the pose offset relative to the corresponding sub-cameras, and the corresponding first poses of the corresponding sub-cameras in the initial 3D home map, improving the integrity of the target 3D home map.

[0089] Step 106: Obtain the second pose of the target object in the target 3D home map at the current moment.

[0090] In the embodiment of the present application, the control device can send a third acquisition instruction to the camera and / or the embodied robot through the network at the current moment. The camera and / or the embodied robot receive and respond to the third acquisition instruction, perform image acquisition on the home environment to obtain the fourth environmental image, and send the fourth environmental image to the control device through the network. The control device receives the fourth environmental image returned by the camera, performs image recognition on the target object in the fourth environmental image, estimates the distance between the target object and the camera according to the depth estimation algorithm to obtain the current depth point cloud data, constructs the target object coordinate system corresponding to the target object based on the geometric features of the target object, determines the origin of the target object coordinate system, aligns the target object coordinate system and the camera coordinate system according to the current depth point cloud data to obtain the current rotation matrix from the target object coordinate system to the camera coordinate system, obtains the current offset according to the position from the origin of the target object coordinate system to the camera coordinate system, and obtains the second pose of the target object in the target 3D home map according to the current rotation matrix, the current offset, and the first pose.

[0091] Among them, the current rotation matrix can be used to represent the rotation direction of the target object relative to the first orientation, and the current offset can be used to represent the distance of the target object relative to the camera.

[0092] The second pose can include a second position and a second orientation, and the second orientation can be used to represent the direction of the target object in the target 3D home map at the current moment.

[0093] Step 107: Determine whether the pose of the target object has changed according to the third pose and the second pose of the target object in the target 3D home map at the previous moment.

[0094] In an embodiment of the present application, the control device may obtain the third pose of the target object in the target three-dimensional home map at the previous moment, and determine whether the pose of the target object has changed according to the third pose and the second pose.

[0095] The third pose is calculated based on the fifth environmental image of the target object captured by the camera at the previous moment. The process of calculating the third pose according to the fifth environmental image is similar to the process of calculating the second pose according to the fourth environmental image, which will not be elaborated here.

[0096] The third pose may include a third position and a third orientation. The third orientation may be used to represent the direction of the target object in the target three-dimensional home map at the previous moment.

[0097] The control device may calculate the position difference between the third position and the second position, and the orientation difference between the third orientation and the second orientation respectively, and determine whether the pose of the target object has changed according to the orientation difference. The orientation difference may be used to represent the angular difference in the direction of the target object in the target three-dimensional home map at different moments.

[0098] When the position difference is greater than the position difference value and / or the orientation difference is greater than the orientation difference threshold, it is determined that the pose of the target object has changed; when the position difference is less than or equal to the position difference value and the orientation difference is less than or equal to the orientation difference threshold, it is determined that the pose of the target object has not changed.

[0099] Step 108: When it is determined that the pose of the target object has changed, record the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time, and obtain the target four-dimensional home map.

[0100] In an embodiment of the present application, when the control device determines that the pose of the target object has changed, it may record the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time, and obtain the target four-dimensional home map; optionally, the time corresponding to the previous moment and the current moment, as well as the third pose and the second pose, may be recorded, realizing the generation of a four-dimensional home map according to the pose change of an object in a three-dimensional home map at different moments. The corresponding embodied robot may be autonomously controlled to perform corresponding home tasks according to the pose change of the target object in the four-dimensional home map, which is beneficial to improving the intelligent control degree of the embodied robot.

[0101] Furthermore, a three-dimensional home map is constructed based on the data collected by the embodied robot and the data collected by the cameras in the home environment, which can inhibit the problem that the data collected due to the height limitation of the embodied robot is missing information, resulting in an incomplete three-dimensional home map constructed only based on the data collected by the embodied robot and unable to reflect the environmental information of the entire home environment, and improve the construction integrity of the three-dimensional home map.

[0102] Please refer to Figure 3 , which shows a flowchart of a map construction method for an embodied robot control system provided by another embodiment of the present application. In a specific embodiment, the map construction method of the embodied robot control system can be applied to a control device 400 in an embodied robot control system as shown in Figure 1 . Taking the control device 400 as an example, the process shown in Figure 3 will be elaborated in detail below. The map construction method of the embodied robot control system may include the following steps 201 to step 210.

[0103] Step 201: Obtain the 3D point cloud data and the first environmental image collected when the embodied robot moves in the home environment.

[0104] Step 202: Fuse the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map.

[0105] Step 203: Obtain the second environmental data collected by the cameras in the home environment.

[0106] Step 204: Match the second environmental image and the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map.

[0107] Step 205: Determine the target three-dimensional home map according to the first pose, the second environmental image, and the initial three-dimensional home map.

[0108] Step 206: Obtain the second pose of the target object in the target three-dimensional home map at the current moment.

[0109] Step 207: Determine whether the pose of the target object has changed according to the third pose and the second pose of the target object in the target three-dimensional home map at the previous moment.

[0110] Step 208: When it is determined that the pose of the target object has changed, record the pose change of the target object from the third pose to the second pose in the target three-dimensional home map according to time dynamics to obtain a target four-dimensional home map.

[0111] In this embodiment, steps 201, 202, 203, 204, 205, 206, 207, and 208 may refer to the corresponding steps in the foregoing embodiments, and will not be elaborated herein.

[0112] Step 209: Determine the target home task according to the second pose and the third pose.

[0113] In this embodiment, when the control device determines that the pose of the target object has changed, it records the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time. After obtaining the four-dimensional home map, it can determine the target pose change of the target object according to the second pose and the third pose, and search the preset home task table according to the target pose change to obtain the target home task.

[0114] Among them, the target home task may include at least one of cleaning tasks and putting-back tasks, etc., and is not limited herein.

[0115] The preset home task table can be used to represent the correspondence between the object and its pose change and the home task. For example, the objects may include object 1, object 2, object 3, object 4, and object 5, the pose changes may include pose change 1, pose change 2, and pose change 3, and the home tasks may include task 1, task 2, task 3, task 4, task 5, task 6, task 7, task 8, task 9, task 10, task 11, task 12, task 13, task 14, and task 15.

[0116] The correspondence between the object and its pose change and the home task can be shown in Table 1, that is, the preset home task table. According to this correspondence, the target home task can be obtained.

[0117] Table 1

[0118]

[0119] It should be noted that the correspondence between the object and its pose change and the home task is not limited to that shown in Table 1, and can be specifically set according to actual needs.

[0120] Step 210: Send the target home task to one or more embodied robots and / or fixed home robots associated with the target object, so that one or more embodied robots and / or fixed home robots execute the target home task.

[0121] In this embodiment, the control device can send a target home task to one or more embodied robots and / or fixed home robots associated with the target object through a network. The one or more embodied robots and / or fixed home robots receive and execute the target home task, realizing autonomous control of the corresponding embodied robot to execute the corresponding target home task according to the pose change of the target object, and improving the intelligent control level of the embodied robot.

[0122] In some embodiments, the target object can be a water cup, the target home task can be a cleaning task, the second pose is the water cup falling to the ground, and the third pose is the water cup placed on the table. The embodied robot can include a humanoid robot and a cleaning robot associated with the water cup.

[0123] The control device can send the cleaning task to the humanoid robot and the cleaning robot through the network. The humanoid robot receives and responds to the cleaning task, picks up the water cup from the ground and places it back on the table. The cleaning robot receives and responds to the cleaning task, and mops the floor where the water cup has fallen.

[0124] In some embodiments, the target object can be a trash can, the target home task can be a repositioning task, the second pose is the trash can placed in the corner of the coffee table, and the third pose is the trash can placed in the corner of the living room. The embodied robot can include a humanoid robot associated with the trash can.

[0125] The control device can send the repositioning task to the humanoid robot through the network. The humanoid robot receives and responds to the repositioning task, and moves the trash can from the corner of the coffee table to the corner of the living room.

[0126] The solution provided in this embodiment autonomously controls the corresponding embodied robot to execute the corresponding target home task according to the pose change of the target object, and improves the intelligent control level of the embodied robot.

[0127] Please refer to Figure 4 , which shows a flowchart of a map construction method for an embodied robot control system provided in another embodiment of the present application. In a specific embodiment, the map construction method of the embodied robot control system can be applied to a control device 400 in an embodied robot control system as shown in Figure 1 . Taking the control device 400 as an example, the following will elaborate in detail on the process shown in Figure 4 . The map construction method of the embodied robot control system can include the following steps 301 to step 310.

[0128] Step 301: Obtain 3D point cloud data and a first environmental image collected when the embodied robot moves in the home environment.

[0129] Step 302: Fuse the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map.

[0130] Step 303: Obtain second environmental data collected by a camera in the home environment.

[0131] Step 304: Match the second environmental image with the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map.

[0132] Step 305: Determine the target three-dimensional home map according to the first pose, the second environmental image, and the initial three-dimensional home map.

[0133] Step 306: Obtain the second pose of the target object in the target three-dimensional home map at the current moment.

[0134] Step 307: Determine whether the pose of the target object has changed according to the third pose and the second pose of the target object in the target three-dimensional home map at the previous moment.

[0135] Step 308: When it is determined that the pose of the target object has changed, record the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time to obtain the target four-dimensional home map.

[0136] In this embodiment, Steps 301, 302, 303, 304, 305, 306, 307, and 308 may refer to the corresponding steps in the foregoing embodiments, and will not be elaborated here.

[0137] Step 309: Generate the current movement trajectory of the target object according to the second pose and the third pose.

[0138] In this embodiment, when the control device determines that the pose of the target object has changed, after recording the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time to obtain the four-dimensional home map, the current movement trajectory of the target object may be generated according to the second pose and the third pose.

[0139] Wherein, the current movement trajectory may include a current translation trajectory and / or a current rotation trajectory. The current translation trajectory may be used to represent the path of the target object translated from the third position to the second position, and the current rotation trajectory may be used to represent the angular change of the target object rotated from the third orientation to the second orientation.

[0140] Step 310: Generate a target movement trajectory according to the historical movement trajectory and the current movement trajectory of the target object.

[0141] In this embodiment, the control device can add the current movement trajectory of the target object to the historical movement trajectory to update the latest movement trajectory of the target object, obtain the target movement trajectory, and save the target movement trajectory in the target four-dimensional home map, where the target movement trajectory of the target object can be displayed in the target four-dimensional home map.

[0142] Among them, the historical movement trajectory can include a historical translation trajectory and / or a historical rotation trajectory. The historical translation trajectory can be used to represent the historical movement path corresponding to the target object, and the historical rotation trajectory can be used to represent the historical angle change corresponding to the target object.

[0143] The solution provided in this embodiment, based on the real-time update of the historical pose change according to the current pose change of the target object, is beneficial to improving the control accuracy of controlling the embodied robot. And the user can find the target object according to the target movement trajectory of the target object in the target four-dimensional home map. For example, the user can find the mobile phone according to the movement trajectory of the mobile phone, which improves the convenience.

[0144] Please refer to Figure 5 , which shows a map construction device 500 of an embodied robot control system provided in an embodiment of the present application. In a specific embodiment, the map construction device 500 of the embodied robot control system can be applied to a control device 400 in an embodied robot control system as shown in Figure 1 . Taking the control device 400 as an example, the map construction device 500 of the embodied robot control system shown in Figure 5 will be elaborated in detail. The map construction device 500 of the embodied robot control system can include a first acquisition module 501, a fusion module 502, a second acquisition module 503, a matching module 504, a first determination module 505, a third acquisition module 506, a second determination module 507, and a recording module 508.

[0145] The first acquisition module 501 can be used to acquire 3D point cloud data and the first environmental image collected when the embodied robot moves in the home environment; the fusion module 502 can be used to fuse the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map; the second acquisition module 503 can be used to acquire second environmental data collected by a camera in the home environment, and the second environmental data can at least include a second environmental image; the matching module 504 can be used to match the second environmental image and the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map; the first determination module 505 can be used to determine a target three-dimensional home map according to the first pose, the second environmental image, and the initial three-dimensional home map; the third acquisition module 506 can be used to acquire the second pose of the target object in the target three-dimensional home map at the current moment; the second determination module 507 can be used to determine whether the pose of the target object has changed according to the third pose of the target object in the target three-dimensional home map at the previous moment and the second pose; the recording module 508 can be used to, when it is determined that the pose of the target object has changed, record the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time to obtain a target four-dimensional home map.

[0146] In some embodiments, the first determination module 505 may include a first determination unit and a second determination unit.

[0147] The first determination unit can be used to determine the pose offset of each object in the second environmental image relative to the camera; the second determination unit can be used to determine the target three-dimensional home map according to the pose offset, the first pose, and the initial three-dimensional home map.

[0148] In some embodiments, the first determination unit may include an acquisition subunit and a first determination subunit.

[0149] The acquisition subunit can be used to acquire the depth point cloud data of each object; the first determination subunit can be used to determine the pose offset according to the depth point cloud data.

[0150] In some embodiments, the first determination subunit may include a construction sub-subunit, a first determination sub-subunit, an alignment sub-subunit, a translation sub-subunit, and a second determination sub-subunit.

[0151] The construction secondary unit can be used to construct an object coordinate system corresponding to each object based on the geometric features of each object; the first determination secondary unit can be used to determine the origin of the object coordinate system; the alignment secondary unit can be used to align the object coordinate system and the camera coordinate system of the camera according to the depth point cloud data to obtain the rotation matrix from the object coordinate system to the camera coordinate system; the translation secondary unit can be used to obtain the translation vector according to the position from the origin of the object coordinate system to the camera coordinate system; the second determination secondary unit can be used to determine the pose offset according to the rotation matrix and the translation vector.

[0152] In some embodiments, the second environmental data may further include the object types of each object, and the second determination unit may include a second determination secondary unit and a labeling secondary unit.

[0153] The second determination secondary unit can be used to determine the fourth pose of each object in the initial three-dimensional home map according to the pose offset and the first pose; the labeling secondary unit can be used to perform semantic labeling on each object in the initial three-dimensional home map according to the fourth pose and the object type to obtain the target three-dimensional home map.

[0154] In some embodiments, the map construction device 500 of the embodied robot control system may further include a third determination module and a sending module.

[0155] The third determination module can be used to, when it is determined that the pose of the target object has changed, after the recording module 508 records the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time to obtain the target four-dimensional home map, determine the target home task according to the second pose and the third pose; the sending module can be used to send the target home task to one or more embodied robots associated with the target object, so that one or more embodied robots execute the target home task.

[0156] In some embodiments, the map construction device 500 of the embodied robot control system may further include a generation module and an update module.

[0157] The generation module can be used to, when it is determined that the pose of the target object has changed, after the recording module 508 records the pose change of the target object from the third pose to the second pose in the target three-dimensional home map dynamically according to time to obtain the target four-dimensional home map, generate the current movement trajectory of the target object according to the second pose and the third pose; the update module can be used to generate the target movement trajectory according to the historical movement trajectory and the current movement trajectory of the target object.

[0158] In some embodiments, the camera may include multiple sub-cameras, which may be installed at different positions in a home environment. The second environmental image may include multiple third environmental images, and each third environmental image may correspond to a sub-camera.

[0159] The matching module 504 may include a matching unit, and the first determination module 505 may include a third determination unit.

[0160] The matching unit may be configured to match each third environmental image with the first environmental image to determine a first pose of each sub-camera in the initial three-dimensional map, so as to obtain multiple first poses. The third determination unit may be configured to determine the target three-dimensional home map based on the multiple first poses, the multiple third environmental images, and the initial three-dimensional home map.

[0161] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference can be made to each other. For device embodiments, since they are basically similar to method embodiments, they are described relatively simply. For the relevant parts, reference can be made to the description of the method embodiments. For any processing method described in the method embodiments, it can be implemented by the corresponding processing module in the device embodiments, and will not be elaborated one by one in the device embodiments.

[0162] In addition, in each embodiment of the present application, the various functional modules may be integrated into one processing module, or each module may exist physically alone, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0163] Please refer to Figure 6 , which shows a functional block diagram of an embodied robot 600 provided by an embodiment of the present application. The embodied robot 600 may include one or more of the following components: a memory 610, a processor 620, and one or more application programs, where one or more application programs may be stored in the memory 610 and configured to be executed by one or more processors 620. One or more application programs are configured to execute the method described in the foregoing method embodiments.

[0164] The memory 610 may include a Random Access Memory (RAM), or may also include a Read-Only Memory. The memory 610 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 610 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as obtaining 3D point cloud data, obtaining a first environmental image, obtaining second environmental data, fusing data and images, matching the second environmental image and the first environmental image, obtaining a first pose, determining a target three-dimensional home map, obtaining a second pose, determining whether a change has occurred, determining that a change has occurred, recording pose changes, obtaining a target four-dimensional home map, determining a pose offset, obtaining depth point cloud data, constructing an object coordinate system, determining an origin, aligning the object coordinate system and the camera coordinate system of a camera, obtaining a translation vector, determining a fourth pose, semantic annotation, obtaining a target three-dimensional home map, generating a current movement trajectory, and generating a target movement trajectory, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created during the use of the embodied robot 600 (such as the embodied robot, the home environment, 3D point cloud data, the first environmental image, the camera, the second environmental data, the second environmental image, the initial three-dimensional home map, the first pose, the target three-dimensional home map, the current moment, the target object, the second pose, the previous moment, the third pose, the pose change, the target four-dimensional home map, the pose offset, the depth point cloud data, geometric features, the object coordinate system, the camera coordinate system, the rotation matrix, the offset vector, the object type, the fourth pose, the current movement trajectory, the historical movement trajectory, and the target movement trajectory).

[0165] The processor 620 may include one or more processing cores. The processor 620 connects various parts within the embodied robot 600 through various interfaces and lines. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 610, and by calling data stored in the memory 610, it performs various functions of the embodied robot 600 and processes data. Optionally, the processor 620 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 620 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing display content; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the processor 620 and may be implemented separately through a communication chip.

[0166] Please refer to Figure 7 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program code 710 is stored in the computer-readable storage medium 700, and the program code 710 can be called by a processor to execute the method described in the above method embodiment.

[0167] The computer-readable storage medium 700 may be an electronic memory such as a flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 700 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 700 has a storage space for the program code 710 that executes any method step in the above method. These program codes can be read out from or written into one or more computer program products. The program code 710 may be compressed in an appropriate form, for example.

[0168] Please refer to Figure 8, which shows a structural block diagram of a computer program product 800 provided by an embodiment of the present application. The computer program product 800 includes a computer program / instructions 810, and the computer program / instructions 810 are stored in a computer-readable storage medium of a computer device. When the computer program product 800 runs on the computer device, the processor of the computer device reads the computer program / instructions 810 from the computer-readable storage medium, and the processor executes the computer program / instructions 810, so that the computer device executes the method described in the above method embodiment.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions away from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for map construction of an embodied robot control system, characterized in that, Including: Obtaining 3D point cloud data and a first environmental image collected when an embodied robot moves in a home environment; Fusing the 3D point cloud data and the first environmental image to obtain an initial three-dimensional home map; Obtaining second environmental data collected by a camera installed in the home environment, where the second environmental data at least includes a second environmental image, and the second environmental image contains objects that the embodied robot cannot detect; Matching the second environmental image and the first environmental image to obtain a first pose of the camera in the initial three-dimensional home map; Determining a target three-dimensional home map based on the first pose, the second environmental image, and the initial three-dimensional home map to supplement pose information of objects not present in the initial three-dimensional home map; Obtaining a second pose of a target object in the target three-dimensional home map at the current moment; Determining whether the pose of the target object has changed based on a third pose of the target object in the target three-dimensional home map at the previous moment and the second pose; When it is determined that the pose of the target object has changed, recording the times corresponding to the previous moment and the current moment, as well as the third pose and the second pose, to dynamically record in the target three-dimensional home map the pose change of the target object from the third pose to the second pose according to time, obtaining a target four-dimensional home map generated based on the pose changes of the target object in the target three-dimensional home map at different times, and controlling a corresponding embodied robot to perform corresponding home tasks based on the pose changes of the target object in the target four-dimensional home map.

2. The map construction method according to claim 1, wherein The determining the target three-dimensional home map based on the first pose, the second environmental image, and the initial three-dimensional home map includes: Determining pose offset amounts of respective objects in the second environmental image relative to the camera; Determining the target three-dimensional home map based on the pose offset amounts, the first pose, and the initial three-dimensional home map.

3. The map construction method according to claim 2, wherein The determining the pose offset amounts of respective objects in the second environmental image relative to the camera includes: Obtaining depth point cloud data of the respective objects; Determining the pose offset amounts based on the depth point cloud data.

4. The map construction method according to claim 3, characterized in that The determining the pose offset amounts based on the depth point cloud data includes: Constructing an object coordinate system corresponding to each of the objects based on geometric features of the respective objects; Determining an origin of the object coordinate system; Aligning the object coordinate system and a camera coordinate system of the camera based on the depth point cloud data to obtain a rotation matrix from the object coordinate system to the camera coordinate system; Obtaining a translation vector based on a position from the origin of the object coordinate system to the camera coordinate system; Determining the pose offset amounts based on the rotation matrix and the translation vector.

5. The map construction method according to claim 2, characterized in that The second environmental data further includes object types of the respective objects, and the determining the target three-dimensional home map based on the pose offset amounts, the first pose, and the initial three-dimensional home map includes: Determine the fourth pose of each object in the initial three-dimensional home map according to the pose offset and the first pose; Semantically annotate each object in the initial three-dimensional home map according to the fourth pose and the object type to obtain the target three-dimensional home map.

6. The map construction method according to any one of claims 1 to 5, characterized in that When it is determined that the pose of the target object has changed, record the time corresponding to the previous moment and the current moment, as well as the third pose and the second pose, so as to dynamically record in the target three-dimensional home map the pose change of the target object from the third pose to the second pose according to time. After obtaining the target four-dimensional home map generated according to the pose changes of the target object in the target three-dimensional home map at different times, the map construction method further includes: Generate the current movement trajectory of the target object according to the second pose and the third pose; Generate a target movement trajectory according to the historical movement trajectory and the current movement trajectory of the target object.

7. The map construction method according to any one of claims 1 to 5, characterized in that The camera includes a plurality of sub-cameras, the plurality of sub-cameras are installed at different positions in the home environment, the second environmental image includes a plurality of third environmental images, and each third environmental image corresponds to a sub-camera. The step of matching the second environmental image and the first environmental image to obtain the first pose of the camera in the initial three-dimensional home map includes: Match each third environmental image with the first environmental image to determine a first pose of each sub-camera in the initial three-dimensional home map, so as to obtain a plurality of first poses; The step of determining the target three-dimensional home map according to the first pose, the second environmental image and the initial three-dimensional home map includes: Determine the target three-dimensional home map according to the plurality of first poses, the plurality of third environmental images and the initial three-dimensional home map.

8. An embodied robot, characterized in that, Including: A memory; One or more processors, coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the map construction method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that Program code is stored in the computer-readable storage medium, and the program code can be called by the processor to execute the map construction method according to any one of claims 1 to 7.

Images