Map construction method of control system of robot with body, robot with body and medium

Abstract

The invention discloses a map construction method of a body-equipped robot control system, a body-equipped robot and a medium. The method comprises the following steps: acquiring 3D point cloud data and a first environment image acquired when the body-equipped robot moves in a home environment; acquiring a second environment image acquired by a camera in the home environment; constructing a target three-dimensional home map according to the 3D point cloud data, the first environment image and the second environment image; and determining that the posture of the target object is changed according to a second posture of the target object in the target three-dimensional home map at the current moment and a third posture of the target object in the target three-dimensional home map at the previous moment. And dynamically recording the pose change of the target object from the third pose to the second pose in the target three-dimensional home map according to the time to obtain a target four-dimensional home map. According to the method, the four-dimensional home map is generated according to the pose changes of the object in the three-dimensional home map at different moments, and the intelligent control degree of the home robot is improved.

Description

Technical Field

[0001] The present application belongs to the field of smart home technology, and in particular relates to a map construction method for an embodied robot control system, an embodied robot, and a medium. Background Art

[0002] Before performing household tasks, the embodied robot usually detects environmental data of the home environment through sensors while moving, and creates a two-dimensional map based on the detected environmental data.

[0003] At present, when performing household tasks, embodied robots can only execute household tasks set by users based on two-dimensional maps, and the level of intelligent control of embodied robots is relatively low. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a map construction method for an embodied robot control system, an embodied robot, and a medium to overcome the above problems of the prior art.

[0005] In a first aspect, an embodiment of the present application provides a map construction method for an embodied robot control system, comprising: obtaining 3D point cloud data and a first environment image collected when the embodied robot moves in a home environment; fusing the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map; obtaining second environment data collected by a camera in the home environment, the second environment data at least including a second environment image; matching the second environment image with the first environment image to obtain a first posture of the camera in the initial three-dimensional home map; determining a target three-dimensional home map based on the first posture, the second environment image and the initial three-dimensional home map; obtaining a second posture of a target object in the target three-dimensional home map at the current moment; determining whether the posture of the target object has changed based on the third posture of the target object in the target three-dimensional home map at the previous moment and the second posture; when it is determined that the posture of the target object has changed, recording the posture change of the target object from the third posture to the second posture in the target three-dimensional home map according to time dynamics to obtain a target four-dimensional home map.

[0006] Among them, in some optional embodiments, determining the target three-dimensional home map based on the first posture, the second environmental image and the initial three-dimensional home map includes: determining the posture offset of each object in the second environmental image relative to the camera; determining the target three-dimensional home map based on the posture offset, the first posture and the initial three-dimensional home map.

[0007] Among them, in some optional embodiments, determining the posture offset of each object in the second environment image relative to the camera includes: acquiring depth point cloud data of each object; and determining the posture offset based on the depth point cloud data.

[0008] Among them, in some optional embodiments, determining the pose offset according to the depth point cloud data includes: constructing an object coordinate system corresponding to each object based on the geometric features of each object; determining the origin of the object coordinate system; aligning the object coordinate system with the camera coordinate system of the camera according to the depth point cloud data to obtain a rotation matrix from the object coordinate system to the camera coordinate system; obtaining a translation vector according to the position from the origin of the object coordinate system to the camera coordinate system; and determining the pose offset according to the rotation matrix and the translation vector.

[0009] Among them, in some optional embodiments, the second environmental data also includes the object type of each object, and determining the target three-dimensional home map according to the posture offset, the first posture and the initial three-dimensional home map includes: determining a fourth posture of each object in the initial three-dimensional home map according to the posture offset and the first posture; and semantically annotating each object in the initial three-dimensional home map according to the fourth posture and the object type to obtain the target three-dimensional home map.

[0010] Among them, in some optional embodiments, when it is determined that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map according to time dynamics. After the target four-dimensional home map is obtained, the map construction method of the embodied robot control system also includes: determining a target home task according to the second posture and the third posture; sending the target home task to one or more embodied robots associated with the target object, so that the one or more embodied robots perform the target home task.

[0011] Among them, in some optional embodiments, when it is determined that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map according to time dynamics, and after the target four-dimensional home map is obtained, the map construction method of the embodied robot control system also includes: generating the current movement trajectory of the target object according to the second posture and the third posture; generating the target movement trajectory according to the historical movement trajectory of the target object and the current movement trajectory.

[0012] Among them, in some optional embodiments, the camera includes multiple sub-cameras, and the multiple sub-cameras are installed at different positions in the home environment. The second environment image includes multiple third environment images, and each third environment image corresponds to a sub-camera. The matching of the second environment image with the first environment image to obtain the first position of the camera in the initial three-dimensional home map includes: matching each third environment image with the first environment image to determine a first position of each sub-camera in the initial three-dimensional map to obtain multiple first position; determining the target three-dimensional home map according to the first position, the second environment image and the initial three-dimensional home map includes: determining the target three-dimensional home map according to the multiple first position, the multiple third environment images and the initial three-dimensional home map.

[0013] In the second aspect, an embodiment of the present application provides a map construction device for an embodied robot control system, wherein the map construction device for the embodied robot control system includes a first acquisition module, a fusion module, a second acquisition module, a matching module, a first determination module, a third acquisition module, a second determination module and a recording module. a first acquisition module for acquiring 3D point cloud data and a first environment image collected when the embodied robot moves in a home environment; a fusion module for fusing the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map; a second acquisition module for acquiring second environment data collected by a camera in the home environment, the second environment data at least including a second environment image; a matching module for matching the second environment image with the first environment image to obtain a first position of the camera in the initial three-dimensional home map; a first determination module for determining a target three-dimensional home map according to the first position, the second environment image and the initial three-dimensional home map; a third acquisition module for acquiring a second position of a target object in the target three-dimensional home map at a current moment; a second determination module for determining whether the posture of the target object has changed according to the third position and the second position of the target object in the target three-dimensional home map at a previous moment; a recording module for recording the posture change of the target object from the third position to the second position in the target three-dimensional home map according to time dynamics when it is determined that the posture of the target object has changed, so as to obtain a target four-dimensional home map.

[0014] In a third aspect, an embodiment of the present application provides an embodied robot, comprising a memory; one or more processors coupled to the memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the map construction method of the embodied robot control system provided in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute a map construction method for an embodied robot control system as provided in the first aspect above.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer device, enables the computer device to execute the map construction method for the embodied robot control system provided in the first aspect above.

[0017] The solution provided by the present application obtains 3D point cloud data and a first environment image collected when the embodied robot moves in a home environment, and fuses the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map, and obtains second environment data collected by a camera in the home environment, the second environment data at least includes a second environment image, and matches the second environment image with the first environment image to obtain a first position of the camera in the initial three-dimensional home map, and determines a target three-dimensional home map according to the first position, the second environment image and the initial three-dimensional home map, and obtains a second position of the target object in the target three-dimensional home map at the current moment, and determines whether the posture of the target object has changed according to the third position and the second position of the target object in the target three-dimensional home map at the previous moment, and when it is determined that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map according to the time dynamics to obtain a target four-dimensional home map, thereby realizing the generation of a four-dimensional home map according to the posture change of the object in the three-dimensional home map at different moments, and the corresponding embodied robot can be autonomously controlled to perform corresponding home tasks according to the four-dimensional home map, which is conducive to improving the intelligent control level of the embodied robot.

[0018] Furthermore, constructing a three-dimensional home map based on the data collected by the embodied robot and the data collected by the cameras in the home environment can suppress the problem of information missing in the collected data due to the height limitation of the embodied robot, resulting in the three-dimensional home map constructed only based on the data collected by the embodied robot being incomplete and unable to reflect the environmental information of the entire home environment, thereby improving the construction integrity of the three-dimensional home map. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a scenario of an embodied robot control system provided in an embodiment of the present application is shown.

[0021] Figure 2 A flow chart of a map construction method for an embodied robot control system provided in an embodiment of the present application is shown.

[0022] Figure 3 Another flowchart of a map construction method for an embodied robot control system provided in an embodiment of the present application is shown.

[0023] Figure 4 Another flowchart of the map construction method of the embodied robot control system provided in the embodiment of the present application is shown.

[0024] Figure 5 A structural block diagram of a map building device of an embodied robot control system provided in an embodiment of the present application is shown.

[0025] Figure 6 A functional block diagram of the embodied robot provided in an embodiment of the present application is shown.

[0026] Figure 7 A computer-readable storage medium provided in an embodiment of the present application for storing or carrying program code for implementing a map construction method for an embodied robot control system provided in an embodiment of the present application is shown.

[0027] Figure 8 A computer program product provided in an embodiment of the present application for storing or carrying program codes for implementing a map construction method for an embodied robot control system provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0030] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0031] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0032] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0033] Before performing household tasks, the embodied robot usually detects environmental data of the home environment through sensors while moving, and creates a two-dimensional map based on the detected environmental data.

[0034] At present, when performing household tasks, embodied robots can only execute household tasks set by users based on two-dimensional maps, and the level of intelligent control of embodied robots is relatively low.

[0035] In response to the above problems, the map construction method of the embodied robot control system, the embodied robot and the medium provided in the embodiments of the present application obtain the 3D point cloud data and the first environment image collected when the embodied robot moves in the home environment, and fuse the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map, and obtain the second environment data collected by the camera in the home environment, the second environment data at least includes the second environment image, and match the second environment image with the first environment image to obtain the first posture of the camera in the initial three-dimensional home map, and determine the target three-dimensional home map according to the first posture, the second environment image and the initial three-dimensional home map, and obtain The second posture of the target object in the target three-dimensional home map at the current moment, and according to the third posture and the second posture of the target object in the target three-dimensional home map at the previous moment, determine whether the posture of the target object has changed; and when it is determined that the posture of the target object has changed, record the posture change of the target object from the third posture to the second posture in the target three-dimensional home map according to the time dynamics, and obtain the target four-dimensional home map, thereby realizing the generation of a four-dimensional home map according to the posture change of the object in the three-dimensional home map at different moments, and the corresponding embodied robot can be autonomously controlled according to the four-dimensional home map to perform corresponding home tasks, which is conducive to improving the intelligent control level of the embodied robot.

[0036] Furthermore, constructing a three-dimensional home map based on the data collected by the embodied robot and the data collected by the cameras in the home environment can suppress the problem of information missing in the collected data due to the height limitation of the embodied robot, resulting in the three-dimensional home map constructed only based on the data collected by the embodied robot being incomplete and unable to reflect the environmental information of the entire home environment, thereby improving the construction integrity of the three-dimensional home map.

[0037] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0038] See also Figure 1 , which shows a schematic diagram of an application scenario of the embodied robot control system provided in an embodiment of the present application. The embodied robot control system may include an embodied robot 100, a camera 200, an object 300, and a control device 400, etc. The embodied robot 100, the camera 200, the object 300, and the control device 400 are all installed in a home environment.

[0039] Among them, the embodied robot 100 is a mobile home robot, which can move in a home environment. For example, the mobile home robot may include at least any one of a cleaning robot (such as a sweeping robot, a mopping robot, and a sweeping and mopping robot), a robotic arm, and a humanoid robot, etc., which is not limited here.

[0040] Camera 200 can be used to collect environmental data of a home environment. Camera 200 may include any one of a wide-angle camera, a macro camera, an ultra-wide-angle camera, a panoramic camera, a depth camera, a monocular camera or a binocular camera. The type of camera 200 is not limited here and can be set according to actual needs.

[0041] Object 300 may include at least any one of a water cup, a sofa, a refrigerator, an air conditioner, a table and chairs, a television, and a trash can, etc., which is not limited here.

[0042] The control device 400 can be connected to the embodied robot 100 and the camera 200 through a network, and perform data exchange with the embodied robot 100 and the camera 200 through the network.

[0043] The control device 400 may include any one of a terminal device and a server, etc. The type of the control device 400 is not limited here and can be specifically set according to actual needs.

[0044] The terminal device can be a mobile terminal device (for example, any one of a mobile phone, a personal digital assistant (PDA), a tablet personal computer (Tablet PC), a laptop computer, a smart watch, a smart bracelet or a wearable device, etc.), or a fixed terminal device (smart gateway device, desktop computer, smart panel, etc.), etc., and is not limited here.

[0045] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be any cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), big data or artificial intelligence platforms, etc., without limitation here.

[0046] The network may be any one of a ZigBee network, a Bluetooth (BT) network, a Wireless Fidelity (Wi-Fi) network, a Thread network, a Long Range Radio (LoRa) network, a Low-Power Wide-Area Network (LPWAN), an infrared network, a Narrow Band Internet of Things (NB-IoT), a Controller Area Network (CAN), a Digital Living Network Alliance (DLNA) network, a Wide Area Network (WAN), a Local Area Network (LAN), a Metropolitan Area Network (MAN), or a Wireless Personal Area Network (WPAN), etc., without limitation here.

[0047] In some embodiments, the camera 200 may include multiple sub-cameras, which may be installed at different locations in a home environment, and each sub-camera may be used to collect environmental data for different areas in the home environment.

[0048] The control device 400 can be connected to multiple sub-cameras via a network, and perform data exchange with the multiple sub-cameras via the network.

[0049] In some embodiments, the embodied robot control system may also include a fixed home robot, which is fixedly installed in a home environment. For example, the fixed home robot may include at least any one of a curtain robot, a switch robot, and a smart lock, etc., which is not limited here.

[0050] See also Figure 2 , which shows a flow chart of a map construction method of an embodied robot control system provided by an embodiment of the present application. In a specific embodiment, the map construction method of the embodied robot control system can be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. 1 can also be applied to Figure 1 The embodied robot 100 shown in FIG. 1 is used in a control device 400 as an example. Figure 2 The process shown in FIG. 1 is described in detail, and the map construction method of the embodied robot control system may include the following steps 101 to 108.

[0051] Step 101: Acquire 3D point cloud data and a first environment image collected when the embodied robot moves in a home environment.

[0052] In an embodiment of the present application, when the user needs to build a map of the embodied robot control system, a composition instruction can be sent to the control device. The control device receives and responds to the composition instruction and sends a first acquisition instruction to the embodied robot through the network. The embodied robot receives and responds to the first acquisition instruction, moves in the home environment, and collects environmental data of the home environment to obtain 3D point cloud data and a first environmental image. The 3D point cloud data and the first environmental image are sent to the control device through the network. The control device receives the 3D point cloud data and the first environmental image returned by the embodied robot until the embodied robot has traversed all rooms in the home environment.

[0053] The embodied robot may be provided with a robot sensor and a robot camera, and the robot sensor and the robot camera move along with the embodied robot as it moves in the home environment.

[0054] The embodied robot can control the robot sensor to collect 3D point cloud data of objects that the embodied robot can detect when moving in the home environment in real time or at a scheduled time, and control the robot camera to collect the first environment image when the embodied robot moves in the home environment in real time or at a scheduled time.

[0055] In one embodiment, the robot camera can also be controlled to capture a first environment image of the embodied robot when it moves in a home environment in real time or at a fixed time, and 3D point cloud data of detectable objects can be obtained based on the first environment image.

[0056] The robot sensor may include any one of a laser radar or a visual sensor, and the robot camera may include any one of an RGB camera, a depth camera or a stereo camera, etc., which are not limited here.

[0057] In some embodiments, the control device may detect the user's operation, and when it is determined based on the detected user operation that the user has input a composition instruction for composing the embodied robot control system, the composition instruction for composing the embodied robot control system is received. For example, when the user needs to construct a map of the embodied robot control system, the user may perform a touch operation on the operation panel of the control device, and the control device responds to the user's touch operation, generates a corresponding touch signal, and analyzes the touch signal. When it is determined that the touch signal is a preset signal for representing the composition of the embodied robot control system, it is determined that the composition instruction for composing the embodied robot control system is received.

[0058] In some embodiments, the control device may be provided with a voice recognition module. When a user needs to construct a map of the embodied robot control system, the user may send a voice message within the voice collection range of the voice recognition module. The voice recognition module collects the voice message sent by the user, and performs voice recognition on the collected voice message. Based on the recognition result of the voice recognition, it is determined that the recognition result contains keywords for indicating the composition of the embodied robot control system, such as "construct a map of the embodied robot control system", and for example, "embodied robot control system" and "composition", etc., then it is determined that a composition instruction for composing the embodied robot control system has been received.

[0059] As an example, the voice information sent by the user is: compose the embodied robot control system, and the recognition result of the voice recognition contains the keywords "embodied robot control system" and "compose", then it is determined that a composition instruction for composing the embodied robot control system is received.

[0060] In some embodiments, the embodied robot control system may further include a client, which is connected to the control device via a network and exchanges data with the control device via the network.

[0061] When the user needs to build a map of the embodied robot control system, he can send a composition instruction to the client. The client receives and responds to the composition instruction, forwards the composition instruction to the control device through the network, and the control device receives the composition instruction forwarded by the client.

[0062] Among them, the client can be any one of a mobile client (for example, a mobile phone client, a PDA client, a Tablet PC client, a laptop client, a smart watch client, a smart bracelet client or a wearable client, etc.) or a fixed client (for example, a desktop computer client, a smart panel client, etc.). The type of client is not limited here and can be set according to actual needs.

[0063] Step 102: Fusing the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map.

[0064] In an embodiment of the present application, the control device can fuse the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map with image features of the home environment.

[0065] Specifically, the control device can depict an initial three-dimensional spatial image of the home environment based on the 3D point cloud data, determine the matching feature points between the initial three-dimensional spatial image and the first environment image, and fuse the initial three-dimensional spatial image and the first environment image based on the matching feature points to obtain an initial three-dimensional home map.

[0066] Step 103: Acquire second environment data collected by a camera in the home environment.

[0067] In the embodiment of the present application, the control device can send a second acquisition instruction to the camera via the network, the camera receives and responds to the second acquisition instruction, collects environmental data of the home environment, obtains second environmental data, and sends the second environmental data to the control device via the network, and the control device receives the second environmental data returned by the camera. The second environmental data can at least include a second environmental image, and the second environmental image contains objects that the embodied robot cannot detect, such as Figure 1 As shown, the embodied robot 100 (cleaning robot) can detect the table due to height limitation, but cannot detect the water cup on the table. The camera 200 installed on the ceiling in the home environment can capture the second environment image containing the water cup.

[0068] In some embodiments, the second environment image may include multiple third environment images. The camera may include multiple sub-cameras, which may be installed at different locations in the home environment (e.g., various corners of the ceiling, near windows, on doors, etc.), and each sub-camera may be used to collect environment data for different areas in the home environment to obtain multiple third environment images.

[0069] The control device can send a second acquisition instruction to multiple sub-cameras via the network. Each sub-camera receives and responds to the second acquisition instruction, collects environmental data for the corresponding area covered by each sub-camera in the home environment, obtains a third environmental image for each sub-camera, and sends the third environmental image to the control device via the network. The control device receives a third environmental image returned by each sub-camera to obtain multiple third environmental images.

[0070] Step 104: Match the second environment image with the first environment image to obtain the first position of the camera in the initial three-dimensional home map.

[0071] In an embodiment of the present application, the control device may match the image features in the second environment image with the image features in the first environment image to obtain the first position of the camera in the initial three-dimensional home map.

[0072] The first posture may include a first position and a first orientation, and the first orientation may be used to characterize the direction of the camera in the initial three-dimensional home map.

[0073] In some embodiments, the camera may include multiple sub-cameras, and the multiple sub-cameras may be installed at different locations in a home environment. The second environment image may include multiple third environment images, and each third environment image may correspond to a sub-camera.

[0074] The control device may match each third environment image with the first environment image, determine a first position of each sub-camera in the initial three-dimensional map, and obtain multiple first positions.

[0075] Step 105: Determine a target three-dimensional home map according to the first posture, the second environment image and the initial three-dimensional home map.

[0076] In an embodiment of the present application, the control device can determine the pose offset of each object in the second environment image relative to the camera, and determine the target three-dimensional home map based on the pose offset, the first pose and the initial three-dimensional home map; specifically, it can determine the pose offset of objects in the second environment image that cannot be detected by the embodied robot relative to the camera, and determine the target three-dimensional home map based on the pose offset, the first pose and the initial three-dimensional home map.

[0077] According to the position offset of the object relative to the camera and the first position of the camera in the initial 3D home map, a target 3D home map is constructed, which can supplement the position information of the object not in the initial 3D home map and improve the integrity of the target 3D home map.

[0078] The pose offset may be used to characterize the pose (ie, position and direction) of each object in the second environment image relative to the camera.

[0079] Regarding the process of the above-mentioned control device determining the posture offset of each object in the second environment image relative to the camera, in some embodiments, the control device can obtain the depth point cloud data of each object in the second environment image, and determine the posture offset based on the depth point cloud data, and calculate the posture offset of each object based on the depth point cloud data, thereby improving the calculation accuracy of the posture offset.

[0080] Among them, the control device can perform object recognition on the second environment image, and estimate the distance between the object and the camera according to the depth estimation algorithm to obtain depth point cloud data. Optionally, the target detection algorithm (such as YOLO, SSD, Faster R-CNN, etc.) can be used to first identify and generate the bounding box of the object in the second environment image, and then obtain the 3D point cloud data of the scene corresponding to the camera, and further project the 3D point cloud data into the second environment image, and according to the bounding box of the object, the pixels located in the bounding box of the object are screened out in the projected second environment image, and then the 3D point cloud data corresponding to these pixels are extracted, that is, the 3D point cloud data of the object is obtained.

[0081] The control device can construct the object coordinate system corresponding to each object based on the geometric features of each object, determine the origin of the object coordinate system, and align the object coordinate system with the camera coordinate system of the camera according to the depth point cloud data, obtain the rotation matrix from the object coordinate system to the camera coordinate system, and obtain the translation vector according to the position from the origin of the object coordinate system to the camera coordinate system, and determine the pose offset according to the rotation matrix and the translation vector, align and translate the object coordinate system based on the depth point cloud data, and obtain the pose offset, thereby improving the accuracy of the pose offset. The geometric features may include shape information for characterizing the three-dimensional features of the object, and position and contour information for characterizing the two-dimensional features of the object.

[0082] In some embodiments, the second environment data may also include the object type of each object. The control device may determine the fourth posture of each object in the initial three-dimensional home map based on the posture offset and the first posture, and semantically annotate each object in the initial three-dimensional home map based on the fourth posture and the object type to obtain a target three-dimensional home map. The three-dimensional home map is obtained by semantically annotating the objects in the three-dimensional home map based on the posture and type of the objects, which enriches the semantic information of the three-dimensional home map and is conducive to improving the control accuracy of the embodied robot according to the three-dimensional home map.

[0083] Among them, the control device can calculate the sum of the posture offset and the first posture to obtain the fourth posture; the control device can extract image features from the second environment image, and input the extracted features into a classification model (for example, ResNet, Inception, etc.) to identify the type of object, and then use a large language model (such as GPT-3) to generate a semantic interpretation of the recognition result, so that the large language model generates natural language, thereby semantically annotating the objects to obtain a target three-dimensional home map with semantic annotation information for each object.

[0084] The fourth posture may include a fourth position and a fourth orientation, and the fourth orientation may be used to characterize the direction of the object in the target three-dimensional home map.

[0085] The object type may include at least any one of a first type for characterizing a water cup, a second type for characterizing a sofa, a third type for characterizing a refrigerator, a fourth type for characterizing an air conditioner, a fifth type for characterizing a table and chair, a sixth type for characterizing a television, and a seventh type for characterizing a trash can, etc., without limitation here.

[0086] In some embodiments, the camera may include multiple sub-cameras, and the multiple sub-cameras may be installed at different locations in a home environment. The second environment image may include multiple third environment images, and each third environment image may correspond to a sub-camera.

[0087] The control device can determine the target three-dimensional home map according to multiple first postures, multiple third environment images and the initial three-dimensional home map.

[0088] Among them, the target three-dimensional home map can be constructed based on the position offset of the same object relative to multiple sub-cameras and the multiple first poses of the multiple sub-cameras in the initial three-dimensional home map, thereby improving the accuracy of the target three-dimensional home map. Since the multiple sub-cameras are installed in different positions in the home environment, the areas corresponding to the third environment images collected by each sub-camera are different. Therefore, the target three-dimensional home map can also be constructed based on the position offset of different objects in different third environment images collected by the multiple sub-cameras relative to the corresponding sub-camera, and the corresponding first poses of the corresponding sub-camera in the initial three-dimensional home map, thereby improving the integrity of the target three-dimensional home map.

[0089] Step 106: Obtain the second position of the target object in the target three-dimensional home map at the current moment.

[0090] In an embodiment of the present application, the control device can send a third acquisition instruction to the camera and / or the embodied robot through the network at the current moment. The camera and / or the embodied robot receives and responds to the third acquisition instruction, captures images of the home environment, obtains a fourth environment image, and sends the fourth environment image to the control device through the network. The control device receives the fourth environment image returned by the camera, performs image recognition on the target object in the fourth environment image, and estimates the distance between the target object and the camera according to the depth estimation algorithm to obtain the current depth point cloud data, and constructs a target object coordinate system corresponding to the target object based on the geometric features of the target object, and determines the origin of the target object coordinate system, and aligns the target object coordinate system with the camera coordinate system of the camera according to the current depth point cloud data to obtain the current rotation matrix from the target object coordinate system to the camera coordinate system, and obtains the current offset according to the position from the origin of the target object coordinate system to the camera coordinate system, and obtains the second pose of the target object in the target three-dimensional home map according to the current rotation matrix, the current offset and the first pose.

[0091] The current rotation matrix may be used to represent the rotation direction of the target object relative to the first orientation, and the current offset may be used to represent the distance of the target object relative to the camera.

[0092] The second posture may include a second position and a second orientation, and the second orientation may be used to characterize the direction of the target object in the target three-dimensional home map at the current moment.

[0093] Step 107: Determine whether the posture of the target object has changed according to the third posture and the second posture of the target object in the target three-dimensional home map at the previous moment.

[0094] In an embodiment of the present application, the control device can obtain the third posture of the target object in the target three-dimensional home map at the previous moment, and determine whether the posture of the target object has changed based on the third posture and the second posture.

[0095] Among them, the third posture is calculated based on the fifth environmental image of the target object captured by the camera at the previous moment. The process of calculating the third posture according to the fifth environmental image is similar to the process of calculating the second posture according to the fourth environmental image, and will not be repeated here.

[0096] The third posture may include a third position and a third orientation, and the third orientation may be used to characterize the direction of the target object in the target three-dimensional home map at a previous moment.

[0097] The control device can calculate the position difference between the third position and the second position, and the orientation difference between the third orientation and the second orientation, and determine whether the posture of the target object has changed according to the orientation difference. The orientation difference can be used to characterize the angular difference of the direction of the target object in the target three-dimensional home map at different times.

[0098] When the position difference is greater than the position difference value and / or the orientation difference value is greater than the orientation difference threshold, it is determined that the posture of the target object has changed; when the position difference is less than or equal to the position difference value and the orientation difference value is less than or equal to the orientation difference threshold, it is determined that the posture of the target object has not changed.

[0099] Step 108: When it is determined that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map according to time dynamics to obtain the target four-dimensional home map.

[0100] In an embodiment of the present application, when the control device determines that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture can be recorded in the target three-dimensional home map according to the time dynamics to obtain the target four-dimensional home map; optionally, the time corresponding to the previous moment and the current moment, as well as the third posture and the second posture can be recorded, thereby realizing the generation of a four-dimensional home map according to the posture changes of the object in the three-dimensional home map at different times, and the corresponding embodied robot can be autonomously controlled to perform corresponding home tasks according to the changes in the posture of the target object in the four-dimensional home map, which is conducive to improving the degree of intelligent control of the embodied robot.

[0101] Furthermore, constructing a three-dimensional home map based on the data collected by the embodied robot and the data collected by the cameras in the home environment can suppress the problem of information missing in the collected data due to the height limitation of the embodied robot, resulting in the three-dimensional home map constructed only based on the data collected by the embodied robot being incomplete and unable to reflect the environmental information of the entire home environment, thereby improving the construction integrity of the three-dimensional home map.

[0102] See also Figure 3 , which shows a flowchart of a map construction method of an embodied robot control system provided by another embodiment of the present application. In a specific embodiment, the map construction method of the embodied robot control system can be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. Figure 3 The process shown is described in detail, and the map construction method of the embodied robot control system may include the following steps 201 to 210.

[0103] Step 201: Acquire 3D point cloud data and a first environment image collected when the embodied robot moves in a home environment.

[0104] Step 202: Fusing the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map.

[0105] Step 203: Acquire second environment data collected by a camera in the home environment.

[0106] Step 204: Match the second environment image with the first environment image to obtain the first position of the camera in the initial three-dimensional home map.

[0107] Step 205: Determine a target three-dimensional home map according to the first posture, the second environment image and the initial three-dimensional home map.

[0108] Step 206: Obtain the second position of the target object in the target three-dimensional home map at the current moment.

[0109] Step 207: Determine whether the posture of the target object has changed according to the third posture and the second posture of the target object in the target three-dimensional home map at the previous moment.

[0110] Step 208: When it is determined that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map according to time dynamics to obtain the target four-dimensional home map.

[0111] In this embodiment, step 201, step 202, step 203, step 204, step 205, step 206, step 207 and step 208 can refer to the contents of the corresponding steps in the above-mentioned embodiments, which will not be repeated here.

[0112] Step 209: Determine the target home task according to the second posture and the third posture.

[0113] In this embodiment, when the control device determines that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map according to time dynamics. After obtaining the four-dimensional home map, the target posture change of the target object can be determined according to the second posture and the third posture, and the preset home task table can be searched according to the target posture change to obtain the target home task.

[0114] Among them, the target home task may include at least any one of a cleaning task and a returning task, etc., which is not limited here.

[0115] The preset home task table can be used to characterize the correspondence between objects and their posture changes and home tasks. For example, objects may include object 1, object 2, object 3, object 4, and object 5, posture changes may include posture change 1, posture change 2, and posture change 3, and home tasks may include task 1, task 2, task 3, task 4, task 5, task 6, task 7, task 8, task 9, task 10, task 11, task 12, task 13, task 14, and task 15.

[0116] The correspondence between objects and their position changes and home tasks can be shown in Table 1, that is, a preset home task table, and the target home task can be obtained according to the correspondence.

[0117] Table 1

[0118] It should be noted that the corresponding relationship between the objects and their position changes and the home tasks is not limited to that shown in Table 1, and can be specifically set according to actual needs.

[0119] Step 210: Send the target home task to one or more embodied robots and / or fixed home robots associated with the target object, so that the one or more embodied robots and / or fixed home robots perform the target home task.

[0120] In this embodiment, the control device can send the target home task to one or more embodied robots and / or fixed home robots associated with the target object through the network. One or more embodied robots and / or fixed home robots receive and execute the target home task, thereby autonomously controlling the corresponding embodied robots to execute the corresponding target home tasks according to the changes in the position of the target object, thereby improving the intelligent control level of the embodied robots.

[0121] In some embodiments, the target object may be a water cup, the target household task may be a cleaning task, the second posture is the water cup falling to the ground, the third posture is the water cup placed on the table, and the embodied robot may include a humanoid robot and a cleaning robot associated with the water cup.

[0122] The control device can send cleaning tasks to the humanoid robot and the cleaning robot through the network. The humanoid robot receives and responds to the cleaning tasks, picks up the water cup from the ground and puts it back on the table. The cleaning robot receives and responds to the cleaning tasks, and mops the ground where the water cup falls.

[0123] In some embodiments, the target problem may be a trash can, the target home task may be a homing task, the second posture is that the trash can is placed in the corner of the coffee table, the third posture is that the trash can is placed in the corner of the living room, and the embodied robot may include a humanoid robot associated with the trash can.

[0124] The control device can send a homing task to the humanoid robot through the network. The humanoid robot receives and responds to the homing task and moves the trash can from the corner of the coffee table to the corner of the living room.

[0125] The solution provided in this embodiment autonomously controls the corresponding embodied robot to perform the corresponding target household tasks according to the changes in the position and posture of the target object, thereby improving the intelligent control level of the embodied robot.

[0126] See also Figure 4 , which shows a flowchart of a map construction method of an embodied robot control system provided by another embodiment of the present application. In a specific embodiment, the map construction method of the embodied robot control system can be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. Figure 4 The process shown is described in detail, and the map construction method of the embodied robot control system may include the following steps 301 to 310.

[0127] Step 301: Acquire 3D point cloud data and a first environment image collected when the embodied robot moves in a home environment.

[0128] Step 302: Fuse the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map.

[0129] Step 303: Acquire second environment data collected by a camera in the home environment.

[0130] Step 304: Match the second environment image with the first environment image to obtain the first position of the camera in the initial three-dimensional home map.

[0131] Step 305: Determine a target three-dimensional home map according to the first posture, the second environment image and the initial three-dimensional home map.

[0132] Step 306: Obtain the second position of the target object in the target three-dimensional home map at the current moment.

[0133] Step 307: Determine whether the posture of the target object has changed according to the third posture and the second posture of the target object in the target three-dimensional home map at the previous moment.

[0134] Step 308: When it is determined that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map according to time dynamics to obtain the target four-dimensional home map.

[0135] In this embodiment, steps 301 , 302 , 303 , 304 , 305 , 306 , 307 and 308 may refer to the contents of the corresponding steps in the aforementioned embodiments, and will not be described in detail here.

[0136] Step 309: Generate a current moving trajectory of the target object according to the second posture and the third posture.

[0137] In this embodiment, when the control device determines that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map according to time dynamics. After obtaining the four-dimensional home map, the current movement trajectory of the target object can be generated according to the second posture and the third posture.

[0138] Among them, the current movement trajectory may include a current translation trajectory and / or a current rotation trajectory. The current translation trajectory may be used to characterize the path of the target object from the third position to the second position, and the current rotation trajectory may be used to characterize the angle change of the target object from the third orientation to the second orientation.

[0139] Step 310: Generate a target movement trajectory according to the historical movement trajectory and current movement trajectory of the target object.

[0140] In this embodiment, the control device can add the current movement trajectory of the target object to the historical movement trajectory to update the latest movement trajectory of the target object, obtain the target movement trajectory, and save the target movement trajectory in the target four-dimensional home map, in which the target movement trajectory of the target object can be displayed.

[0141] The historical movement trajectory may include a historical translation trajectory and / or a historical rotation trajectory. The historical translation trajectory may be used to characterize the historical movement path corresponding to the target object, and the historical rotation trajectory may be used to characterize the historical angle change corresponding to the target object.

[0142] The solution provided in this embodiment is based on the real-time update of the historical posture changes of the target object based on the current posture changes, which is conducive to improving the control accuracy of the embodied robot, and the user can find the target object according to the target movement trajectory of the target object in the four-dimensional map of the target home. For example, the user can find a mobile phone according to the movement trajectory of the mobile phone, which improves convenience.

[0143] See also Figure 5 , which shows a map construction device 500 of an embodied robot control system provided by an embodiment of the present application. In a specific embodiment, the map construction device 500 of the embodied robot control system can be applied to Figure 1 The control device 400 in the embodied robot control system shown in FIG. Figure 5 The map construction device 500 of the embodied robot control system shown is described in detail. The map construction device 500 of the embodied robot control system may include a first acquisition module 501, a fusion module 502, a second acquisition module 503, a matching module 504, a first determination module 505, a third acquisition module 506, a second determination module 507 and a recording module 508.

[0144] The first acquisition module 501 can be used to acquire 3D point cloud data and a first environment image collected when the embodied robot moves in the home environment; the fusion module 502 can be used to fuse the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map; the second acquisition module 503 can be used to acquire the second environment data collected by the camera in the home environment, and the second environment data can at least include the second environment image; the matching module 504 can be used to match the second environment image with the first environment image to obtain the first position of the camera in the initial three-dimensional home map; the first determination module 505 can be used to determine the target three-dimensional home map according to the first position, the second environment image and the initial three-dimensional home map; the third acquisition module 506 can be used to acquire the second position of the target object in the target three-dimensional home map at the current moment; the second determination module 507 can be used to determine whether the posture of the target object has changed according to the third position and the second position of the target object in the target three-dimensional home map at the previous moment; the recording module 508 can be used to record the position change of the target object from the third position to the second position in the target three-dimensional home map according to time dynamics when it is determined that the posture of the target object has changed, so as to obtain the target four-dimensional home map.

[0145] In some implementations, the first determining module 505 may include a first determining unit and a second determining unit.

[0146] The first determination unit can be used to determine the posture offset of each object in the second environment image relative to the camera; the second determination unit can be used to determine the target three-dimensional home map based on the posture offset, the first posture and the initial three-dimensional home map.

[0147] In some implementations, the first determining unit may include an acquiring subunit and a first determining subunit.

[0148] The acquisition subunit can be used to acquire the depth point cloud data of each object; the first determination subunit can be used to determine the posture offset according to the depth point cloud data.

[0149] In some embodiments, the first determination sub-unit may include a construction sub-sub-unit, a first determination sub-sub-unit, an alignment sub-sub-unit, a translation sub-sub-unit, and a second determination sub-sub-unit.

[0150] The construction sub-subunit can be used to construct the object coordinate system corresponding to each object based on the geometric features of each object; the first determination sub-subunit can be used to determine the origin of the object coordinate system; the alignment sub-subunit can be used to align the object coordinate system and the camera coordinate system of the camera according to the depth point cloud data, and obtain the rotation matrix from the object coordinate system to the camera coordinate system; the translation sub-subunit can be used to obtain the translation vector according to the position from the origin of the object coordinate system to the camera coordinate system; the second determination sub-subunit can be used to determine the posture offset according to the rotation matrix and the translation vector.

[0151] In some implementations, the second environment data may further include an object type of each object, and the second determination unit may include a second determination subunit and a labeling subunit.

[0152] The second determination subunit can be used to determine the fourth posture of each object in the initial three-dimensional home map according to the posture offset and the first posture; the labeling subunit can be used to semantically label each object in the initial three-dimensional home map according to the fourth posture and object type to obtain the target three-dimensional home map.

[0153] In some implementations, the map building device 500 of the embodied robot control system may further include a third determination module and a sending module.

[0154] The third determination module can be used to record the posture change of the target object from the third posture to the second posture in the target three-dimensional home map according to the time dynamics when it is determined that the posture of the target object has changed. After obtaining the target four-dimensional home map, the target home task is determined according to the second posture and the third posture; the sending module can be used to send the target home task to one or more embodied robots associated with the target object, so that one or more embodied robots perform the target home task.

[0155] In some implementations, the map building device 500 of the embodied robot control system may further include a generation module and an update module.

[0156] The generation module can be used to, when it is determined that the posture of the target object has changed, record the posture change of the target object from the third posture to the second posture in the target three-dimensional home map according to the time dynamics, and after obtaining the target four-dimensional home map, generate the current movement trajectory of the target object according to the second posture and the third posture; the update module can be used to generate the target movement trajectory according to the historical movement trajectory and current movement trajectory of the target object.

[0157] In some embodiments, the camera may include multiple sub-cameras, and the multiple sub-cameras may be installed at different locations in a home environment. The second environment image may include multiple third environment images, and each third environment image may correspond to a sub-camera.

[0158] The matching module 504 may include a matching unit, and the first determining module 505 may include a third determining unit.

[0159] The matching unit can be used to match each third environment image with the first environment image to determine a first pose of each sub-camera in the initial three-dimensional map to obtain multiple first poses; the third determination unit can be used to determine the target three-dimensional home map based on multiple first poses, multiple third environment images and the initial three-dimensional home map.

[0160] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated one by one in the device embodiment.

[0161] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0162] See also Figure 6 , which shows a functional block diagram of an embodied robot 600 provided in one embodiment of the present application, the embodied robot 600 may include one or more of the following components: a memory 610, a processor 620, and one or more applications, wherein the one or more applications may be stored in the memory 610 and configured to be executed by the one or more processors 620, and the one or more applications are configured to execute the method described in the aforementioned method embodiment.

[0163] The memory 610 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). The memory 610 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 610 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as obtaining 3D point cloud data, obtaining a first environment image, obtaining a second environment data, fusing data and images, matching the second environment image with the first environment image, obtaining a first pose, determining a target three-dimensional home map, obtaining a second pose, determining whether a change has occurred, determining a change has occurred, recording a pose change, obtaining a target four-dimensional home map, determining a pose offset, obtaining depth point cloud data, constructing an object coordinate system, determining an origin, aligning an object coordinate system with a camera coordinate system of a camera, obtaining a translation vector, determining a fourth pose, semantic annotation, obtaining a target three-dimensional home map, generating a current moving trajectory, and generating a target moving trajectory, etc.), instructions for implementing the following various method embodiments, etc. The storage data area can also store data created by the embodied robot 600 during use (such as the embodied robot, home environment, 3D point cloud data, first environment image, camera, second environment data, second environment image, initial three-dimensional home map, first pose, target three-dimensional home map, current moment, target object, second pose, previous moment, third pose, pose change, target four-dimensional home map, pose offset, depth point cloud data, geometric features, object coordinate system, camera coordinate system, rotation matrix, offset vector, object type, fourth pose, current movement trajectory, historical movement trajectory and target movement trajectory), etc.

[0164] The processor 620 may include one or more processing cores. The processor 620 uses various interfaces and lines to connect the various parts of the entire embodied robot 600, and executes various functions and processes data of the embodied robot 600 by running or executing instructions, programs, code sets or instruction sets stored in the memory 610, and calling data stored in the memory 610. Optionally, the processor 620 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 620 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 620, but may be implemented separately through a communication chip.

[0165] Please refer to Figure 7 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 700 stores a program code 710, which can be called by a processor to execute the method described in the above method embodiment.

[0166] The computer-readable storage medium 700 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 700 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 700 has storage space for program code 710 for executing any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 710 can be compressed, for example, in an appropriate form.

[0167] Please refer to Figure 8, which shows a structural block diagram of a computer program product 800 provided in an embodiment of the present application. The computer program product 800 includes a computer program / instruction 810, and the computer program / instruction 810 is stored in a computer-readable storage medium of a computer device. When the computer program product 800 is run on a computer device, a processor of the computer device reads the computer program / instruction 810 from the computer-readable storage medium, and the processor executes the computer program / instruction 810, so that the computer device performs the method described in the above method embodiment.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A map construction method for an embodied robot control system, characterized in that: include: Acquire 3D point cloud data and a first environment image collected when the embodied robot moves in the home environment; fusing the 3D point cloud data and the first environment image to obtain an initial three-dimensional home map; Acquire second environment data collected by a camera in the home environment, where the second environment data at least includes a second environment image; Matching the second environment image with the first environment image to obtain the first position of the camera in the initial three-dimensional home map; Determine a target three-dimensional home map according to the first posture, the second environment image and the initial three-dimensional home map; Obtaining a second position of the target object in the target three-dimensional home map at the current moment; Determining whether the posture of the target object has changed according to the third posture of the target object in the target three-dimensional home map at a previous moment and the second posture; When it is determined that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map dynamically according to time to obtain a target four-dimensional home map.

2. The map construction method according to claim 1, characterized in that: The step of determining a target three-dimensional home map according to the first posture, the second environment image, and the initial three-dimensional home map includes: Determining a position offset of each object in the second environment image relative to the camera; The target three-dimensional home map is determined according to the posture offset, the first posture and the initial three-dimensional home map.

3. The map construction method according to claim 2, characterized in that: The determining of the position offset of each object in the second environment image relative to the camera includes: Acquire the depth point cloud data of each object; The posture offset is determined according to the depth point cloud data.

4. The map construction method according to claim 3, characterized in that: Determining the posture offset according to the depth point cloud data includes: Constructing an object coordinate system corresponding to each object based on the geometric features of each object; Determining the origin of the object coordinate system; According to the depth point cloud data, align the object coordinate system with the camera coordinate system of the camera to obtain a rotation matrix from the object coordinate system to the camera coordinate system; Obtaining a translation vector according to the position from the origin of the object coordinate system to the camera coordinate system; The posture offset is determined according to the rotation matrix and the translation vector.

5. The map construction method according to claim 2, characterized in that: The second environment data further includes the object type of each object, and determining the target three-dimensional home map according to the posture offset, the first posture and the initial three-dimensional home map includes: Determining a fourth posture of each object in the initial three-dimensional home map according to the posture offset and the first posture; According to the fourth posture and the object type, semantic annotation is performed on each object in the initial three-dimensional home map to obtain the target three-dimensional home map.

6. The map construction method according to any one of claims 1 to 5, characterized in that: When it is determined that the posture of the target object has changed, the posture change of the target object from the third posture to the second posture is recorded in the target three-dimensional home map dynamically according to time, and after the target four-dimensional home map is obtained, the map construction method further includes: Generate a current moving trajectory of the target object according to the second posture and the third posture; A target movement trajectory is generated according to the historical movement trajectory of the target object and the current movement trajectory.

7. The map construction method according to any one of claims 1 to 5, characterized in that: The camera includes a plurality of sub-cameras, and the plurality of sub-cameras are installed at different positions in the home environment. The second environment image includes a plurality of third environment images, and each third environment image corresponds to a sub-camera. The second environment image is matched with the first environment image to obtain the first position of the camera in the initial three-dimensional home map, including: Matching each third environment image with the first environment image, determining a first position of each sub-camera in the initial three-dimensional map, to obtain multiple first positions; The step of determining a target three-dimensional home map according to the first posture, the second environment image, and the initial three-dimensional home map includes: The target three-dimensional home map is determined according to the multiple first poses, the multiple third environment images and the initial three-dimensional home map.

8. An embodied robot, characterized in that: include: Memory; One or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the map construction method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the map construction method according to any one of claims 1 to 7.

Images