Image acquisition terminal-based mapping method and system and cleaning equipment with same

Abstract

The invention relates to a mapping method and system based on an image acquisition terminal and cleaning equipment with the mapping system. The method comprises the steps of obtaining first collection information in the moving process of the cleaning equipment; the first collection information refers to information collected by a sensor arranged in the cleaning equipment when the cleaning equipment moves; receiving second acquisition information sent by the image acquisition terminal; the second acquisition information refers to an environment image of the cleaning equipment acquired by the image acquisition terminal when the cleaning equipment moves; the image acquisition terminal is independent of the cleaning equipment; and according to the first collection information and the second collection information, generating a corresponding environment map when the cleaning equipment moves. When the cleaning equipment maps, the external image acquisition terminal is combined, the existing hardware of the cleaning equipment does not need to be changed, high-precision positioning of the cleaning equipment can be realized, and the mapping precision of the cleaning equipment is effectively improved.

Description

[0001] This application is application number "202211126600.3", application date "September 16, 2022", and invention name "

[0002] The divisional application is titled "Mapping method, system and cleaning equipment based on image acquisition terminal". Technical Field

[0003] This invention relates to the field of mapping of cleaning equipment, and particularly to a mapping method, system, and cleaning equipment having the same based on an image acquisition terminal. Background Technology

[0004] With the development of science and technology, intelligent cleaning equipment such as robot vacuum cleaners are gradually entering people's lives.

[0005] During the operation of intelligent cleaning devices such as robotic vacuum cleaners, real-time localization and mapping are often required. Currently, existing intelligent cleaning devices typically rely on their own sensors for real-time localization and mapping, primarily using single-line LiDAR, low-resolution cameras, low-resolution distance sensors, wheel speed sensors, and gyroscopes. However, the hardware configuration of these intelligent cleaning devices is gradually lagging behind current mainstream software algorithms, becoming a major bottleneck for high-precision mapping, and consequently preventing existing intelligent cleaning devices from achieving high-precision localization and mapping. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is how to achieve high-precision mapping of cleaning equipment such as sweepers.

[0007] To address the aforementioned technical problems, this invention provides a mapping method based on an image acquisition terminal, comprising:

[0008] Acquire first collected information during the movement of the cleaning equipment; the first collected information refers to the information collected by the built-in sensors of the cleaning equipment when the cleaning equipment moves;

[0009] The system receives second acquisition information sent by an image acquisition terminal; the second acquisition information refers to the environmental image of the cleaning equipment acquired by the image acquisition terminal when the cleaning equipment moves; the image acquisition terminal is independent of the cleaning equipment.

[0010] An environmental map corresponding to the movement of the cleaning equipment is generated based on the first and second collected information.

[0011] Optionally, the image acquisition terminal moves synchronously with the cleaning equipment.

[0012] Optionally, the image acquisition terminal is fixed to the cleaning device at least at a preset fixed position and fixed direction when acquiring the second acquisition information.

[0013] Optionally, the first collected information includes at least one of the following: inertial data of the cleaning equipment measured by the inertial measurement unit built into the cleaning equipment, wheel speed data of the cleaning equipment measured by the wheel speed meter built into the cleaning equipment, and laser point cloud data collected by the lidar built into the cleaning equipment.

[0014] Optionally, generating an environmental map corresponding to the movement of the cleaning equipment based on the first collected information and the second collected information includes:

[0015] The relative positional relationship between the image acquisition terminal and the built-in sensors of the cleaning equipment is obtained;

[0016] Based on the relative positional relationship, the first collected information and the second collected information are fused to obtain the environmental map corresponding to the movement of the cleaning equipment.

[0017] Optionally, before fusing the first collected information and the second collected information according to the relative positional relationship to obtain the environmental map corresponding to the movement of the mobile device, the method further includes:

[0018] The first and second collected information are synchronized in time.

[0019] Optionally, the time synchronization processing of the first collected information and the second collected information includes:

[0020] A first timestamp is added to the first collected information, and a second timestamp is added to the second collected information;

[0021] Based on the first timestamp and the second timestamp, the first collected information or the second collected information is adjusted so that the first collected information and the second collected information are synchronized in time.

[0022] Furthermore, this invention also proposes a mapping system based on an image acquisition terminal, comprising:

[0023] The first data acquisition module is communicatively connected to the built-in sensors of the cleaning equipment and is used to acquire first collected information during the movement of the cleaning equipment; the first collected information refers to the information collected by the built-in sensors of the cleaning equipment when the cleaning equipment is moving.

[0024] The second data acquisition module is communicatively connected to the image acquisition terminal and is used to receive second acquisition information sent by the image acquisition terminal; the second acquisition information refers to the environmental image of the cleaning equipment acquired by the image acquisition terminal when the cleaning equipment moves; and

[0025] The mapping module is communicatively connected to both the first data acquisition module and the second data acquisition module, and is used to generate an environmental map corresponding to the movement of the cleaning equipment based on the first collected information and the second collected information.

[0026] Furthermore, the present invention also proposes a cleaning device, comprising:

[0027] Equipment body;

[0028] A sensor, mounted on the device body, is used to collect first-level information as the cleaning device moves; and

[0029] The aforementioned mapping system based on the image acquisition terminal is mounted on the device body and is communicatively connected to both the image acquisition terminal and the sensor.

[0030] Optionally, the cleaning equipment includes at least one of the following: a vacuum cleaner, a floor scrubber, or a robotic vacuum cleaner.

[0031] The technical solution provided by this invention has the following advantages:

[0032] The present invention provides a mapping method, system, and cleaning equipment based on an image acquisition terminal. Utilizing data interaction between the image acquisition terminal and the cleaning equipment, the image acquisition terminal serves as a second platform for mapping via external sensors of the cleaning equipment. During mapping, the external image acquisition terminal is used in conjunction with the cleaning equipment to acquire second-stage information—environmental images—during the cleaning equipment's movement. This approach reduces costs while maintaining mapping accuracy. First-stage information acquired during the cleaning equipment's movement is obtained from its internal sensors and fused with the environmental images to generate an environmental map. Compared to traditional mapping methods, this invention incorporates information from the cleaning equipment's internal sensors, improving the robustness of the cleaning equipment's positioning. Furthermore, the high resolution of the environmental images acquired by the image acquisition terminal results in a much higher information density of environmental features in the second-stage information acquired compared to current cleaning equipment. Therefore, the fusion of the environmental images acquired by the image acquisition terminal and the first-stage information acquired by the internal sensors achieves high-precision positioning of the cleaning equipment without requiring modifications to its existing hardware, thereby effectively improving the mapping accuracy of the cleaning equipment. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart of a mapping method based on an image acquisition terminal according to Embodiment 1 of the present invention;

[0035] Figure 2 This is a flowchart of generating an environment map according to Embodiment 1 of the present invention;

[0036] Figure 3 This is a structural diagram of a mapping system based on an image acquisition terminal according to Embodiment 2 of the present invention;

[0037] Figure 4 This is a structural diagram of a cleaning device according to Embodiment 3 of the present invention. Detailed Implementation

[0038] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0040] In this invention, unless otherwise stated, directional terms such as "upper," "lower," "top," and "bottom" are generally used in relation to the direction shown in the accompanying drawings, or in relation to the vertical, perpendicular, or gravitational direction of the component itself; similarly, for ease of understanding and description, "inner" and "outer" refer to the inner and outer contours of each component itself, but the above directional terms are not intended to limit this invention.

[0041] In traditional technologies, existing intelligent cleaning devices typically rely on their onboard sensors for real-time localization and map building. However, the hardware of these devices is increasingly lagging behind mainstream software algorithms, becoming a major bottleneck for high-precision mapping and preventing them from achieving high-accuracy localization and mapping. To address these technical problems, this invention proposes a mapping method, system, and cleaning device based on an image acquisition terminal.

[0042] The mapping method, system, and cleaning equipment based on an image acquisition terminal proposed in this invention can be applied to any cleaning equipment that requires positioning and mapping, such as a sweeper. In the following embodiments, this invention is illustrated using a sweeper as an example.

[0043] In one scenario example provided in this specification, the image acquisition terminal can be a mobile phone. For example, the mobile phone can be paired with a robot vacuum cleaner to establish bidirectional communication data transmission.

[0044] You can also set a fixed bracket on the robot vacuum cleaner to fix the position and orientation of the phone. In this way, the calibration between the phone and the robot vacuum cleaner can be completed in one go, without having to calibrate the phone and the robot vacuum cleaner every time the map is built, thus reducing the complexity of using the phone and robot vacuum cleaner to build a map together.

[0045] For example, the fixed position and direction of the fixed bracket can be preset. Correspondingly, sweepers with fixed brackets of the same fixed position and direction can be considered as sweepers of the same calibration type. Calibration can be performed on any one or more sweepers of this calibration type, obtaining calibration information between the sweeper and the mobile phone. The calibration method between the mobile phone and the sweeper is not limited in this manual. Correspondingly, calibration information of the calibration type to which the sweeper belongs can be pre-stored in the sweeper. During map construction, the mobile phone can be fixed on the fixed bracket, and the mobile phone can extract the calibration information between the sweeper and the mobile phone from the sweeper to use for map construction.

[0046] The calibration information between the robot vacuum and the mobile phone can be the coordinate system transformation relationship between the mobile phone's coordinate system and the robot vacuum's coordinate system, or the coordinate system transformation relationship between the image acquisition coordinate system of the image acquisition element in the mobile phone and the robot vacuum's coordinate system.

[0047] Given the differences in phone size and the location of image acquisition components within the phone, even with a fixed support, the calibration information between the phone and the robot vacuum cleaner may vary significantly. Therefore, calibration can be performed separately for the phone and the robot vacuum cleaner based on their respective phone models and calibration types.

[0048] Before map building, after obtaining the calibration information between the robot vacuum and the mobile phone, the calibration information can be further updated using real-time collected data to further ensure the accuracy of map building.

[0049] After the mobile phone is fixed on the bracket, it can move synchronously with the robot vacuum. For ease of description, the information collected by the robot vacuum can be described as the first collected information, and the information collected by the mobile phone can be described as the second collected information. In this scenario example, the information collected by the robot vacuum can be determined based on the type of sensors built into the robot vacuum. Accordingly, the first collected information may include inertial data measured by the inertial measurement unit built into the robot vacuum, wheel speed data measured by the wheel speed sensor built into the robot vacuum, and laser point cloud data collected by the lidar built into the robot vacuum, etc. The second collected information is the environmental image of the cleaning equipment captured by the image acquisition element in the mobile phone.

[0050] Given the calibration information of both the mobile phone and the robot vacuum cleaner, the coordinate system transformation relationship between the image acquisition element in the mobile phone and the various internal sensors in the robot vacuum cleaner can also be determined. The robot vacuum cleaner can transmit the first acquired information to the mobile phone based on the established two-way communication. The mobile phone can then combine the first and second acquired information based on the corresponding coordinate system transformation relationship to generate an environmental map corresponding to the robot vacuum cleaner's movement.

[0051] When the mobile phone combines the first and second collected information, it can also use the timestamps of the first and second collected information to synchronize the time of the first and second collected information, thereby realizing the synchronous fusion processing between the information collected by the mobile phone and the robot vacuum cleaner, and ensuring the accuracy of map construction during the robot vacuum cleaner's movement.

[0052] The mobile phone can also generate the real-time pose of the robot vacuum cleaner based on the first and second collected information during its movement, and send the pose back to the robot vacuum cleaner so that it can plan a path based on the returned pose information. The robot vacuum cleaner will then move according to the planned path until the map construction is complete. Alternatively, the mobile phone can also plan a path based on the robot vacuum cleaner's real-time pose and feed the path planning result back to the robot vacuum cleaner so that it can move according to the planned path until the map construction is complete.

[0053] The mobile phone can optimize the constructed map and store the final map information. The robot vacuum cleaner can send the sparse map back to the robot vacuum cleaner in the form of 3D coordinates and descriptors, so that the robot vacuum cleaner can perform cleaning work based on the map.

[0054] Once the mapping is complete, the user can remove their phone from the robot vacuum and disconnect the two-way communication link between the phone and the robot vacuum.

[0055] Mobile phones have relatively high image acquisition accuracy and algorithm processing capabilities. By linking mobile phones with robotic vacuum cleaners to build high-precision maps, it is possible to achieve this. Especially for non-vision-driven robotic vacuum cleaners, no hardware modifications are required. Based on the method provided in this scenario example, high-precision map construction can be achieved, thereby significantly reducing the hardware cost of high-precision robotic vacuum cleaners.

[0056] Of course, the above scenario examples are merely preferred examples and do not constitute a direct limitation on the solutions provided in the embodiments of this specification. The image acquisition terminal can also be other intelligent devices. Alternatively, the image acquisition terminal can be a component that only performs image acquisition functions, and correspondingly, the image processing procedure based on the first and second acquisition information can also be performed by a robot vacuum cleaner.

[0057] Example 1

[0058] like Figure 1 As shown, this embodiment provides a mapping method based on an image acquisition terminal, the method including:

[0059] S1: Acquire the first data during the movement of the cleaning equipment; the first data refers to the information collected by the built-in sensors of the cleaning equipment when the cleaning equipment is moved;

[0060] S2: Receive the second acquisition information sent by the image acquisition terminal; the second acquisition information refers to the environmental image of the cleaning equipment acquired by the image acquisition terminal when the cleaning equipment moves; the image acquisition terminal is independent of the cleaning equipment;

[0061] S3: Generate an environmental map corresponding to the movement of the cleaning equipment based on the first and second collected information.

[0062] In this embodiment, data interaction between an image acquisition terminal and the cleaning equipment is utilized. The image acquisition terminal serves as a second platform for mapping using external sensors of the cleaning equipment. During mapping, the external image acquisition terminal is used in conjunction with the cleaning equipment to acquire second acquisition information—i.e., environmental images—during the movement of the cleaning equipment. This approach reduces costs while ensuring the mapping accuracy of the cleaning equipment. First acquisition information acquired during the movement of the cleaning equipment is obtained using internal sensors within the cleaning equipment and fused with the environmental images to generate an environmental map. Compared to traditional mapping methods, this approach incorporates information acquired by the internal sensors of the cleaning equipment, improving the robustness of the cleaning equipment's positioning. Furthermore, due to the high resolution of the environmental images acquired by the image acquisition terminal, the information density of environmental features in the second acquisition information is significantly higher than that of current cleaning equipment. Therefore, based on the fusion of the environmental images acquired by the image acquisition terminal and the first acquisition information acquired by the internal sensors, high-precision positioning of the cleaning equipment can be achieved without modifying its existing hardware, thereby effectively improving the mapping accuracy of the cleaning equipment.

[0063] Preferably, the image acquisition terminal moves synchronously with the cleaning equipment.

[0064] Synchronizing the image acquisition terminal with the cleaning equipment ensures a more accurate fusion of the first and second acquisition information collected by the two devices, thereby improving the accuracy of the generated environmental map.

[0065] For example, in one specific embodiment, the image acquisition terminal may be fixed at a designated location on the cleaning device at least when acquiring the second acquisition information.

[0066] By fixing the image acquisition terminal at a designated location on the cleaning equipment, the synchronous movement between the image acquisition terminal and the cleaning equipment can be better ensured, which facilitates the subsequent fusion of the first and second acquisition information and the generation of an environmental map when the cleaning equipment moves.

[0067] In the above specific embodiments, the image acquisition terminal is a mobile phone or tablet computer equipped with a camera, and the cleaning device is a robot vacuum cleaner. The mobile phone or tablet computer is placed at a designated position on the robot vacuum cleaner, and the mobile phone or tablet computer is paired with the robot vacuum cleaner through a Bluetooth network to establish a two-way data transmission channel between the two.

[0068] Of course, the image acquisition terminal can also move synchronously with the cleaning equipment in other ways, which are not limited here.

[0069] Preferably, the image acquisition terminal is fixed to the cleaning device based on a preset fixed position and fixed direction, at least when acquiring the second acquisition information. For example, the fixed position and fixed direction can be predetermined, and the user can install a fixing bracket on the cleaning device based on the fixed position and fixing method. The image acquisition terminal can be fixed on the fixing bracket to achieve the fixation of the image acquisition terminal. Of course, the image acquisition terminal can also be fixed in other ways so that the fixed position and fixed direction of the image acquisition terminal meet the preset fixed position and fixed direction, which is not limited here.

[0070] Correspondingly, the calibration information between the current cleaning device and the image acquisition terminal can be obtained. This calibration information can be constructed for other cleaning devices with the same fixed position and direction as the current cleaning device. Before map construction, the image acquisition terminal and the cleaning device can update the acquired calibration information based on the real-time acquired information, so as to locate the robot vacuum's pose and construct the map based on the updated calibration information.

[0071] This embodiment allows for the calibration of the image acquisition terminal and the cleaning equipment in one step, eliminating the need to calibrate them each time a map is built, thus reducing the complexity of using the image acquisition terminal and the cleaning equipment together for map construction.

[0072] When the image acquisition terminal is a smart terminal such as a mobile phone or tablet, the calibration information may include the coordinate system transformation relationship between the terminal coordinate system where the image acquisition terminal is located and the cleaning equipment coordinate system, or the coordinate system transformation relationship between the image acquisition coordinate system of the image acquisition element in the image acquisition terminal and the cleaning equipment coordinate system. When the image acquisition terminal is a component that only performs image acquisition, the calibration information may be the coordinate system transformation relationship between the terminal coordinate system where the image acquisition terminal is located and the cleaning equipment coordinate system. Of course, other forms of coordinate system transformation relationships are also possible, and this specification does not limit them.

[0073] Preferably, the first acquired information includes at least one of the following: inertial data of the cleaning equipment measured by the inertial measurement unit built into the cleaning equipment, wheel speed data of the cleaning equipment measured by the wheel speed meter built into the cleaning equipment, and laser point cloud data acquired by the lidar built into the cleaning equipment.

[0074] Since the first collected information is closely related to the environment in which the cleaning equipment is located, a high-precision environmental map can be constructed more accurately by combining the first collected information with the second collected information collected by the image acquisition terminal, without the need to modify the existing hardware.

[0075] Preferably, such as Figure 2 As shown, S3 includes:

[0076] S31: Obtain the relative positional relationship between the image acquisition terminal and the built-in sensors of the cleaning equipment;

[0077] S32: Based on the relative positional relationship, the first and second collected information are fused to obtain the environmental map corresponding to the movement of the cleaning equipment.

[0078] The relative positional relationship between the image acquisition terminal and the built-in sensors of the cleaning equipment can be considered a coordinate system transformation relationship between the image acquisition terminal and the built-in sensors. When the image acquisition terminal is a smart terminal such as a mobile phone or tablet, this relative positional relationship can be considered a coordinate system transformation relationship between the image acquisition coordinate system of the image acquisition element in the image acquisition terminal and the coordinate system of the built-in sensors in the cleaning equipment. This relative positional relationship can be determined based on the calibration information determined in the above-described embodiments and the coordinate transformation relationship between the built-in sensors of the cleaning equipment and the cleaning equipment itself.

[0079] Since the image acquisition terminal is independent of the cleaning equipment, there is a positional difference between the two during operation, which in turn leads to a positional difference between the first and second acquisition information collected by the two respectively. By obtaining the relative positional relationship between the image acquisition terminal and the sensors built into the cleaning equipment, the first and second acquisition information are fused based on the relative positional relationship, thereby improving the reliability and accuracy of the fusion of the first and second acquisition information and improving the mapping accuracy of the environmental map.

[0080] Preferably, before S32, the method further includes:

[0081] The first and second collected information are synchronized in time.

[0082] Since the image acquisition terminal is independent of the cleaning equipment, even if the image acquisition terminal is placed at the designated location of the cleaning equipment and the two move synchronously, the information collected by the two may not be synchronized in time. For example, the first acquisition information is collected at the first moment and the second acquisition information is collected at the second moment. Therefore, through the above-mentioned time synchronization processing, it can be ensured that the first acquisition information and the second acquisition information correspond to the same environment where the cleaning equipment is located at each point in time. This further ensures the reliability and accuracy of the subsequent fusion processing of the first acquisition information and the second acquisition information, and further improves the mapping accuracy of the environmental map.

[0083] Preferably, the first and second collected information are time-synchronized, including:

[0084] The first time stamp is added to the first piece of information collected, and the second time stamp is added to the second piece of information collected.

[0085] Based on the first and second timestamps, the first or second collected information is adjusted to synchronize the first and second collected information in time.

[0086] By using the above-mentioned method of timestamping and adjusting the first or second collected information using timestamps, the synchronization of the first and second collected information in time can be achieved in a simple and easy way without consuming a lot of resources or providing a lot of computing power.

[0087] Preferably, S32 includes:

[0088] S321: Extract environmental feature information from the second collected information;

[0089] S322: Based on the relative positional relationship, the environmental feature information and the first collected information are fused to generate a pose estimation model for the moving cleaning equipment;

[0090] S323: Solve the pose estimation model to obtain the target pose estimate when the cleaning equipment moves;

[0091] S324: Construct an environmental map of the cleaning equipment as it moves, based on the target pose estimation.

[0092] The second type of information acquired is the environmental image captured by the image acquisition terminal. This image contains environmental feature information during the movement of the cleaning equipment, such as object features, environmental boundary features, and environmental contour features identified in the image. This information facilitates the mapping of the cleaning equipment. Therefore, the environmental feature information is first extracted and then fused with the first type of information based on their relative positional relationships. This eliminates the positional differences between the first type of information and the environmental feature information, resulting in a more accurate pose estimation model for estimating the target pose of the cleaning equipment. By solving the model, a precise target pose estimate can be obtained, thereby constructing a high-precision environmental map.

[0093] In one specific implementation of this embodiment, before S321, the second acquired information is first processed by image processing, such as grayscale conversion and filtering, to facilitate accurate extraction of subsequent features; in S321, feature extraction methods such as SIFT (i.e., Scale-invariant feature transform) can be used to extract environmental feature information.

[0094] In one specific embodiment of this example, S322 includes:

[0095] S3221: Determine the first motion trajectory of the cleaning equipment based on the first collected information;

[0096] S3222: Determine the second motion trajectory of the cleaning equipment based on environmental characteristic information;

[0097] S3223: Based on the relative positional relationship, the second motion trajectory is adjusted to obtain the third motion trajectory;

[0098] S3224: Determine the fusion coefficient between the first motion trajectory and the third motion trajectory based on the similarity between the first motion trajectory and the third motion trajectory at each time point;

[0099] S3225: Based on the fusion coefficient, the first motion trajectory and the third motion trajectory are fused to generate a pose estimation model for the movement of the cleaning equipment.

[0100] Due to their relative positions, there is a difference between the first motion trajectory determined by the first acquisition information and the second motion trajectory determined by the environmental feature information. Since the first acquisition information is collected by built-in sensors, it is closer to the motion trajectory and target pose of the cleaning equipment. Therefore, this embodiment first determines the first and second motion trajectories respectively, and adjusts the second motion trajectory based on their relative positions to make the adjusted third motion trajectory closer to the actual trajectory of the cleaning equipment. Simultaneously, since the first acquisition information and the environmental feature information are acquired synchronously, the information involving the environmental map in both exhibits similarity at each time point; that is, the first and third motion trajectories are similar in time. This embodiment calculates the fusion coefficient between the two motion trajectories based on this similarity. Fusing the two motion trajectories using the fusion coefficient yields an optimized pose estimation model, which is beneficial for accurate target pose estimation.

[0101] The specific methods for determining the first and second motion trajectories in S3221 and S3222, and the adjustment of the second motion trajectory using relative positional relationships in S3223, all employ existing technologies; the fusion coefficient in S3224 includes a translation matrix, a rotation matrix, and a scaling factor, and the formula for calculating the similarity between the first and third motion trajectories is as follows:

[0102]

[0103] The formula for calculating the fusion coefficient based on similarity is as follows:

[0104]

[0105] Where T, R, and S are the translation matrix, rotation matrix, and scaling ratio, respectively; P1(t) and R1(t) are the position and orientation at time t in the first motion trajectory, respectively; P2(t) and R2(t) are the position and orientation at time t in the third motion trajectory, respectively; P1(t1) is the position at time t1 in the first motion trajectory, P1(t... n ) represents the earliest time point t in the first motion trajectory. n The position below, P2(t1) is the position below time point t1 in the third motion trajectory, P2(t n ) represents the earliest time point t in the third motion trajectory. n The position below, where n is the earliest time point t. n The total number of time points between time point t1 and time point t2 i The earliest time point t n At any time point between t1 and t2, P1(t i ) and R1(t i ) are respectively the t in the first motion trajectory i Position and orientation at time point, P2(t) i ) and R2(t i ) represent the t values ​​in the third motion trajectory. i Position and posture under time.

[0106] In S3225 above, the fusion coefficients calculated in S3224 are substituted into the similarity calculation formula to obtain the pose estimation model.

[0107] In one specific implementation of this embodiment, in S323, the target pose estimation at multiple moments during the movement of the cleaning equipment is obtained. In S324, a global 3D semantic mesh map can be constructed based on the target pose estimation at multiple moments using a 3D semantic mapping method, which is the final environment map. The specific implementation steps of the 3D semantic mapping method are not limited here.

[0108] Example 2

[0109] like Figure 3 As shown, this embodiment provides a mapping system based on an image acquisition terminal, the system comprising:

[0110] The first data acquisition module is communicatively connected to the built-in sensors of the cleaning equipment and is used to acquire first collected information during the movement of the cleaning equipment; the first collected information refers to the information collected by the built-in sensors of the cleaning equipment when the cleaning equipment is moving.

[0111] The second data acquisition module is communicatively connected to the image acquisition terminal and is used to receive the second acquisition information sent by the image acquisition terminal; the second acquisition information refers to the environmental images of the cleaning equipment acquired by the image acquisition terminal when the cleaning equipment moves; and

[0112] The mapping module is communicatively connected to both the first data acquisition module and the second data acquisition module, and is used to generate an environmental map corresponding to the movement of the cleaning equipment based on the first and second collected information.

[0113] In this embodiment, the mapping system based on the image acquisition terminal has a second data acquisition module that is communicatively connected to the image acquisition terminal. The image acquisition terminal serves as a second platform for mapping the cleaning equipment using external sensors. During the mapping process, the system collaborates with the external image acquisition terminal to acquire second-level information (environmental images) during the cleaning equipment's movement. This reduces costs while maintaining mapping accuracy. The first data acquisition module is communicatively connected to the cleaning equipment's internal sensors. It uses these internal sensors to acquire first-level information during the cleaning equipment's movement and fuses it with the environmental images to generate an environmental map. Compared to traditional mapping methods, this system incorporates information acquired by the cleaning equipment's internal sensors, improving the robustness of the cleaning equipment's positioning. Furthermore, the high resolution of the environmental images acquired by the image acquisition terminal results in a much higher information density of environmental features in the second-level information acquired compared to current cleaning equipment. Therefore, by fusing the environmental images acquired by the image acquisition terminal with the first-level information acquired by the internal sensors, high-precision positioning of the cleaning equipment can be achieved without modifying its existing hardware, effectively improving the mapping accuracy.

[0114] The mapping system based on the image acquisition terminal described in this embodiment corresponds to the mapping method based on the image acquisition terminal described above. For details not covered in this embodiment, please refer to Embodiment 1 and... Figure 1 and Figure 2 The specific details will not be elaborated here.

[0115] Example 3

[0116] like Figure 4 As shown, this embodiment provides a cleaning device, including:

[0117] Equipment body;

[0118] Sensors, mounted on the device itself, are used to collect initial information as the cleaning equipment moves; and

[0119] The mapping system based on the image acquisition terminal as described in claim 8 is mounted on the device body and is communicatively connected to both the image acquisition terminal and the sensor.

[0120] In this embodiment, the cleaning equipment uses an image acquisition terminal as a second platform for mapping via its external sensors. Based on the fusion of the environmental image acquired by the image acquisition terminal and the first acquisition information acquired by the internal sensors, high-precision positioning of the cleaning equipment can be achieved without modifying the existing hardware of the cleaning equipment, thereby effectively improving the mapping accuracy of the cleaning equipment.

[0121] The mapping system based on the image acquisition terminal described in this embodiment is the same as that in Embodiment 2. For details not covered in this embodiment, please refer to Embodiment 1, Embodiment 2, and... Figures 1 to 3 The specific details will not be elaborated here.

[0122] Obviously, the embodiments described above are merely some, not all, embodiments of the present invention. Based on the embodiments of the present invention, those skilled in the art can make other variations or modifications without creative effort, and all such variations or modifications should fall within the scope of protection of the present invention.

Claims

1. A mapping method based on an image acquisition terminal, characterized in that, The method comprises the following steps: acquiring first collection information in the movement of the cleaning device; the first collection information refers to information collected by the internal sensor of the cleaning device when the cleaning device moves; receiving second collection information sent by an image collection terminal; the second collection information refers to an environmental image of the cleaning device collected by the image collection terminal when the cleaning device moves; the image collection terminal is independent of the cleaning device; acquiring a relative position relationship between the image collection terminal and the internal sensor of the cleaning device; the relative position relationship is a coordinate system conversion relationship between the image collection terminal and the internal sensor, or a coordinate system conversion relationship between an image collection coordinate system of an image collection element in the image collection terminal and a coordinate system of the internal sensor; according to the relative position relationship, the first collection information and the second collection information are fused to obtain an environmental map corresponding to the movement of the cleaning device. 2.The image collection terminal based mapping method of claim 1, wherein, According to the relative position relationship, the first collection information and the second collection information are fused to obtain an environmental map corresponding to the movement of the cleaning device, comprising: extracting environmental feature information from the second collection information; according to the relative position relationship, the environmental feature information and the first collection information are fused to generate a pose estimation model of the movement of the cleaning device; solving the pose estimation model to obtain a target pose estimation of the movement of the cleaning device; according to the target pose estimation, the environmental map of the movement of the cleaning device is constructed. 3.The image collection terminal based mapping method of claim 1, wherein, Before the relative position relationship is used to fuse the first collection information and the second collection information to obtain an environmental map corresponding to the movement of the cleaning device, the method further comprises: performing time synchronization processing on the first collection information and the second collection information. 4.The image collection terminal based mapping method of claim 3, wherein, The time synchronization processing on the first collection information and the second collection information comprises: stamping a first time stamp on the first collection information and a second time stamp on the second collection information; based on the first time stamp and the second time stamp, adjusting the first collection information or the second collection information so that the first collection information and the second collection information are synchronized in time. 5.The image collection terminal based mapping method of claim 1, wherein, The image collection terminal and the cleaning device keep synchronous movement. 6.The image collection terminal based mapping method of claim 5, wherein, The image collection terminal is fixed on the cleaning device based on a preset fixed position and a fixed direction at least when collecting the second collection information. 7.The image collection terminal based mapping method of claim 1, wherein, The first collection information at least includes one of cleaning device inertia data measured based on an inertial measurement unit built in the cleaning device, cleaning device wheel speed data measured based on a wheel speed sensor built in the cleaning device, and laser point cloud data collected based on a laser radar built in the cleaning device.

8. An image acquisition terminal based mapping system, characterized by, The method comprises the following steps: a first data acquisition module in communication connection with an internal sensor of a cleaning device is used to acquire first collection information in the movement of the cleaning device; the first collection information refers to information collected by the internal sensor of the cleaning device when the cleaning device moves; The second data acquisition module is in communication connection with the image acquisition terminal, and is configured to receive second acquisition information sent by the image acquisition terminal. The second acquisition information refers to an environmental image of the cleaning device acquired by the image acquisition terminal when the cleaning device moves. The mapping module is in communication connection with the first data acquisition module and the second data acquisition module, and is configured to acquire a relative position relationship between the image acquisition terminal and an internal sensor of the cleaning device; and perform fusion processing on the first acquisition information and the second acquisition information according to the relative position relationship, to obtain an environmental map corresponding to the movement of the cleaning device; the relative position relationship is a coordinate system conversion relationship between the image acquisition terminal and the internal sensor, or a coordinate system conversion relationship between an image acquisition coordinate system of an image acquisition element in the image acquisition terminal and a coordinate system of the internal sensor. The cleaning device comprises at least one of the following: a dust collector, a floor washing machine, and a sweeping robot.

9. A cleaning apparatus, characterized by The cleaning device comprises at least one of the following: a dust collector, a floor washing machine, and a sweeping robot. ​ ​ ​ ​ 10. The cleaning apparatus of claim 9, wherein, ​

Images