Map processing method and apparatus, and self-moving device

Abstract

Embodiments of the present application provide a map processing method, device and self-moving equipment. The map processing method comprises: acquiring at least one of working state information of a current working process of the self-moving equipment in a working area and a real-time obstacle map, the working state information comprising at least one of positioning quality and working area coverage rate, the real-time obstacle map being obtained based on real-time images collected in a moving process; determining whether to update at least one of an original visual feature map and an original obstacle map used by the self-moving equipment according to at least one of the working state information and the real-time obstacle map, the original visual feature map being used for positioning of the self-moving equipment in the working area, and the original obstacle map being used for path planning of the self-moving equipment; if the update is needed, acquiring an image frame of the working area, and updating at least one of the original visual feature map and the original obstacle map using the image frame. The method can update the map in time.

Description

Technical Field

[0001] This invention relates to the field of smart device technology, and in particular to a map processing method, apparatus, and self-moving device. Background Technology

[0002] Existing self-propelled gardening equipment (also known as self-service gardening equipment, smart gardening equipment, or automatic gardening equipment, etc.), such as automatic lawnmowers, can automatically mow lawns without human supervision or control, thereby reducing the time spent on users and reducing repetitive labor.

[0003] To ensure that automated mobile devices can operate within the designated work area without venturing out of it and wasting time or damaging the device, or to prevent collisions with obstacles, a map of the work area is required. This can be achieved through visual mapping, such as using satellite positioning, lidar, cameras, wheel speedometers, and inertial measurement units (IMUs).

[0004] The problem with existing mapping is that the structure and environment of the work area may change over time, causing the map to be unable to accurately locate the mobile device, thus making the mobile device unable to work reliably. Summary of the Invention

[0005] In view of the above problems, embodiments of the present invention are proposed, which provide a map processing method and a self-moving device to at least solve the problem that existing maps are prone to not matching the environment of the actual working area.

[0006] One or more embodiments of the present invention provide a map processing method, comprising: acquiring at least one of working status information of a self-moving device in a working area during a current working process and a real-time obstacle map, wherein the working status information includes at least one of the following: positioning quality and working area coverage, wherein the positioning quality is determined based on at least one of the ratio of the number of times the self-moving device has been successfully positioned using an original visual feature map to the total number of times the self-moving device has been positioned during the current working process, and the ratio of the interior points matched when the self-moving device is successfully positioned using the original visual feature map to the total number of interior points, and the real-time obstacle map is obtained based on real-time images collected during the movement; determining whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device based on at least one of the working status information and the real-time obstacle map, wherein the original visual feature map is used for positioning of the self-moving device in the working area, and the original obstacle map is used for path planning of the self-moving device; if updating, acquiring image frames of the working area and using the image frames to update at least one of the original visual feature map and the original obstacle map.

[0007] According to another aspect of the present invention, a map processing apparatus is provided, characterized in that it comprises: an acquisition module, configured to acquire at least one of working status information of a self-moving device in a working area during a current working process and a real-time obstacle map, wherein the working status information includes at least one of the following: positioning quality and working area coverage, wherein the positioning quality is determined based on at least one of the ratio of the number of times the self-moving device has been successfully positioned using an original visual feature map to the total number of times the self-moving device has been positioned during the current working process, and the ratio of the interior points matched when successfully positioned using the original visual feature map to the total number of interior points, and the real-time obstacle map is obtained based on real-time images acquired during movement; a determination module, configured to determine whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device based on at least one of the working status information and the real-time obstacle map, wherein the original visual feature map is used for positioning of the self-moving device in the working area, and the original obstacle map is used for path planning of the self-moving device; and an update module, configured to, if an update is required, acquire image frames of the working area and update at least one of the original visual feature map and the original obstacle map using the image frames.

[0008] According to another aspect of the present invention, a self-moving device is provided, which includes a controller for performing the aforementioned map processing method.

[0009] This embodiment determines whether to update the original visual feature map or original obstacle map used for positioning and navigation based on the current working status information of the self-moving device and the real-time obstacle map. Since the working status information includes positioning quality and working area coverage, and can be combined with the real-time obstacle map to determine obstacle changes, the accuracy of the update decision is ensured. This guarantees timely updates to the original visual feature map or original obstacle map while avoiding the computational load and resource waste caused by frequent updates. When an update is determined to be needed, it is performed based on the acquired image frames of the working area, thereby ensuring the accuracy of the updated original visual feature map or original obstacle map. This ensures that the self-moving device can adapt to environmental changes in a timely manner and perform reliable positioning and navigation when the environment changes. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1AA flowchart illustrating the steps of the map processing method provided in Embodiment 1 of this application;

[0012] Figure 1B This is a schematic diagram of the walking coverage of the self-moving device in Embodiment 1 of this application;

[0013] Figure 2A A flowchart illustrating the steps of the map processing method provided in Embodiment 2 of this application;

[0014] Figure 2B This is a schematic diagram of the original obstacle map, the real-time obstacle map, and the obstacle changes in Embodiment 2 of this application;

[0015] Figure 3 This is a structural block diagram of the map processing device according to Embodiment 3 of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0017] To facilitate explanation and understanding, before describing the control method of the self-moving device, the structure and working scenario of the self-moving device are briefly described as follows:

[0018] In this embodiment, the self-moving device can be an automatic lawnmower. Of course, in other embodiments, the self-moving device can also be other self-service gardening equipment. Alternatively, the self-moving device can be any other device capable of self-propelled movement.

[0019] Automatic lawnmowers are used to trim lawns to ensure the grass reaches the required height. An automatic lawnmower includes a positioning component. This component may include one or more of the following: satellite positioning, ultra-wideband wireless communication positioning (UWB), a camera, an inertial measurement unit (IMU), and a wheel speed sensor. The positioning component detects the current pose of the automatic lawnmower. The pose includes its position and orientation. The position can be represented by its coordinates on the X, Y, and Z axes of the positioning coordinate system. The orientation can be determined by its angles relative to the X, Y, and Z axes.

[0020] Taking structure from motion (SFM) mapping via camera as an example, an automatic lawnmower moves within its working area, and the camera captures an image frame of its current location at regular intervals. After movement, based on the captured image frames, a trained first neural network model extracts features from the real-time image to obtain the visual features of that image frame. Simultaneously, based on the visual 3D reconstruction algorithm, the pose of the automatic lawnmower when the image frame was captured is calculated. Based on this pose and visual features, an original visual feature map is determined. In this way, a complete original visual feature map can be obtained after the automatic lawnmower has moved within the working area.

[0021] In the process of visual 3D reconstruction and mapping, other sensors (such as satellite positioning sensors, lidar, cameras, wheel speedometers, and IMUs) can be used to assist in positioning, thereby achieving mapping.

[0022] Similarly, for the acquired image frames, a trained second neural network model can be used to identify obstacles in the real-time images, thereby determining the passability status of each pixel in the image frame. For example, pixels corresponding to obstacles (such as trees, rocks, pits, people, animals, or impassable boundaries) are considered impassable, while pixels corresponding to lawns are considered passable. Based on the image's pose, the corresponding pixels in the original obstacle map are determined, and the passability status of the image's pixels is used as the passability status of their corresponding pixels in the original obstacle map. In this way, a complete original obstacle map can be obtained after the automatic lawnmower has moved within the working area.

[0023] After establishing the original visual feature map and the original obstacle map, the original visual feature map is used to locate the automatic lawnmower within the work area, and the original obstacle map is used to plan the path for the automatic lawnmower.

[0024] For example, when the automated lawnmower subsequently operates within the work area, a planned path is determined based on the original obstacle map. During its movement along the planned path, the camera captures image frames of its location, and a first neural network model extracts feature data from these image frames. Based on this feature data and the original visual feature map, the real-time pose of the automated lawnmower is determined. This real-time pose allows the automated lawnmower to be navigated, enabling it to move along the planned path.

[0025] To ensure that the original obstacle map and original visual feature map meet the usage requirements, map processing methods can be performed at appropriate times to update the map when needed.

[0026] The implementation process of the map processing method is explained below:

[0027] Example 1

[0028] Reference Figure 1A The diagram shows a flowchart of the map processing method according to Embodiment 1 of this application.

[0029] The method includes the following steps:

[0030] Step S102: Obtain at least one of the following: the working status information of the self-moving device in the current working process within the working area and the real-time obstacle map.

[0031] The operational status information includes at least one of the following: positioning quality and operational area coverage.

[0032] The positioning quality is determined by the ratio of the number of times the device is successfully positioned using the original visual feature map during the current operation to the total number of times the device is positioned.

[0033] During operation, the self-moving device performs visual positioning at regular intervals. This involves acquiring real-time images of the surrounding environment, matching the feature data from the real-time images with the raw visual feature data, and determining the current pose of the self-moving device based on the matched feature points. If this method fails to match the current pose, visual positioning fails, and the current pose of the self-moving device can be determined using methods such as inertial measurement units, wheel speedometers, or satellite positioning.

[0034] An automated mobile device may perform N localization operations during a complete working process. The ratio of the number of successful localizations using the original visual feature map to the total number of localizations (i.e., N) can be used as the successful localization frequency, which can be used as the localization quality.

[0035] Alternatively, the positioning quality can be determined by the ratio of matched inliers to the total number of inliers when successful positioning is achieved using the original visual feature map. For example, when successful positioning is achieved using the original visual feature map, the ratio of the number of matched inliers to the total number of inliers is used as the positioning quality.

[0036] Alternatively, the two methods described above can be combined to determine the positioning quality. For example, the number of matched inliers can be mapped to the range [0-1], and then the mapped value can be multiplied by the success frequency, with the result of the multiplication used as the positioning quality.

[0037] In one example, the positioning quality of the entire work area can be calculated directly, or the work area can be divided into multiple partitions, and the positioning quality of each partition can be calculated separately.

[0038] The work area coverage rate is the ratio between the area covered by the self-moving device within the work area during the current work process and the total area of ​​the work area. For example... Figure 1B As shown, it illustrates a schematic diagram of the self-moving device moving to cover a specific location within the work area during a single work process. Figure 1BThe gray area represents the location reached by the mobile device, the black area represents the accessible but unreachable location within the work area, and the white area represents the inaccessible location such as the boundary or obstacle.

[0039] The area corresponding to the gray area is the mobile coverage area, and its ratio to the total area of ​​the work area (i.e., all passable areas) can be used as the work area coverage rate. Work area coverage rate and positioning quality can reflect environmental changes in the work area to a certain extent. Therefore, based on the work area coverage rate and positioning quality, it can be determined whether the original visual feature map needs to be updated.

[0040] The real-time obstacle map in this current operation is obtained through visual localization based on real-time images acquired during the movement. For example, image recognition is performed on the real-time images to determine the passability of each pixel in the real-time images. Then, based on the real-time pose corresponding to the real-time images, each pixel is mapped onto the real-time obstacle map, thereby determining the passability of the pixels in the real-time obstacle map. The passability includes passable and impassable states.

[0041] Step S104: Based on at least one of the working status information and the real-time obstacle map, determine whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device.

[0042] As previously stated, the original visual feature map is used for the localization of the self-moving device within the work area. The original obstacle map is used for path planning of the self-moving device.

[0043] In one scenario, if the positioning quality of the working status information is lower than a quality threshold or the working area coverage is lower than a coverage threshold, then updating the original visual feature map is determined. It should be noted that the quality threshold and coverage threshold can be determined as needed. For example, if the positioning quality of the original visual feature data when it is first used is 0.8, then the quality threshold can be 0.4; if the positioning quality is 0.5, then the quality threshold can be 0.3. In other words, the quality threshold can be determined based on the positioning quality when the original visual feature data is first used.

[0044] The coverage threshold is similar to the quality threshold; it can also be determined based on the coverage rate of the original visual feature data and the first use of the real-time obstacle map, so it will not be elaborated further.

[0045] In another scenario, if the area of ​​obstacle change between the real-time obstacle map and the original obstacle map is greater than an area threshold, then the original obstacle map is updated.

[0046] Of course, in other embodiments, determining whether to update the original obstacle map or the original visual feature map can be done in other ways, and there are no limitations on this.

[0047] If it is determined that at least one of the original obstacle map and the original visual feature map should be updated, then step S106 is executed; otherwise, if it is determined that no update is needed, the process can be terminated.

[0048] Step S106: If an update is required, obtain an image frame of the working area and use the image frame to update at least one of the original visual feature map and the original obstacle map.

[0049] In one scenario, if it is determined that the original obstacle map needs updating, the self-moving device can be driven to move within the working area. After movement, a new original obstacle map can be built based on a visual 3D reconstruction algorithm. Similarly, if it is determined that the original visual feature map needs updating, a visual 3D reconstruction algorithm can also be used to build the map, so this will not be elaborated further.

[0050] In another scenario, if it is determined that the original obstacle map or the original visual feature map needs to be updated, a planned path can be planned based on the existing visual obstacle map, and the self-moving device can be controlled to move along the planned path. After the movement, a new original obstacle map or original visual feature data can be created using a visual 3D reconstruction algorithm.

[0051] Of course, other methods can also be used to update the original visual feature map or the original obstacle map, and there are no restrictions on this.

[0052] This embodiment determines whether to update the original visual feature map or original obstacle map used for positioning and navigation based on the current working status information of the self-moving device and the real-time obstacle map. Since the working status information includes positioning quality and working area coverage, and can be combined with the real-time obstacle map to determine obstacle changes, the accuracy of the update decision is ensured. This guarantees timely updates to the original visual feature map or original obstacle map while avoiding the computational load and resource waste caused by frequent updates. When an update is determined to be needed, it is performed based on the acquired image frames of the working area, thereby ensuring the accuracy of the updated original visual feature map or original obstacle map. This ensures that the self-moving device can adapt to environmental changes in a timely manner and perform reliable positioning and navigation when the environment changes.

[0053] Example 2

[0054] Reference Figure 2A The flowchart of the map processing method according to Embodiment 2 of this application is shown.

[0055] In this embodiment, the method includes the following steps:

[0056] Step S202: Obtain at least one of the following: the working status information of the self-moving device in the current working process within the working area and the real-time obstacle map.

[0057] The working status information includes at least one of the following: positioning quality and working area coverage. As described in the previous embodiment, the positioning quality is determined by the ratio of at least one of the following to the total number of positioning times of the self-moving device: the number of times the device successfully locates itself using the original visual feature map during the current working process, and the ratio of the number of interior points matched when successfully located using the original visual feature map to the total number of interior points.

[0058] For example, positioning quality can be obtained as follows: Divide the work area into multiple partitions. After each work process is completed, count the success frequency and matching quality of visual positioning for each partition, and calculate the positioning quality of that partition in the current work process. The success frequency can be the ratio of the number of successful visual positioning attempts for that partition to the total number of visual positioning attempts. The matching quality can be determined based on the number of matched interior points (also called feature points) in the real-time image feature data and the original feature data map during each matching attempt, or the ratio of matched interior points to the total number of interior points, thus ensuring the accuracy of the positioning quality.

[0059] The work area coverage rate is the ratio between the area covered by the self-moving device during the current work process and the total area of ​​the work area.

[0060] Real-time obstacle maps are obtained based on real-time images captured during movement, ensuring accurate obstacle identification. For example, this can be achieved through the following process:

[0061] Process A1: During the current work process, the self-moving device acquires real-time images of the work area and matches the feature data of the real-time images with the original visual feature map.

[0062] The acquired real-time images can be input into a first neural network model to obtain corresponding feature data. This feature data is then matched with the original visual feature map to obtain the matching results. The matching results indicate the matched feature points.

[0063] Process B1: Based on the matching results, determine the real-time pose of the self-moving device in the coordinate system of the original visual feature map.

[0064] Based on the positions of matched feature points in the image frames acquired during the creation of the original visual feature map, the pose of the mobile device at the time of image frame acquisition, and the relationship between feature points in the feature data and matched feature points, the real-time pose of the mobile device can be calculated. This achieves localization of the mobile device based on the original visual feature data.

[0065] Process C1: Based on the real-time pose and the real-time image, determine the real-time obstacle map obtained in the current working process.

[0066] In one example, process C1 can be implemented as follows: performing passable area identification on the real-time image to determine the passability status of pixels in the real-time image; mapping the real-time image to the coordinate system of the original obstacle map based on the real-time pose to determine the passability status of corresponding pixels in the original obstacle map coordinate system; and determining the real-time obstacle map based on the new passability status of each pixel in the original obstacle map coordinate system. This method can quickly determine the real-time obstacle map while ensuring its accuracy.

[0067] Accessible region identification can use a trained second neural network model to perform semantic segmentation on real-time images, that is, to determine the accessible and inaccessible regions in the real-time image. It can identify grass, pits, trees, rocks, people, animals, etc. in the real-time image. Pixels corresponding to grass, etc., are considered accessible, while pixels corresponding to obstacles or boundaries such as pits and trees are considered inaccessible.

[0068] Based on the real-time pose, pixels in the real-time image can be mapped to the coordinate system of the original obstacle map. The corresponding pixels in the coordinate system of the original obstacle map are determined, and the passage status of the pixels in the real-time image is used as the new passage status of the corresponding pixels in the coordinate system of the original obstacle map. In this way, the new obstacle map is the real-time obstacle map.

[0069] For example, if pixel 1 in a real-time image is in a passable state, and it is mapped to pixel a in the original obstacle map coordinate system, then the passable state of pixel a is determined to be passable.

[0070] Step S204: Based on at least one of the working status information and the real-time obstacle map, determine whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device.

[0071] In one feasible approach, determining whether to update the original obstacle map used by the self-moving device based on the real-time obstacle map can be achieved through the following sub-steps:

[0072] Sub-step S2041: Determine the obstacle change area of ​​the real-time obstacle map based on the original obstacle map and the real-time obstacle map.

[0073] The change in the area of ​​the obstacle can be achieved through the following process:

[0074] Process A2: Divide the original obstacle map and the real-time obstacle map into partitions to obtain at least one first partition corresponding to the original obstacle map and at least one second partition corresponding to the real-time obstacle map.

[0075] The partitioning method can be determined as needed. For example, the original obstacle map and the real-time obstacle map can be divided into multiple rectangular partitions. For ease of description, the partitions divided from the original obstacle map are called the first partition, and the partitions divided from the real-time obstacle map are called the second partition.

[0076] Alternatively, in other segmentation methods, the shape of the segmented partitions can be consistent with the shape mapped from the real-time images captured by the camera onto the original obstacle map. For example, if the mapping is a fan shape, then the partitions are segmented according to the fan shape.

[0077] Process B2: Based on the traffic status of pixels indicating the same location in the first and second partitions, determine the number of pixels in each of the second partitions that have different traffic statuses relative to the corresponding first partition.

[0078] For each second partition, determine the number of pixels whose accessibility differs from that of the first partition. For example, if second partition 1 and first partition 1 in the original obstacle map indicate the same location, the accessibility of each pixel in second partition 1 can be compared with the corresponding pixel in first partition 1 to determine which obstacle appeared first and then disappeared (e.g., obstacles that were present before being absent). Figure 2B As shown in the lower left corner of the image (i.e., the white part), and obstacles that appeared later (such as...). Figure 2B As shown in the lower right corner of the image (i.e., the white part), the number of pixels of obstacles that appeared first and then disappeared, plus the number of pixels of obstacles that appeared later and then disappeared, is the number of pixels with different passage states. Figure 2B Original obstacle map such as Figure 2B As shown in the top left corner of the image, the real-time obstacle map is as follows: Figure 2B As shown in the upper right corner of the image, Figure 2B The gray areas are passable areas, the white areas are impassable areas corresponding to obstacles, and the black areas are unknown areas.

[0079] In a specific example, pixel A in the second partition 1 corresponds to pixel A' in the first partition 1. If the passage status of pixel A' in the first partition 1 is passable (that is, there is no obstacle at this position), while the passage status of pixel A in the second partition 1 is impassable (that is, there is an obstacle at this position), then pixel A is determined to be a pixel with a different passage status.

[0080] Process C2: Determine the obstacle change area of ​​the real-time obstacle map based on the number of pixels showing changes in the passage status of each of the second partitions.

[0081] The number of pixels showing a change in traffic flow, multiplied by the actual area corresponding to each pixel, can be used as the obstacle change area of ​​the real-time obstacle map. If each pixel corresponds to the same actual area, then the number of pixels can be equated to the obstacle change area of ​​the real-time obstacle map.

[0082] Sub-step S2042: If the changed area of ​​the obstacle exceeds the area threshold, then determine to update the original obstacle map.

[0083] The area threshold can be determined as needed, based on experience, or based on the total area of ​​all obstacles within the work area, such as one-tenth of the total area.

[0084] If the area of ​​the obstacle changes exceeds the area threshold, it indicates that the obstacle may have been added or removed, or that the obstacle's location has moved. In this case, it is possible to update the original obstacle map, or to update both the original obstacle map and the original visual feature map, in order to better plan the path.

[0085] By dividing the working area into zones and setting area thresholds, misjudgments and false updates caused by minor changes in obstacle position due to pose errors can be effectively avoided. In other words, if the obstacle area change within a zone is small, possibly due to pose errors, this change can be excluded from the calculation of the changed area for that zone. This prevents multiple small changes in zones from accumulating into a large area change when there is actually no significant obstacle change. This improves accuracy.

[0086] Optionally, if the changed area of ​​the obstacle exceeds an area threshold and the positioning quality is greater than or equal to a quality threshold, then only the original obstacle map will be updated. This ensures timely and accurate updates without requiring frequent updates and calculations, reducing computational load and saving energy.

[0087] In another feasible approach, the real-time obstacle map can be sent to the user, who can then determine whether the original obstacle map needs to be updated. This better avoids the resource waste caused by frequent updates to the original obstacle map when temporary obstacles appear in the work area. To this end, determining whether to update the original obstacle map and the original visual feature map can be achieved through the following process:

[0088] Process A3: Generate an update request instruction based on the real-time obstacle map, and send the update request instruction to the smart terminal to instruct the smart terminal to display the real-time obstacle map and a request message for confirming whether to update at least one of the original visual feature map and the original obstacle map.

[0089] To facilitate user viewing, self-moving devices can connect to smart terminals (such as smartphones, personal computers, etc.) via wired or wireless networks and send update request commands to the smart terminals. These update request commands can carry information about the real-time obstacle map, allowing the smart terminals to display the real-time obstacle map and related request information. Request information may include explanatory text asking the user whether to update the original visual feature map or the original obstacle map, as well as control buttons.

[0090] By observing the real-time obstacle map, users can determine whether the changing obstacles in the work area are temporary obstacles. If so, they can choose not to update the original visual feature map and the original obstacle map. At this time, they can input an instruction not to update through the smart terminal. The smart terminal generates a response message based on the user's operation to indicate that the original visual feature map and the original obstacle map should not be updated, and sends it to the self-moving device.

[0091] Alternatively, if the user determines that the obstacles in the work area have changed and need to be updated, they can input an instruction to update the original visual feature map, the original obstacle map, or both via a smart terminal. The smart terminal receives the user's instruction and generates a response message to instruct the user to update the original visual feature map, the original obstacle map, or both, and sends it to the self-moving device.

[0092] Process B3: Obtain the response message generated by the feedback operation based on the requested information.

[0093] Process C3: Determine whether to update at least one of the original visual feature map and the original obstacle map based on the response message.

[0094] The mobile device receives a response message, and then uses the message's indication to determine whether to update, and if so, which map to update.

[0095] In another feasible approach, determining whether to update at least one of the original visual feature map and the original obstacle map can be implemented as follows: if the positioning quality in the working status information is lower than a quality threshold or the working area coverage is lower than a coverage threshold, then determine to update the original visual feature map and the original obstacle map used by the self-moving device.

[0096] When the positioning quality is low or the coverage of the working area is low, it means that the self-moving device is unable to perform well in terms of positioning and operation using the original visual feature map and the original obstacle map. Therefore, it needs to be updated to ensure the working effect.

[0097] Alternatively, you can determine whether an update is needed based on the current time and weather information. For example, the time can determine the season; if the season is significantly different, an update is warranted. Similarly, a significant weather change can also necessitate an update; for instance, if the weather is snowy, leading to substantial environmental changes, an update is recommended.

[0098] Regardless of the method described above, if it is determined that at least one of the original visual feature map and the original obstacle map should be updated, then step S206 is executed; otherwise, if it is determined that no update is needed, the process can be terminated.

[0099] Step S206: If an update is required, obtain an image frame of the working area and use the image frame to update at least one of the original visual feature map and the original obstacle map.

[0100] A feasible way to update the original visual feature map or the original obstacle map can be achieved through the following sub-steps:

[0101] Sub-step S2061: Acquire multiple image frames of the working area.

[0102] In one scenario, the image frame can be a real-time image of the location captured by a camera, which is a self-moving device that moves randomly within the working area.

[0103] Alternatively, in another scenario, the image frames can be acquired by moving along a planned route within the work area, capturing images of the work area during this movement. The planned route is a path designed based on the original visual feature map and the original obstacle map. This improves the movement efficiency of the self-moving device within the work area, covering the entire area of ​​the work area with the shortest possible path, thereby increasing the efficiency of image frame acquisition and the speed of updates.

[0104] Alternatively, in another scenario, existing image frames from historical workflows can be selected. These existing image frames can be part or all of the real-time images obtained in the most recent workflow. Alternatively, these existing image frames can be image frames obtained from multiple saved workflows, and part or all of the image frames from one or more workflows with higher coverage can be selected as the image frames used for this update.

[0105] Alternatively, when selecting image frames, target areas where obstacle positions change or visual positioning fails can be identified within the working area. New image frames are acquired within these target areas, and existing image frames corresponding to the remaining areas outside the target areas are selected. For example, if the working area is divided into multiple regions, and some regions have obstacle changes or a high probability of visual positioning failure, these regions are identified as target areas, and new image frames are acquired within these target areas. For the remaining areas outside the target areas, real-time images acquired in the previous or current working process can be used as image frames. This improves efficiency and makes full use of existing image frames.

[0106] Sub-step S2062: Based on the plurality of image frames, determine the acquisition pose of the image frame, the passage status of the pixels in the image frame, the new original visual feature map, and the new original obstacle map.

[0107] Based on the acquired image frames, a first neural network model can be used to identify them, thereby obtaining a visual localization feature map. Based on the positional relationships between feature points in the visual localization feature map, the acquisition pose of the self-moving device corresponding to the image frame can be calculated using the SFM method, which has a high localization accuracy. At the same time, the position of each feature point in the image frame in the original visual feature map is determined during the calculation of the acquisition pose, thus obtaining a new original visual feature map.

[0108] Similarly, the second neural network model can perform feature recognition on the acquired image frames to determine the passability status of each pixel in the image frame. Pixels corresponding to obstacles are marked as impassable, while pixels corresponding to areas such as grass outside the obstacles are marked as passable. Based on the passability status and acquisition pose of each pixel in the image frame, the corresponding pixel in the original obstacle map can be determined. The passability status of the corresponding pixel in the original obstacle map is then updated using the passability status of the pixels in the image frame, thereby obtaining a new original obstacle map.

[0109] Optionally, if the coordinate systems of the updated original obstacle map and the original visual feature map change relative to the coordinate systems of the original obstacle map or the original visual feature map before the update, the method may further include the following steps to align the coordinate systems.

[0110] Step S208: Align the updated map with the original map in terms of coordinates.

[0111] For ease of description, the updated map can be either an updated version of the original visual feature map or an updated version of the original obstacle map. The original map can be either the original visual feature map or the original obstacle map before the update.

[0112] In one feasible approach, if satellite positioning was not used to map the original map to a satellite coordinate system during the initial map creation process, coordinate alignment can be achieved through the following sub-steps:

[0113] Sub-step S2081: Select a first image frame for generating the original map and a second image frame for generating the updated map, and perform visual mapping to generate a reference map.

[0114] For example, the original map is generated using first image frames 1-5, and the updated map is generated using second image frames a-h. When generating the reference map, some or all of the first image frames 1-5 and some or all of the second image frames a-h can be selected. For example, first image frames 1 and 3, and second image frames e, f, and h can be selected. Based on the selected first image frames 1 and 3 and second image frames e, f, and h, a visual 3D reconstruction algorithm similar to that used to generate the original map is employed to create the reference map.

[0115] Sub-step S2082: Based on the first reference pose of the first image frame in the reference map, the first pose of the first image frame in the original map, the second reference pose of the second image frame in the reference map, and the second pose of the second image frame in the updated map, determine the mapping relationship between the coordinate system of the updated map and the coordinate system of the original map.

[0116] The selected first image frame has a corresponding pose in the original map, which is denoted as the first pose for ease of description. It also has a corresponding pose in the reference map, denoted as the first reference pose. Similarly, the selected second image frame has a corresponding pose in the updated map, denoted as the second pose, and it also has a corresponding pose in the reference map, denoted as the second reference pose.

[0117] Since the first reference pose and the second reference pose are in the same coordinate system, and the first pose and the first reference pose are representations of the same object or several objects in different coordinate systems, and the second pose and the second reference pose are also representations of the same object or several objects in different coordinate systems, and the relative poses of the object corresponding to the first pose and the object corresponding to the second pose in space are fixed, the transformation relationship between the first reference pose and the first pose, the relationship between the first reference pose and the second reference pose, and the transformation relationship between the second reference pose and the second position can determine the transformation relationship between the first pose and the second pose, thereby achieving coordinate alignment between the original map and the updated map.

[0118] In another feasible approach, if satellite positioning is used to map the original map to a satellite positioning coordinate system during the original map creation process, then the coordinate alignment can be achieved by: determining the mapping relationship between the coordinate systems of the updated map and the original map based on the pose of the updated map in the satellite positioning coordinate system and the pose of the original map in the satellite positioning coordinate system.

[0119] If the coordinate system of the original map has already been mapped to the satellite positioning coordinate system, the updated map can also be mapped to the satellite positioning coordinate system. Then, based on the pose transformation relationship between the two maps mapped to the satellite positioning coordinate system, the mapping relationship between the original map and the updated map can be determined, thereby achieving coordinate alignment.

[0120] Alternatively, in another feasible approach, if the updated map uses image frames (i.e., real-time images) acquired during the work process, and the real-time pose of the image frame in the original map is calculated when positioning is performed based on the image frame during the work process, then the mapping relationship between the original map and the updated map can be determined based on the real-time pose and the pose of the image frame in the updated map, thereby achieving coordinate system alignment between the original map and the updated map.

[0121] This method can solve the problem of being unable to locate when the scene of the work area changes with the seasons or the movement of obstacles in the work area. Since the original visual feature map and the original obstacle map are constructed and located visually, there is no need to lay positioning cables. Based on the location and image semantic segmentation, the method identifies changes in the work area, automatically judges scene changes and automatically updates the real-time obstacle map using real-time images of the self-moving device during the work process, thereby improving intelligence.

[0122] In this embodiment, the changes in obstacle area within the working area are determined based on the changes in obstacle area between the real-time obstacle map and the original obstacle map. The real-time obstacle map is generated based on the visual positioning of the mobile device and semantic segmentation of the collected real-time images, ensuring accuracy and timeliness of judgment.

[0123] When establishing the original visual feature map and the original obstacle map, a calculation method with high positioning accuracy is used to ensure the accuracy of the established original visual feature map and the original obstacle map. Similarly, when updating the original obstacle map or the original visual feature map, a positioning calculation method with slightly lower positioning accuracy but less computational load can be used to ensure fast positioning.

[0124] Example 3

[0125] Reference Figure 3 The diagram shows a structural block diagram of the map processing apparatus according to Embodiment 3 of this application.

[0126] The device includes:

[0127] The acquisition module 302 is used to acquire at least one of the working status information of the self-moving device in the current working process within the working area and the real-time obstacle map. The working status information includes at least one of the following: positioning quality and working area coverage. The real-time obstacle map is obtained based on real-time images collected during the movement.

[0128] The determining module 304 is used to determine, based on at least one of the working status information and the real-time obstacle map, whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device, wherein the original visual feature map is used for the positioning of the self-moving device in the working area and the original obstacle map is used for the path planning of the self-moving device.

[0129] The update module 306 is configured to, if an update is required, acquire an image frame of the working area and use the image frame to update at least one of the original visual feature map and the original obstacle map.

[0130] Optionally, the acquisition module 302 is configured to, when acquiring a real-time obstacle map of the self-moving device in the work area during the current working process, acquire a real-time image of the work area during the current working process, and match the feature data of the real-time image with the original visual feature map; determine the real-time pose of the self-moving device in the coordinate system of the original visual feature map based on the matching result; and determine the real-time obstacle map obtained in the current working process based on the real-time pose and the real-time image.

[0131] Optionally, the acquisition module 302 is configured to, when determining the real-time obstacle map obtained in the current working process based on the real-time pose and the real-time image, identify passable areas in the real-time image to determine the passability status of pixels in the real-time image; map the real-time image to the coordinate system of the original obstacle map based on the real-time pose to determine the passability status of the corresponding pixels in the original obstacle map; and determine the real-time obstacle map based on the passability status of the corresponding pixels in the original obstacle map.

[0132] Optionally, the determining module 304 is configured to determine the obstacle change area of ​​the real-time obstacle map based on the original obstacle map and the real-time obstacle map when determining whether to update the original obstacle map used by the self-moving device based on the real-time obstacle map; if the obstacle change area exceeds an area threshold, then it is determined to update the original obstacle map.

[0133] Optionally, the determining module 304 is configured to, when determining the obstacle change area of ​​the real-time obstacle map based on the original obstacle map and the real-time obstacle map, partition the original obstacle map and the real-time obstacle map to obtain at least one first partition corresponding to the original obstacle map and at least one second partition corresponding to the real-time obstacle map; determine the number of pixels whose passage status changes relative to the corresponding first partition in each second partition based on the passage status of pixels indicating the same position in the first and second partitions; and determine the obstacle change area of ​​the real-time obstacle map based on the number of pixels whose passage status changes in each second partition.

[0134] Optionally, the positioning quality is determined based on the ratio of the number of times the self-mobile device is successfully positioned using the original visual feature map during the current working process to the total number of times the self-mobile device is positioned; the working area coverage rate is the ratio between the area covered by the self-mobile device moving within the working area during the current working process and the total area of ​​the working area.

[0135] Optionally, the determining module 304 is configured to determine whether to update the original visual feature map and the original obstacle map used by the self-moving device when determining whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device based on the working status information. If the positioning quality in the working status information is lower than a quality threshold or the working area coverage is lower than a coverage threshold, then the module 304 determines to update the original visual feature map and the original obstacle map used by the self-moving device.

[0136] Optionally, the determining module 304 is configured to, when determining whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device based on the real-time obstacle map, generate an update request instruction based on the real-time obstacle map, and send the update request instruction to the smart terminal to instruct the smart terminal to display the real-time obstacle map and request information for requesting confirmation on whether to update at least one of the original visual feature map and the original obstacle map; obtain a response message generated by a feedback operation based on the request information; and determine whether to update at least one of the original visual feature map and the original obstacle map based on the response message.

[0137] Optionally, the update module 306 includes: acquiring multiple image frames of the working area; and determining the acquisition pose of the image frames, the passage status of pixels in the image frames, a new original visual feature map, and a new original obstacle map based on the multiple image frames.

[0138] Optionally, the update module 306 is configured to randomly move within the working area when acquiring multiple image frames of the working area, and acquire image frames of the working area during the movement; or, move within the working area according to a planned route, and acquire image frames of the working area during the movement, wherein the planned route is a path planned based on the original visual feature map and the original obstacle map; or, select existing image frames from the historical working process.

[0139] Optionally, when selecting existing image frames from the historical work process, the update module 306 determines the target area in the work area where the obstacle position changes or visual positioning fails, acquires new image frames in the target area, and selects existing image frames corresponding to the remaining areas outside the target area.

[0140] Optionally, the device further includes:

[0141] Alignment module 308 is used to align the updated map with the original map in terms of coordinates. The updated map can be the updated original visual feature map or the updated original obstacle map, and the original map can be the original visual feature map or the original obstacle map before the update.

[0142] Optionally, the alignment module 308 is used to select a first image frame for generating the original map and a second image frame for generating the updated map when aligning the updated map with the original map, and to perform visual mapping to generate a reference map; based on the first reference pose of the first image frame in the reference map, the first reference pose of the first image frame in the original map, the second reference pose of the second image frame in the reference map, and the second reference pose of the second image frame in the updated map, the mapping relationship between the coordinate system of the updated map and the coordinate system of the original map is determined.

[0143] Optionally, the alignment module 308 is used to determine the mapping relationship between the coordinate system of the updated map and the coordinate system of the original map based on the pose of the updated map in the satellite positioning coordinate system and the pose of the original map in the satellite positioning coordinate system when aligning the updated map with the original map.

[0144] The device can achieve the same effect as the above method, so it will not be described in detail here.

[0145] Example 4

[0146] In this embodiment, a self-moving device is provided, which includes a controller. The controller is used to execute the aforementioned map processing method and achieve the corresponding effect, which will not be described in detail here.

[0147] It should be noted that in the description of this invention, the terms "first" and "second" are used only for convenience in describing different components or names, and should not be construed as indicating or implying a sequential relationship, relative importance, or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0148] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0149] It should be noted that although specific embodiments of the present invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of the present invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of the present invention.

[0150] The examples of the embodiments of the present invention are intended to concisely illustrate the technical features of the embodiments of the present invention, so that those skilled in the art can intuitively understand the technical features of the embodiments of the present invention, and are not intended to be an improper limitation of the embodiments of the present invention.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A map processing method, characterized in that, include: The system acquires at least one of the following: working status information of the self-moving device in the current working process within the working area and a real-time obstacle map. The working status information includes at least one of the following: positioning quality and working area coverage. The positioning quality is determined based on at least one of the following: the ratio of the number of times the self-moving device is successfully positioned using the original visual feature map to the total number of times the self-moving device is positioned during the current working process, and the ratio of the interior points matched when successfully positioned using the original visual feature map to the total number of interior points. The real-time obstacle map is obtained based on real-time images collected during the movement. The working area coverage is the ratio between the area covered by the self-moving device within the working area during the current working process and the total area of ​​the working area. The working area coverage is used to reflect environmental changes in the working area. Based on at least one of the working status information and the real-time obstacle map, determine whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device, wherein the original visual feature map is used for the positioning of the self-moving device in the working area and the original obstacle map is used for the path planning of the self-moving device; If an update is required, an image frame of the working area is acquired, and the image frame is used to update at least one of the original visual feature map and the original obstacle map.

2. The method according to claim 1, characterized in that, The real-time obstacle map obtained from the mobile device during the current working process within the work area includes: During the current work process, the self-moving device acquires real-time images of the work area and matches the feature data of the real-time images with the original visual feature map; Based on the matching results, the real-time pose of the self-moving device in the coordinate system of the original visual feature map is determined; Based on the real-time pose and the real-time image, determine the real-time obstacle map obtained in the current working process.

3. The method according to claim 2, characterized in that, The step of determining the real-time obstacle map obtained in the current working process based on the real-time pose and the real-time image includes: The passable area is identified in the real-time image to determine the passability status of pixels in the real-time image; Based on the real-time pose, the real-time image is mapped to the coordinate system of the original obstacle map to determine the passage status of the corresponding pixel in the coordinate system of the original obstacle map. The real-time obstacle map is determined based on the passage status of the corresponding pixels in the coordinate system of the original obstacle map in the real-time image.

4. The method according to any one of claims 1-3, characterized in that, Based on the real-time obstacle map, determine whether to update the original obstacle map used by the self-moving device, including: Based on the original obstacle map and the real-time obstacle map, determine the obstacle change area of ​​the real-time obstacle map; If the changed area of ​​the obstacle exceeds the area threshold, then the original obstacle map is updated.

5. The method according to claim 4, characterized in that, The step of determining the obstacle change area in the real-time obstacle map based on the original obstacle map and the real-time obstacle map includes: The original obstacle map and the real-time obstacle map are partitioned to obtain at least one first partition corresponding to the original obstacle map and at least one second partition corresponding to the real-time obstacle map; Based on the traffic status of pixels indicating the same location in the first and second partitions, determine the number of pixels in each second partition that have different traffic status relative to the corresponding first partition; The obstacle change area of ​​the real-time obstacle map is determined based on the number of pixels showing changes in the passage status of each of the second zones.

6. The method according to claim 1, characterized in that, The step of determining whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device based on the working status information includes: If the positioning quality in the working status information is lower than the quality threshold or the working area coverage is lower than the coverage threshold, then it is determined to update the original visual feature map and the original obstacle map used by the self-moving device.

7. The method according to claim 1, characterized in that, Based on the real-time obstacle map, determining whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device includes: An update request instruction is generated based on the real-time obstacle map, and the update request instruction is sent to the smart terminal to instruct the smart terminal to display the real-time obstacle map and a request message for requesting confirmation on whether to update at least one of the original visual feature map and the original obstacle map; Obtain the response message generated by the feedback operation based on the requested information; Based on the response message, determine whether to update at least one of the original visual feature map and the original obstacle map.

8. The method according to claim 1, characterized in that, The step of acquiring image frames of the working area and using the image frames to update at least one of the original visual feature map and the original obstacle map includes: Acquire multiple image frames of the working area; Based on the multiple image frames, the acquisition pose of the image frames, the passage status of the pixels in the image frames, the new original visual feature map, and the new original obstacle map are determined.

9. The method according to claim 8, characterized in that, The acquisition of multiple image frames of the working area includes: The device moves randomly within the work area and acquires image frames of the work area during the movement; or... The operator moves within the work area according to a planned route and captures image frames of the work area during the movement. The planned route is a path planned based on the original visual feature map and the original obstacle map. Alternatively, select existing image frames from the historical work process.

10. The method according to claim 9, characterized in that, The selection of existing image frames from the historical work process includes: Identify the target area in the working area where the obstacle position changes or visual positioning fails, acquire new image frames within the target area, and select existing image frames corresponding to the remaining areas outside the target area.

11. The method according to claim 1, characterized in that, The method further includes: The updated map is aligned with the original map in terms of coordinates. The updated map can be either the updated original visual feature map or the updated original obstacle map. The original map can be either the original visual feature map or the original obstacle map before the update.

12. The method according to claim 11, characterized in that, The process of aligning the updated map with the original map in terms of coordinates includes: A first image frame used to generate the original map and a second image frame used to generate the updated map are selected to perform visual mapping to generate a reference map; Based on the first reference pose of the first image frame in the reference map, the first pose of the first image frame in the original map, the second reference pose of the second image frame in the reference map, and the second pose of the second image frame in the updated map, the mapping relationship between the coordinate system of the updated map and the coordinate system of the original map is determined.

13. The method according to claim 11, characterized in that, The process of aligning the updated map with the original map in terms of coordinates includes: Based on the pose of the updated map in the satellite positioning coordinate system and the pose of the original map in the satellite positioning coordinate system, the mapping relationship between the coordinate systems of the updated map and the original map is determined.

14. A map processing apparatus, characterized in that, include: The acquisition module is used to acquire at least one of the following: the working status information of the self-moving device in the current working process within the working area and the real-time obstacle map. The working status information includes at least one of the following: positioning quality and working area coverage. The positioning quality is determined based on at least one of the following: the ratio of the number of times the self-moving device is successfully positioned using the original visual feature map to the total number of times the self-moving device is positioned during the current working process, and the ratio of the interior points matched when successfully positioned using the original visual feature map to the total number of interior points. The real-time obstacle map is obtained based on real-time images collected during the movement. The working area coverage is the ratio between the area covered by the self-moving device within the working area during the current working process and the total area of ​​the working area. The working area coverage is used to reflect environmental changes in the working area. The determination module is used to determine, based on at least one of the working status information and the real-time obstacle map, whether to update at least one of the original visual feature map and the original obstacle map used by the self-moving device, wherein the original visual feature map is used for the positioning of the self-moving device in the working area and the original obstacle map is used for the path planning of the self-moving device. An update module is configured to, if an update is required, acquire an image frame of the working area and use the image frame to update at least one of the original visual feature map and the original obstacle map.

15. A self-moving device, characterized in that, Includes a controller for performing the map processing method according to any one of claims 1-13.

Images