Image processing method and device, storage medium and self-moving equipment

Abstract

The invention provides an image processing method and device, a storage medium and self-moving equipment, and relates to the field of mobile positioning. According to the image processing method, an image tracking method and an image feature point algorithm are fused, so that the tracking continuity and stability are improved, the environment perception capability is enhanced, and the walking control precision of the self-moving equipment is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mobile positioning, in particular to an image processing method and device, storage medium and self-moving equipment. BACKGROUND

[0002] With the rapid development of artificial self-moving equipment, the control technology of robots also develops rapidly. In the prior art, the robot can automatically track the surrounding target object during the working process, and successfully identify the target object. However, in the prior art, the feature point detection and feature description algorithm are used to realize visual tracking of multiple frames of images, but it also has obvious defects, such as uneven distribution of extracted feature points in the image and serious aggregation, which leads to poor image matching accuracy and large deviation of the equipment pose estimation result. SUMMARY

[0003] The main purpose of the present application is to provide an image processing method and device, which aims to realize accurate control of the equipment during walking and meet the needs of users.

[0004] The first aspect of the embodiment of the present application provides an image processing method, which comprises:

[0005] An image sequence is obtained, which comprises environment images collected by at least one image collection device within the target range of the self-moving equipment;

[0006] According to a first image algorithm, initial feature points of a plurality of point clouds in the image sequence are extracted;

[0007] The image sequence is segmented to generate a plurality of image blocks, and the initial feature points of each image block are uniformly extracted to generate target feature points;

[0008] Based on a second image algorithm, the target feature points are tracked and processed, and an image translation vector is outputted to realize the calculation of the pose of the self-moving equipment.

[0009] The second aspect of the embodiment of the present application provides an image processing device, which comprises:

[0010] An image collection module is configured to obtain an image sequence, which comprises environment images collected by at least one image collection device within the target range of the self-moving equipment;

[0011] An image processing module is configured to extract initial feature points of a plurality of point clouds in the image sequence according to a first image algorithm;

[0012] The image sequence is segmented to generate a plurality of image blocks, and the initial feature points of each image block are uniformly extracted to generate target feature points;

[0013] Based on the second image algorithm, the target feature points are tracked, and an image translation vector is output to realize the calculation of the pose of the self-moving device.

[0014] In a third aspect, the embodiments of the present application provide a self-moving device, comprising a memory and a processor, the memory storing a computer program, and the processor realizing the steps of any one of the above methods when executing the computer program.

[0015] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program realizes the steps of any one of the above methods when executed by a processor.

[0016] The embodiments of the present application provide an image processing method, which fuses the optical flow tracking method and the feature point matching method. This fusion method not only improves the continuity and stability of tracking, enhances the environmental perception ability, and improves the robustness and adaptability of the algorithm.

[0017] Compared with the prior art, the image processing method provided by the embodiments of the present application has the following advantages:

[0018] Improved positioning accuracy: Accurate feature point matching helps the self-moving device to realize high-precision positioning. By matching the same feature points between different frames, the specific position and direction of the self-moving device in the environment can be estimated.

[0019] Enhanced obstacle avoidance capability: The fusion method in visual tracking can more accurately identify obstacles in the environment. By tracking the positions and motion states of these obstacles in real time, the self-moving device can flexibly adjust its path planning to avoid collisions with obstacles.

[0020] Robustness enhancement: The fusion method can maintain good tracking effect in complex and variable environments by comprehensively utilizing the advantages of optical flow tracking and feature point matching. Even in the face of challenges such as light changes, occlusions, and dynamic scenes, it can maintain a certain robustness. BRIEF DESCRIPTION OF DRAWINGS

[0021] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings:

[0022] Figure 1 is an application scenario diagram of the image processing method provided by an embodiment of the present application;

[0023] Figure 2 is a flowchart of the image processing method provided by an embodiment of the present application;

[0024] Figure 3 is an image processing diagram in the image processing method provided by an embodiment of the present application Figure 1 ;

[0025] Figure 4A is a schematic diagram of image processing in an image processing method provided by an embodiment of the present application Figure 2 ;

[0026] Figure 4B is a schematic diagram of image processing in an image processing method provided by an embodiment of the present application Figure 3 ;

[0027] Figure 5 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms used in the present application are merely for the purpose of describing specific embodiments and are not intended to limit the present application. For example, the words "upper", "lower", "front", "back", and the like indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the drawings, merely for the purpose of facilitating the description of the present application and simplifying the description, and are not intended to indicate or imply that the device / element referred to must have a particular orientation or be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0029] Visual target tracking refers to continuous monitoring and tracking of a pre-selected target in a video sequence to obtain its motion information such as position, speed, acceleration, and feature information such as shape and size. In the application of an intelligent mower, visual target tracking is mainly used to identify and track obstacles, boundary lines and other objects that need special attention in the lawn.

[0030] The VIO system (visual-inertial odometry) in the prior art tracks feature points by using an optical flow method. In order to adapt to a lower-performance arm platform, a specific number of feature points need to be selected for pose calculation. Therefore, a reasonable, effective and fast feature point selection method is needed. Most VIO systems use a simple grid to extract feature points, that is, a grid is drawn, and at most N feature points can exist intelligently in the grid. Therefore, when the number of feature points exceeds N, the feature points will not be added to the feature point database and will not be used for subsequent pose calculation. However, this method of screening feature points is uniformly distributed in an image. However, when the image is unevenly bright and dark, such as the right lower corner having a higher brightness and less texture, and the left upper corner having rich texture and sufficient features, the grid method can still extract N feature points in the right lower corner, but the quality of these feature points is not high.

[0031] In the visual target tracking method, the first type of method is a feature point matching method. This method is based on feature point detection and descriptor matching to achieve target tracking, and has the advantage of high precision. However, with the increase of time, the computational complexity increases sharply, and it is easy to lose, resulting in low real-time performance and poor robustness of the algorithm, so there are greater scene constraints.

[0032] The second type of method is an optical flow tracking method. This method mainly uses an optical flow tracking method based on corner points, and has the advantages of strong robustness and good real-time performance. However, this method is too sensitive to noise, light source and scale change, and the detection of corner points has randomness, resulting in low tracking accuracy of the target.

[0033] In view of the problem of low accuracy and low computational efficiency caused by the above visual target tracking processing, the embodiment of the application provides an image processing method. The method combines the respective advantages of the traditional optical flow tracking method and the feature point matching method, can overcome the low accuracy of the optical flow tracking method in weak texture scenes, can solve the problem of high computational complexity and low robustness of the feature point matching method in long-time running environment, can be applied to a target tracking algorithm in a wide range of real environment, and has the advantages of rapidity, accuracy and robustness.

[0034] The scheme in the embodiment of the application can be implemented in various computer languages, for example, object-oriented programming languages such as Java and interpreted scripting languages such as JavaScript.

[0035] In order to make the technical solutions and advantages in the embodiments of the application clearer and more apparent, the following will further describe the exemplary embodiments of the application in detail with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0036] Please refer to the accompanying drawings Figure 1 The method embodiments provided in the embodiments of the application can be executed in a self-moving device or similar operation device. The self-moving device 10 can include one or more processors 101 or programmable logic devices (PLD) 101, a camera device 102 for image acquisition function, and a memory 103 for storing data.

[0037] In an exemplary embodiment, the self-moving device described above can further include a transmission device for communication function and an input and output device.

[0038] In an exemplary embodiment, the self-moving device described above can further include a driving device for self-moving function.

[0039] Those skilled in the art will understand that self-moving devices may also include more or fewer components than those shown in the above modules, or have different configurations with equivalent or more functions than those shown in the above-described robot. The memory can be used to store computer programs, such as software programs and modules for application software, like the computer program corresponding to the image processing method in this embodiment of the invention. The processor executes various functional applications and data processing by running the computer program stored in the memory, thereby implementing the above-described method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories.

[0040] Please see the appendix Figure 2 This application provides an image processing method applied to a self-moving device 10. The method is illustrated by taking the example of how the device achieves self-movement through image acquisition and data modeling. The image processing method provided in this application includes the following steps 201-203:

[0041] Step 201: Obtain the image sequence.

[0042] The image sequence includes environmental images acquired by at least one image acquisition device within a target range of the mobile device. The target range can be set according to the working area, the physical parameters of the image acquisition device, or the physical parameters of the mobile device.

[0043] Self-moving devices include those equipped with motors and processors, which can send signals through the processor to control the motors and drive the device, such as intelligent robots, intelligent lawnmowers, intelligent robotic vacuum cleaners, or other work equipment.

[0044] The captured image data is transmitted in real time to the image processing unit inside the device. This unit typically contains a high-performance computer chip or specialized image processing hardware. The image processing unit performs preliminary processing on the raw image data, such as denoising, correction, and enhancement, to improve the accuracy of subsequent analysis.

[0045] Step 202: Extract initial feature points of several point clouds from the image sequence according to the first image algorithm.

[0046] Feature points in point clouds are an important type of feature in point cloud data. These feature points can be used to analyze and describe point cloud data. According to the physical properties of the features, they can be divided into: geometric domain and intensity domain; according to the spatial scale of the features, they can be divided into: single-point features, local features, and global features.

[0047] The first image algorithm mentioned above may include the ORB algorithm, which is a fast and efficient feature point extraction and description algorithm. It combines the advantages of FAST (Features from Accelerated Segment Test) feature point detection and BRIEF (Binary Robust Independent Elementary Features) descriptor, and adds the directionality of feature points, improving rotation invariance.

[0048] For example, FAST keypoint detection extracts feature points from image frames. Corner points are detected by comparing the brightness difference between a pixel and its neighboring pixels; if certain conditions are met, it is considered a feature point. Specifically, it selects a pixel as a candidate point and sets a threshold T (usually a certain proportion of the candidate point's brightness). Then, it examines the pixels on a circle (usually with a radius of 3) surrounding the candidate point. If a sufficient number of consecutive pixels on the circle (e.g., 12) are brighter or darker than the candidate point by more than the threshold T, the candidate point is considered a corner point. The FAST algorithm is relatively simple in principle, easy to implement, reduces computational complexity and resource consumption, and is more stable and reliable in practical applications.

[0049] In natural environments such as turf, lighting conditions can change with time and weather. The FAST algorithm is robust to changes in lighting and can extract corner features relatively stably under different lighting conditions.

[0050] Step 203: Segment the image sequence to generate several image blocks, and uniformly extract the initial feature points in each image block to generate target feature points.

[0051] In the above steps, strategies such as quadtrees or octrees can be used to achieve uniform extraction of initial feature points, thereby improving the accuracy and stability of subsequent feature matching and navigation.

[0052] Homogenization can cluster feature points with high density or similar features into the same sub-region, removing redundant and invalid feature points and retaining feature points that are more valuable for subsequent processing, thereby improving the accuracy and efficiency of feature point matching.

[0053] In visual recognition applications of smart devices, the ORB algorithm achieves uniform feature point extraction by combining FAST feature point detection and quadtree partitioning. This method not only improves the efficiency of feature point extraction but also ensures the uniformity of feature point distribution, providing a reliable foundation for subsequent feature matching and navigation. Furthermore, the ORB algorithm possesses advantages such as rotation invariance and noise resistance, enabling it to meet the visual recognition needs of smart lawnmowers in complex environments.

[0054] Step 204: According to the second image algorithm, track and process the target feature points, and output the image translation vector to realize the pose calculation of the self-moving device.

[0055] The second image algorithm includes at least the LK optical flow method. The LK optical flow method can effectively deal with the unique texture features and dynamic changes (such as grass rustling in the wind) of the grass environment by modeling and analyzing the gray-scale changes of the pixels around the feature points, and accurately track the feature points.

[0056] In existing technologies, feature points in images can be extracted using the ORB (Oriented Fast and Rotated BRIEF) algorithm. However, this application is mainly applied to the working scenarios of lawnmowers, such as lawns. When processing complex textures such as lawns and trees, this algorithm still has some shortcomings: because grass usually has a high degree of texture similarity, that is, grass in different areas may be very similar visually, the ORB algorithm has a high difficulty in distinguishing when extracting feature points, resulting in the algorithm extracting a large number of duplicate or invalid feature points, which affects the subsequent matching and recognition results. Therefore, a second algorithm is needed to improve the image processing capabilities.

[0057] The ORB algorithm, combined with the quadtree feature point partitioning method, can efficiently extract and evenly distribute feature points in the visual recognition application of smart lawnmowers. The filtered feature points serve as the starting point for LK optical flow tracking, providing a reliable foundation for subsequent tracking and localization.

[0058] The LK optical flow method accurately estimates the position and motion state of a lawnmower by tracking feature points extracted from the ORB in real time. This high-precision tracking and positioning capability enables intelligent lawnmowers to achieve precise navigation and obstacle avoidance in complex lawn environments.

[0059] The combination of ORB and LK optical flow is also reflected in its excellent real-time performance and robustness. The ORB algorithm can quickly extract feature points, while the LK optical flow algorithm can efficiently track these feature points. At the same time, both have a certain degree of anti-interference capability and can maintain stable performance in complex environments such as changes in lighting and occlusion.

[0060] In conclusion, the quadtree feature point partitioning method in ORB has significant application value in the visual recognition of intelligent lawnmowers. Combining it with the LK optical flow method for feature point tracking can further improve the navigation accuracy and stability of lawnmowers, providing strong support for the widespread application of intelligent lawnmowers.

[0061] This application provides an image processing method that combines optical flow tracing with feature point matching. This fusion method not only improves the continuity and stability of tracking, enhances environmental perception, and improves the robustness and adaptability of the algorithm, but also promotes the optimization and intelligent development of the algorithm.

[0062] The fusion method provided in this application is highly flexible and can be adjusted and optimized according to different application scenarios and needs. For example, in environments with rich textures, feature point matching can be relied upon more; while in fast-moving or low-texture environments, optical flow tracing can be relied upon more.

[0063] In an optional embodiment, the method provided in this application extracts initial feature points of several point clouds in the image sequence in step 202 above, including:

[0064] Step 301: Extract the grayscale variation of point clouds in different regions of the environmental image;

[0065] Step 302: Obtain the feature points of the point cloud.

[0066] Based on the results of image segmentation and the requirements of feature point distribution, adjust the parameters of the feature point detection algorithm. These parameters may include the threshold for corner detection, the number of levels in the scale space, the radius of non-maximum suppression, etc.; the parameter adjustments should ensure that feature points can be extracted uniformly and effectively in different image patches.

[0067] The adjusted feature point detection algorithm is applied independently within each image patch. The algorithm iterates through each pixel in the image patch, determining the presence of feature points based on preset criteria and parameters.

[0068] In an optional embodiment, the method provided in this application, which involves uniformly extracting the initial feature points of each image block as mentioned in step 203 to generate target feature points, includes at least the following steps:

[0069] Step 401: Split the current frame image into N nodes, where N is greater than or equal to 4;

[0070] Divide the image into root nodes, defining the number of root nodes as w / h, where w is the image pixel width and h is the image pixel height. If the calculated value of w / h is a decimal, round it to N.

[0071] A root node represents an image region. At this time, the image is divided into n regions, and the size of each region is w / n*h.

[0072] Step 402: Gradually determine the number of feature points contained in each node. If the number of feature points in the node is greater than 1, then continue to split it into N child nodes. If the number of feature points in the node is equal to 1, then the node will not be divided into child nodes and the node will be stored.

[0073] Step 403: If the number of nodes after the current split is greater than or equal to the number of feature points to be extracted, stop splitting and calculate the response value of at least one feature point within the node.

[0074] The number of feature points to be extracted mentioned above can be set according to different regions, such as whether the current image patch is a changed region compared with the previous frame: if it is a changed region, the number of features to be extracted can be increased; if it is a non-changed region, the number of features to be extracted can be appropriately reduced.

[0075] Step 404: Based on the response value of the at least one feature point, determine the target feature point to be retained in the current node, and remove the remaining feature points.

[0076] Where Na is the number of feature points that the system needs for optical flow tracking.

[0077] The response value of a feature point can include the Harris response value, which is a numerical value indicating the degree to which the pixel is a corner point.

[0078] The stability of the target feature points is evaluated based on their motion vectors, and the evaluation result for each target feature point is obtained.

[0079] Based on the stability evaluation results of the target feature points, stable and effective feature points are selectively retained, while unstable feature points caused by noise, occlusion, or false detection are removed.

[0080] The above process of averaging the initial feature points can be referred to as attached. Figure 3 The process shown:

[0081] First, divide an image into four equal parts. Then, filter the feature points in the divided regions. If there are more than one feature point in a region, then the region can be divided into four more parts. Repeat the same operation to continue using the four-part division method to divide the image into the desired number of regions.

[0082] The method provided in this application embodiment is not sufficient to simply use the octree method to filter feature points; it is also necessary to determine whether a region can be used to extract feature points during the feature point extraction stage.

[0083] Get the image region corresponding to the current node;

[0084] Extract the feature information of the image region;

[0085] If the feature information of the image region indicates that the image region is located in a region of no interest, then the target feature points are not extracted in the image region. The region of no interest includes at least the bottom of the image, the shadow region of the machine itself, or the edge region.

[0086] Here, methods such as masks can be used to exclude regions of no interest. Of course, for engineering considerations, this application provides a function to extract the region of interest and extract FAST or HARRIS feature points in this region.

[0087] The objective function should at least include: constructing a struct.

[0088] Struct Area{

[0089] Double x_start;

[0090] Double y_start;

[0091] Double x_end;

[0092] Double y_end;

[0093] }

[0094] Construct a vector <area> The `areas` container is used as input to a function. The function first determines the size of the `areas`, i.e., the number of regions of interest, and then extracts valid feature points within these regions. Additionally, the function body includes a check for region coordinate errors and outputs an error log.

[0095] It is worth emphasizing again that the mean-based processing of the initial feature points in the method provided in this application does not completely use the feature point partitioning in the ORB algorithm. After the initial feature points are extracted by the ORB algorithm, the subsequent mean-based processing of the initial feature points does not require the extraction of an image pyramid, and correspondingly, it does not require quadtree partitioning at each level of the pyramid. Instead, quadtree partitioning is performed on the original image, which greatly reduces the amount of data processed in the image, thereby further improving the efficiency of layer processing and the accuracy of feature point extraction.

[0096] In an optional embodiment, the second image algorithm in step 204 of the method provided in this application includes at least an optical flow method. The step of tracking and processing the target feature points based on the second image algorithm includes:

[0097] Step 501: Process the continuous frame image sequence according to the optical flow method, and calculate the motion vector of each target feature point in the current frame image and the previous frame image;

[0098] Step 502: Based on the motion vector of the target feature point, match the calculated position of the feature point in the current frame with the feature point in the previous frame to achieve tracking of the target feature point.

[0099] Using visual target tracking technology, self-moving devices can detect and track obstacles in the lawn in real time, thereby planning a mowing path that avoids obstacles. This not only improves mowing efficiency but also prevents damage caused by collisions between the lawnmower and obstacles.

[0100] Please refer to the attached document. Figure 4A The map shown includes the initial feature points and the attached map. Figure 4B The image shown is the processed map including target feature points, with appended... Figure 4A and attached Figure 4B The distribution effects of different processing strategies on feature points (shown as green points in the image) in the same graph:

[0101] Appendix Figure 4A Based on the method provided in the embodiments of this application, this example uses octree processing to show the result of feature point homogenization. Figure 4B In existing technologies, VIO systems use the VINS-MONO algorithm to average feature points, which involves directly dividing the system into multiple regions and then filtering the feature points.

[0102] By comparing the distribution of feature points in the two images, it can be seen that there are significantly more feature points on the lawn in the first image. This is because the lawn has abundant feature points, so the octree homogenization method yielded more and better feature points. The second image is the result of ordinary homogenization. Although the point distribution is relatively uniform, there are also many feature points on the white wall, and the recognition effect of these feature points is not as good as that on the grass. Therefore, the processing method provided by the embodiments of this application can retain more feature points with high saliency, high discriminancy, and high responsivity in the target area, such as the grass, and the feature points extracted in the target area will also be evenly distributed.

[0103] The visual odometry component of the self-driving device uses the VIO algorithm. It acquires 1280*720 pixel image data using a monocular camera. From this image, four threads extract either FAST or HARRIS feature points, resulting in 1000 feature points. The goal is to select 200 evenly distributed feature points. An octree method is used to select 200 relatively evenly distributed feature points with high response values ​​from the 1000 points. These feature points are assigned individual IDs, and the LK optical flow method is used to track them, identifying matching feature points within the image frame. Finally, feature point distortion correction is performed based on camera intrinsics, deleting feature points that do not meet the criteria or are near edge pixels. The matched feature points with high response values ​​are then stored in a container, completing the extraction and tracking for the current frame.

[0104] The processing method provided in this application calculates the optical flow (i.e., motion vector) of pixels or feature points between adjacent frames using optical flow tracing, which can continuously track the motion trajectory of feature points, thereby maintaining the continuity of tracking and ultimately achieving good tracking effect for smooth and continuous motion.

[0105] In an optional embodiment, step 204 of the method provided in this application, which involves tracking and processing the target feature points, further includes:

[0106] Step 601: If it is determined that the target feature point has been lost during tracking by matching feature points between consecutive frames, then candidate feature points are extracted based on the initial feature points of the several point clouds.

[0107] Feature point tracking loss refers to the situation where, in a series of video frames or image sequences, feature points that were initially successfully identified and tracked suddenly fail to be correctly matched or identified in one or more frames. This situation can be caused by a variety of factors, including but not limited to:

[0108] The target object may be occluded by other objects, making the feature points invisible; or, a sudden change in ambient lighting may cause the appearance of the feature points to change, making it impossible for the matching algorithm to identify them correctly; or, the rapid movement of the target object or camera shake may cause the image to be blurred, making the feature points unclear; or, a sudden change in the camera's perspective (such as rotation, zoom, etc.) may cause the position of the feature points in the image to change significantly, exceeding the search range of the tracking algorithm; it is also possible that the tracking algorithm itself has certain limitations, such as insufficient ability to identify specific types of feature points or insufficient robustness of the matching algorithm.

[0109] Step 602: Track the target feature point based on the candidate feature points;

[0110] The extraction of candidate feature points includes:

[0111] The motion model of the object is used to predict the position of feature points, and feature point matching is performed near the predicted position.

[0112] The method for determining whether feature point tracking loss has occurred in the embodiments of this application may include: determining whether loss has occurred based on the stability or continuity of the tracked feature points: if the displacement of a feature point in a certain frame is significantly increased compared with the previous few frames, it can be considered that tracking loss has occurred.

[0113] For example, to determine whether feature point loss has occurred based on the feature point's identification information: optical flow tracing algorithms generate a status flag for each feature point, indicating whether each feature point has been successfully tracked. For instance, when using the function in OpenCV, it returns a status vector, where each element corresponds to the status of a feature point, with 1 indicating successful tracking and 0 indicating tracking failure.

[0114] For example, the tracking quality can be used to determine whether a feature point is lost. If the tracking error of a feature point is greater than a certain threshold, it can be considered that the feature point has been lost.

[0115] For example, abnormal motion vectors: If the calculated motion vectors do not match the expected motion pattern (such as sudden changes in direction or excessive speed), this could also be a sign of tracking loss.

[0116] The processing method provided in this application can address the phenomenon of feature point loss during the tracking process by addressing it from the following aspects:

[0117] Example 1: Feature point detection can be re-performed in the current frame or subsequent frames by reapplying the feature point detection algorithm to the image to find new, traceable feature points. In the initial stage, as many feature points as possible are detected and tracked. Even if some feature points are lost, others can still be used to track and locate the target object.

[0118] Example 2: The position of feature points can be predicted by using the motion model of the object (such as uniform motion, accelerated motion, etc.) and feature point matching can be performed near the predicted position, thereby alleviating the tracking loss problem caused by motion blur or viewpoint changes to some extent.

[0119] Example 3: The tracking algorithm can be optimized to improve its robustness to factors such as changes in lighting, occlusion, and viewpoint. For example, strategies such as lighting-invariant feature descriptors and multi-scale feature detection can be introduced.

[0120] Example 4: A failure recovery mechanism can also be used to automatically switch to re-detection mode or take other remedial measures when the tracking algorithm fails to track feature points correctly for multiple consecutive frames, thereby ensuring that the system can maintain stable tracking performance when facing complex scenes.

[0121] Example 5: Visual tracking results can be fused and verified with other sensor data by combining various sensors (such as inertial navigation units, GPS, etc.) in self-moving devices. This can help improve the overall stability and accuracy of the system and reduce the risk of tracking loss due to the failure of a single sensor.

[0122] The processing method provided in this application embodiment can alleviate or solve the problem of feature point tracking loss to a certain extent in visual recognition applications through reasonable strategies and optimized algorithm design, thereby improving the stability and reliability of the system and further enhancing the intelligence level and user experience of the smart lawnmower.

[0123] Please see Figure 5 One embodiment of this application provides an image processing apparatus 50, comprising:

[0124] Image acquisition module 501 acquires an image sequence, the image sequence including environmental images acquired from the target range of the mobile device by at least one image acquisition device;

[0125] Image processing module 502 is used to extract initial feature points of several point clouds in the image sequence according to the first image algorithm;

[0126] The image sequence is segmented to generate several image blocks, and the initial feature points of each image block are extracted uniformly to generate target feature points;

[0127] Based on the second image algorithm, the target feature points are tracked and processed, and an image translation vector is output to calculate the pose of the self-moving device.

[0128] This application provides an image processing apparatus that integrates optical flow tracing with feature point matching. This integration method not only improves the continuity and stability of tracking, enhances environmental perception, and improves the robustness and adaptability of the algorithm, but also promotes the optimization and intelligent development of the algorithm.

[0129] For specific limitations on the aforementioned image processing device, please refer to the limitations on image processing methods for self-moving devices mentioned above, which will not be repeated here.

[0130] Each module in the mobile device of the aforementioned self-moving device can be implemented entirely or partially through software, hardware, or a combination thereof.

[0131] The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or they can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0132] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, can perform any step of the image processing method for the self-moving device described above.

[0133] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. An image processing method, characterized by, The method is applied to a self-moving device, and comprises: acquiring an image sequence, the image sequence comprising environment images of a target range of the self-moving device collected by at least one image collection device; extracting initial feature points of a plurality of point clouds in the image sequence according to a first image algorithm; segmenting the image sequence to generate a plurality of image blocks, and uniformly extracting the initial feature points in each image block to generate target feature points; tracking the target feature points according to a second image algorithm, and outputting an image translation vector to realize calculation of a pose of the self-moving device.

2. The image processing method of claim 1, wherein, The method comprises: extracting local pixel gray scale variation degrees of point clouds in different regions to obtain feature points of the point clouds.

3. The image processing method according to claim 1, characterized by, The processing result comprises at least: splitting a current frame image into N nodes, wherein N is greater than or equal to 4; gradually judging the number of feature points contained in each node, if the number of feature points in the node is greater than 1, the node is further split into N sub-nodes, if the number of feature points in the node is equal to 1, the node is not divided into sub-nodes and the node is stored; if the number of nodes after the current splitting is greater than or equal to the number of feature points to be extracted, the splitting is stopped, and a response value of at least one feature point in the node is calculated; determining a target feature point to be reserved in the current node according to the response value of the at least one feature point, and eliminating the remaining feature points.

4. The image processing method of claim 3, wherein, The response value is determined according to at least the stability and the recognizability of the feature points.

5. The image processing method of claim 4, wherein, The second image algorithm comprises at least an optical flow method, and the tracking of the target feature points according to the second image algorithm comprises: processing a continuous frame image sequence according to the optical flow method to calculate a motion vector of each target feature point in a current frame image and a previous frame image; matching the position of the feature point calculated in the current frame with the feature point in the previous frame according to the motion vector of the target feature point to realize tracking of the target feature point.

6. The image processing method of claim 4, wherein, The tracking of the target feature point is realized through matching of the feature points between continuous frames, and comprises: if it is determined through matching of the feature points between continuous frames that the target feature point has lost tracking, candidate feature points are extracted according to the initial feature points of the plurality of point clouds; tracking the target feature point according to the candidate feature points; The extraction of the candidate feature points comprises: predicting the position of the feature point by using a motion model of an object, and matching the feature point in the vicinity of the predicted position.

7. The image processing method of claim 3, wherein, Before determining the target feature point to be reserved in the current node, the method further comprises: acquiring an image region corresponding to the current node; extracting feature information of the image region; if the feature information of the image region indicates that the image region is located in a region of no interest, the target feature point is not extracted in the image region, and the region of no interest at least comprises a bottom of the image, a shadow region of the machine itself or an edge region.

8. An image processing apparatus characterized by comprising: The control device comprises: an image collection module, which acquires an image sequence, the image sequence comprising environment images of a target range of the self-moving device collected by at least one image collection device; an image processing module, which is configured to extract initial feature points of a plurality of point clouds in the image sequence according to a first image algorithm; The segmentation processing generates a plurality of image blocks from the image sequence, and uniformly extracts the initial feature points of each image block to generate target feature points; According to a second image algorithm, the target feature points are tracked and processed, and an image translation vector is outputted to realize the calculation of the pose of the self-moving device.

9. A computer readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method in any one of claims 1 to 7 when running.

10. A self-moving device comprising a memory and a processor, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method in any one of claims 1 to 7 when running.

Images