Image processing method, control device, mobile robot and medium

Through optical flow point matching and homography matrix mapping technology, the problem of real-time image processing performance degradation in fields such as mobile robots under limited resource conditions is solved, efficient and highly accurate image processing is achieved, and the system stability and frames are improved. Rate.

CN118941824APending Publication Date: 2024-11-12MIDEA ROBOZONE TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410942856.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Under limited resource conditions, real-time image processing tasks in fields such as mobile robots and drones are difficult to meet the requirements of high efficiency and high accuracy, especially in the case of multi-model parallel processing and high frame rate input data, resulting in system performance degradation .

Method used

By obtaining the reference frame and the frame to be matched by the video stream, the optical flow point matching is performed, the homography matrix is ​​determined, and the segmentation information is mapped to the frame to be matched based on the homography matrix, the matching of the video stream frame is achieved, the resource consumption is reduced, and the system is improved Stability and accuracy.

Benefits of technology

Real-time application of higher frame rates under limited resource conditions, improve system stability and accuracy, reduce resource consumption of multi-model parallel processing, and maintain spatial consistency between different frames.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941824A_ABST
    Figure CN118941824A_ABST
Patent Text Reader

Abstract

The invention discloses an image processing method, a control device, a mobile robot and a medium, and the method comprises the steps: obtaining a reference frame and a to-be-matched frame of a video stream, and obtaining the segmentation information outputted after the reference frame is processed through a preset segmentation model; performing optical flow point matching on the reference frame and the to-be-matched frame to obtain a matching point set; determining a homography matrix according to the matching point set; and mapping the segmentation information to the to-be-matched frame based on the homography matrix to obtain a target image. In the embodiment of the invention, the utilization rate of segmentation model output can be improved while the real-time requirement is met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to an image processing method, a control device, a movable robot and a medium. Background Art

[0002] With the rapid development of computer vision and deep learning technologies, more and more application scenarios require complex image processing tasks, such as object detection, key point detection, and semantic / instance segmentation. These tasks are usually completed using different deep learning models that are trained specifically for specific problem areas. However, in many cases, such as mobile robots, drones, and intelligent monitoring systems, the resources of terminal devices are limited, including but not limited to computing power, memory capacity, and power. Therefore, in these scenarios, it becomes particularly important to efficiently utilize limited hardware resources.

[0003] In the case of multi-model parallel processing, resource consumption increases significantly, especially when the input data is provided at a high frame rate, a single or multiple models may not be able to process all input frames in time. In addition, real-time applications have high requirements for latency and accuracy. If these requirements are not met, the performance of the entire system will be affected. Therefore, a method is needed to address these challenges so that efficient real-time image processing can be achieved under limited resource conditions. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, a control device, a movable robot and a medium, which can improve the utilization rate of the segmentation model output while meeting the real-time requirements.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Obtaining a reference frame and a frame to be matched of a video stream, and obtaining segmentation information output after the reference frame is processed by a preset segmentation model;

[0007] Performing optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set;

[0008] Determine a homography matrix according to the matching point set;

[0009] The segmentation information is mapped to the frame to be matched based on the homography matrix to obtain a target image.

[0010] According to the image processing method of the embodiment of the first aspect of the present application, there are at least the following beneficial effects: the embodiment of the present application obtains the reference frame and the frame to be matched of the video stream, and obtains the segmentation information output after the reference frame is processed by a preset segmentation model, and then performs optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set, and then determines the homography matrix according to the matching point set, and finally maps the segmentation information to the frame to be matched based on the homography matrix to obtain the target image. The embodiment of the present application first calculates the homography matrix between the reference frame and the frame to be matched, and the homography matrix can be used to directly extract and propagate features from the video stream, so that the segmentation model results on the reference frame can be propagated to the frame to be matched, and the embodiment of the present application only needs to calculate the homography matrix to achieve the matching of frames in the video stream, without the need for multi-model parallel processing, which can reduce resource consumption, so that real-time applications with higher frame rates can be achieved under limited resource conditions, and the stability and accuracy of the entire system can be improved.

[0011] In some embodiments of the present application, the step of obtaining a reference frame and a frame to be matched of a video stream includes:

[0012] When the segmentation model is not called, the current frame of the video stream is used as a reference frame;

[0013] When the segmentation model is called, the current frame of the video stream is used as a frame to be matched.

[0014] In the above technical solution, the distinction between the reference frame and the matching frame is achieved by judging whether the current segmentation model is capable of processing the image frames in the video stream.

[0015] In some embodiments of the present application, mapping the segmentation information to the frame to be matched based on the homography matrix to obtain the target image includes:

[0016] For each pixel point on the reference frame, determining the pixel coordinates of the pixel point, and converting the pixel coordinates into homogeneous coordinates;

[0017] Mapping the homogeneous coordinates through the homography matrix, and performing coordinate transformation on the mapped homogeneous coordinates to obtain target coordinates;

[0018] The target coordinates and the segmentation information are fused to the frame to be matched to obtain a target image.

[0019] In the above technical solution, the target coordinates are used on the frame to be matched, and the segmentation information of the reference frame is retained to obtain the mapped segmentation result, which is then fused with the frame to be matched, thereby maintaining the spatial consistency between different frames.

[0020] In some embodiments of the present application, performing optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set includes:

[0021] Determine the feature points of the reference frame, and determine the positions corresponding to the feature points on the frame to be matched to obtain an optical flow field;

[0022] Based on the optical flow field, matching points corresponding to the feature points are determined in the frame to be matched to obtain a matching point set.

[0023] In the above technical solution, optical flow point matching is performed in the optical flow field, that is, matching points corresponding to feature points are determined in the frame to be matched based on the optical flow field to obtain a matching point set, thereby obtaining the corresponding relationship between the reference frame and the frame to be matched.

[0024] In some embodiments of the present application, determining a homography matrix according to the matching point set includes:

[0025] Determine a test point set in the matching point set based on a preset iterative algorithm;

[0026] Performing planar projection on the reference frame and the frame to be matched according to the test point set to obtain a homography matrix.

[0027] In the above technical solution, the reference frame and the frame to be matched are planarly projected according to the test point set to improve the accuracy of point matching and obtain a homography matrix, so as to determine the planar projection relationship from the reference frame to the frame to be matched.

[0028] In some embodiments of the present application, performing planar projection on the reference frame and the frame to be matched according to the test point set to obtain a homography matrix includes:

[0029] Fitting the test point set to obtain a test matrix;

[0030] Projecting all matching points in the matching point set based on the test matrix to obtain an inlier set;

[0031] The test matrix is ​​iteratively fitted according to the interior point set to obtain a homography matrix.

[0032] In the above technical solution, the test matrix is ​​iteratively fitted according to the number of inliers in the inlier set to obtain the homography matrix with the best effect, which is convenient for subsequently mapping the points in the reference frame to the frame to be matched through the homography matrix, and can propagate the deep learning model results on the processed frame to the unprocessed frame while maintaining the colinearity and coplanarity of the pixel points.

[0033] In some embodiments of the present application, determining matching points corresponding to the feature points in the to-be-matched frame based on the optical flow field to obtain a matching point set includes:

[0034] Determining matching points corresponding to the feature points in the to-be-matched frame to obtain multiple groups of matching point pairs;

[0035] Screening the matching point pairs;

[0036] Generate a matching point set based on the filtered matching point pairs.

[0037] In the above technical scheme, the embodiment of the present application also needs to screen the matching point pairs to verify the matching points, avoid the occurrence of incorrect pairing of matching points, and then generate a matching point set based on the screened matching point pairs, thereby reducing the situation of incorrect matching of pixel points, improving the matching accuracy, and facilitating subsequent image fusion.

[0038] In some embodiments of the present application, projecting all matching points in the matching point set based on the test matrix to obtain an inlier point set includes:

[0039] For each matching point in the matching point set, projecting the matching point according to the test matrix to obtain a test error of the matching point;

[0040] Comparing the test error with a preset error threshold;

[0041] All inliers whose test errors are smaller than the error threshold are screened out to obtain an inlier set.

[0042] In the above technical solution, all the inliers whose test errors are less than the error threshold are screened out to obtain an inlier set, thereby improving the fitting accuracy.

[0043] In some embodiments of the present application, before performing optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set, the method further includes:

[0044] Performing distortion correction on the reference frame and the frame to be matched;

[0045] The distortion-corrected reference frame and the frame to be matched are grayed out.

[0046] In the above technical solution, the error introduced by the lens distortion of the device is eliminated by performing distortion correction on the reference frame and the frame to be matched, thereby improving the image quality of the reference frame and the frame to be matched, and facilitating the subsequent improvement of the optical flow point matching accuracy. In addition, the computational complexity can be reduced by performing grayscale processing.

[0047] In a second aspect, an embodiment of the present application further provides a control device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image processing method as described in the first aspect is implemented.

[0048] In a third aspect, an embodiment of the present application further provides a mobile robot, comprising the control device as described in the second aspect.

[0049] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the image processing method as described in the first aspect.

[0050] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0052] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present application;

[0053] Figure 2 This is a flowchart of obtaining a reference frame and a frame to be matched of a video stream provided by an embodiment of the present application;

[0054] Figure 3 is a flowchart of mapping segmentation information to a frame to be matched based on a homography matrix provided by an embodiment of the present application;

[0055] Figure 4 This is a flowchart of performing optical flow point matching on a reference frame and a frame to be matched, provided by an embodiment of the present application;

[0056] Figure 5 is a flowchart of determining a homography matrix according to a matching point set provided by an embodiment of the present application;

[0057] Figure 6 is a flow chart of performing planar projection of a reference frame and a frame to be matched according to a test point set provided by an embodiment of the present application;

[0058] Figure 7 is a flowchart of determining a matching point corresponding to a feature point in a frame to be matched based on an optical flow field provided by an embodiment of the present application;

[0059] Figure 8 is a flow chart of obtaining an interior point set provided by an embodiment of the present application;

[0060] Fig. 9is a flowchart of an image processing method provided by another embodiment of the present application;

[0061] Fig.10 It is a schematic diagram of a control device of an image processing method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. In addition, the characteristics, operations or features described in the specification can be combined in any appropriate manner to form various implementation methods. At the same time, the steps or actions in the method description can also be replaced or adjusted in order in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the accompanying drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a necessary sequence, unless otherwise specified that a certain sequence must be followed.

[0063] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number itself, and "above", "below", "within", etc. are understood to include the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0064] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0065] In the description of this application, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

[0066] With the rapid development of computer vision and deep learning technologies, more and more application scenarios require complex image processing tasks, such as object detection, key point detection, and semantic / instance segmentation. These tasks are usually completed using different deep learning models that are trained specifically for specific problem areas. However, in many cases, such as mobile robots, drones, and intelligent monitoring systems, the resources of terminal devices are limited, including but not limited to computing power, memory capacity, and power. Therefore, in these scenarios, it becomes particularly important to efficiently utilize limited hardware resources.

[0067] In the case of multi-model parallel processing, resource consumption increases significantly, especially when the input data is provided at a high frame rate, a single or multiple models may not be able to process all input frames in time. In addition, real-time applications have high requirements for latency and accuracy. If these requirements are not met, the performance of the entire system will be affected. Therefore, a method is needed to address these challenges so that efficient real-time image processing can be achieved under limited resource conditions.

[0068] Based on the above situation, the embodiment of the present application provides an image processing method, a control device, a movable robot and a medium. The embodiment of the present application obtains a reference frame and a frame to be matched of a video stream, and obtains the segmentation information output after the reference frame is processed by a preset segmentation model, and then performs optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set, and then determines the homography matrix according to the matching point set, and finally maps the segmentation information to the frame to be matched based on the homography matrix to obtain a target image. The embodiment of the present application first calculates the homography matrix between the reference frame and the frame to be matched, and the homography matrix can be used to directly extract and propagate features from the video stream, so that the segmentation model results on the reference frame can be propagated to the frame to be matched, and the embodiment of the present application only needs to calculate the homography matrix to achieve the matching of frames in the video stream, without the need for multi-model parallel processing, which can reduce resource consumption, thereby being able to achieve real-time applications with higher frame rates under limited resource conditions, and improve the stability and accuracy of the entire system.

[0069] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0070] like Figure 1 As shown, Figure 1 1 is a flowchart of an image processing method provided by an embodiment of the present application. The image processing method of the embodiment of the present application includes but is not limited to steps S101 to S104.

[0071] Step S101: obtaining a reference frame and a frame to be matched of a video stream, and obtaining segmentation information output after the reference frame is processed by a preset segmentation model.

[0072] In some embodiments, a reference frame and a frame to be matched of a video stream are obtained, and the reference frame is input into a preset segmentation model for semantic segmentation or instance segmentation to obtain category information of each pixel on the reference frame, and the segmentation information output after the reference frame is processed by the segmentation model is obtained, so that a mask of each pixel on the reference frame can be obtained, and different objects and elements in the scene can be further identified.

[0073] It can be understood that the segmentation model in the embodiments of the present application can be a semantic segmentation mode or an instance segmentation model, for example, a fully convolutional network model, a high-resolution network model, a SegNet model, SOLO (Segmenting Objects by Locations), etc. This embodiment does not make specific restrictions.

[0074] It should be noted that the segmentation information in the embodiment of the present application is used to characterize the category information of each pixel in the reference frame, and is represented by a mask image of each pixel in the embodiment of the present application.

[0075] Step S102: performing optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set.

[0076] In some embodiments, the optical flow points between the reference frame and the frame to be matched are determined, and the optical flow points are matched to obtain a matching point set. The motion of objects in the scene can be estimated through the optical flow field, thereby obtaining the correspondence between the reference frame and the frame to be matched.

[0077] As can be understood, optical flow is used to estimate the motion of pixels between two consecutive video frames.

[0078] Step S103: Determine the homography matrix according to the matching point set.

[0079] In some embodiments, a homography matrix is ​​determined based on the matching point set, so that a planar projection relationship from the reference frame to the frame to be matched can be determined, which facilitates the subsequent propagation of segmentation information.

[0080] It should be noted that the homography matrix in the embodiment of the present application is used to describe the planar projection relationship between the reference frame and the frame to be matched.

[0081] Step S104: Map the segmentation information to the frame to be matched based on the homography matrix to obtain a target image.

[0082] In some embodiments, the segmentation information on the processed reference frame is mapped to the unprocessed frame to be matched through the homography matrix to obtain the target image, which not only reduces the resource consumption of multi-model parallel processing, but also achieves efficient matching between continuous video frames with low computational overhead.

[0083] It can be understood that the method of calculating the homography matrix is ​​a low-cost technology suitable for matching between consecutive video frames. It can represent the planar projection relationship between two pictures and has good matching characteristics between consecutive video frames. By calculating the homography matrix between adjacent frames, the results of the deep learning model on the processed frames can be propagated to the unprocessed frames, thereby improving the real-time performance of the entire system.

[0084] Reference Figure 2 , Figure 2 This is a flowchart of obtaining a reference frame and a frame to be matched of a video stream provided by an embodiment of the present application. The method includes but is not limited to steps S201 to S202.

[0085] Step S201: When the segmentation model is not called, the current frame of the video stream is used as a reference frame.

[0086] Step S202: When the segmentation model is called, the current frame of the video stream is used as a frame to be matched.

[0087] In steps S201 to S202 of some embodiments, in the process of obtaining the reference frame and the frame to be matched of the video stream, since the segmentation model may be called and cannot process the image frames in the video stream, the embodiments of the present application need to first determine whether the segmentation model is called, that is, determine whether the current segmentation model can process the image frames in the video stream. When the segmentation model is not called, it means that the segmentation model is not processing any image frames at this time, and the current frame of the video stream is used as the reference frame; when the segmentation model is called, it means that the segmentation model is processing other image frames at this time, and the current frame of the video stream is used as the frame to be matched.

[0088] Reference Figure 3 , Figure 3 This is a flowchart of mapping segmentation information to a frame to be matched based on a homography matrix provided by an embodiment of the present application. The method includes but is not limited to steps S301 to S303.

[0089] Step S301: for each pixel point on the reference frame, determine the pixel coordinates of the pixel point, and convert the pixel coordinates into homogeneous coordinates.

[0090] Step S302: Align the secondary coordinates through the homography matrix for mapping, and perform coordinate transformation on the mapped homogeneous coordinates to obtain target coordinates.

[0091] Step S303: merging the target coordinates and segmentation information into the frame to be matched to obtain a target image.

[0092] In step S301 to step S303 of some embodiments, in the process of mapping the segmentation information to the frame to be matched based on the homography matrix, for each pixel on the reference frame, the pixel coordinates are determined, and the pixel coordinates are converted into homogeneous coordinates, that is, the two-dimensional coordinates of the pixel are converted into three-dimensional coordinates, and then the homogeneous coordinates are mapped point by point through the homography matrix to obtain new homogeneous coordinates, and the mapped homogeneous coordinates are converted into coordinates, and the three-dimensional coordinates are converted into two-dimensional coordinates again to obtain the target coordinates, and finally, the target coordinates are used on the frame to be matched, and the segmentation information of the reference frame is retained to obtain the segmentation result after mapping, and then the segmentation result after mapping is fused with the frame to be matched to obtain the target image with segmentation information, so as to maintain the spatial consistency between different frames and reduce the accumulated error. The embodiment of the present application can realize real-time application with higher frame rate without significantly reducing performance by propagating the deep learning model results on the processed frame to the unprocessed frame. And in the case of multi-model parallel processing, the present application can reduce the consumption of computing and storage resources by matching using the homography matrix, and the computing demand is lower than other traditional methods, thereby improving the operation efficiency of the overall system.

[0093] Specifically, in the embodiment of the present application, the coordinates of each pixel point on the reference frame are (x, y), and the pixel coordinates are converted into homogeneous coordinates, expressed as (X, Y, Z), and then the homogeneous coordinates are mapped point by point using the homography matrix H to obtain new homogeneous coordinates (X', Y', Z'): (X', Y', Z') = H*(X, Y, Z), and then the mapped homogeneous coordinates (X', Y', Z') are converted back to two-dimensional coordinates to obtain the target coordinates (x', y'), that is, divided by the Z' coordinates: (x', y') = (X' / Z', Y' / Z'), and finally, the new target coordinates (x', y') are used on the frame to be matched, and the original mask value in the segmentation information is retained to generate the mapped segmentation result, and the mapped segmentation result is fused with the frame to be matched to obtain the target image.

[0094] It should be noted that the embodiment of the present application can use the homography matrix to directly extract and propagate features from the image data, avoiding the noise and cumulative errors caused by relying on the motion equation.

[0095] Reference Figure 4 , Figure 4 This is a flow chart of performing optical flow point matching on a reference frame and a frame to be matched provided by an embodiment of the present application. The method includes but is not limited to steps S401 to S402.

[0096] Step S401: determining feature points of the reference frame, and determining positions corresponding to the feature points on the frame to be matched, to obtain an optical flow field.

[0097] It should be noted that the optical flow field in the embodiment of the present application is a two-dimensional vector field, which is used to characterize the moving direction and speed of each feature point on the reference frame and the frame to be matched.

[0098] Step S402: determining matching points corresponding to feature points in the frame to be matched based on the optical flow field to obtain a matching point set.

[0099] In steps S401 to S402 of some embodiments, in the process of performing optical flow point matching on the reference frame and the frame to be matched, it is necessary to determine the optical flow points between the reference frame and the frame to be matched. Specifically, firstly detect significant corner points or edges in the reference frame to determine the feature points of the reference frame, so as to facilitate the subsequent tracking of the feature points on the reference frame, and determine the positions corresponding to the feature points on the matching frame to obtain an optical flow field, wherein the optical flow field is used to characterize the displacement of all points from the reference frame to the frame to be matched, so that the movement of objects in the scene can be estimated through the optical flow field. Finally, perform optical flow point matching in the optical flow field, that is, determine the matching points corresponding to the feature points in the frame to be matched based on the optical flow field to obtain a matching point set, so that the correspondence between the reference frame and the frame to be matched can be obtained.

[0100] It should be noted that, in the process of determining the position corresponding to the feature point on the frame to be matched, the embodiment of the present application can determine the corresponding position of each feature point on the reference frame on the frame to be matched through an optical flow estimation algorithm, thereby obtaining the displacement relationship of all points from the reference frame to the frame to be matched.

[0101] It is worth noting that after obtaining the matching point set, the embodiment of the present application can also perform post-processing on the matching point set, for example, improving the quality of the matching result through operations such as clustering, smoothing or filtering.

[0102] Reference Figure 5 , Figure 5 This is a flow chart of determining a homography matrix according to a matching point set provided by an embodiment of the present application. The method includes but is not limited to steps S501 to S502.

[0103] Step S501: determining a test point set in a matching point set based on a preset iterative algorithm.

[0104] Step S502: performing planar projection on the reference frame and the frame to be matched according to the test point set to obtain a homography matrix.

[0105] In steps S501 to S502 of some embodiments, in the process of determining the homography matrix based on the matching point set, first, a set of any size is randomly selected from the matching point set based on a preset iterative algorithm to obtain a test point set, and then the reference frame and the frame to be matched are planar projected according to the test point set to improve the accuracy of point matching and obtain the homography matrix, so as to determine the planar projection relationship from the reference frame to the frame to be matched.

[0106] It should be noted that the iterative algorithm in this embodiment can be a random sample consensus algorithm (RANSAC), a gradient descent algorithm, a conjugate gradient method, etc. In this embodiment, the RANSAC algorithm is used, and this embodiment does not make any specific restrictions on the choice of iterative algorithm.

[0107] Reference Figure 6 , Figure 6 This is a flow chart of performing planar projection on a reference frame and a frame to be matched according to a test point set provided by an embodiment of the present application. The method includes but is not limited to steps S601 to S603.

[0108] Step S601: Fit the test point set to obtain a test matrix.

[0109] Step S602: Project all matching points in the matching point set based on the test matrix to obtain an inlier point set.

[0110] Step S603: iteratively fit the test matrix according to the interior point set to obtain a homography matrix.

[0111] In steps S601 to S603 of some embodiments, in the process of planar projection of the reference frame and the frame to be matched, first, the test point set is fitted, and multiple non-coplanar matching points are randomly selected as the minimum point set, and the selected matching points can calculate a homography matrix, and then the selected minimum point set is linearly transformed to obtain a test matrix. Thereafter, each matching point in the matching point set is projected based on the test matrix, and the inner points and outer points in the matching point set are screened out to obtain an inner point set. Finally, the test matrix is ​​iteratively fitted according to the number of inner points in the inner point set to obtain the homography matrix with the best effect, so as to facilitate the subsequent mapping of points in the reference frame to the frame to be matched through the homography matrix, and the deep learning model results on the processed frame can be propagated to the unprocessed frame while maintaining the colinearity and coplanarity of the pixel points.

[0112] It is worth noting that in the process of iteratively testing the matrix according to the inlier set, the present embodiment first records the current number of inliers in the current inlier set, and then compares the historical number of inliers in the inlier set before the iteration with the current number of inliers. If the current number of inliers is greater than the number of inliers found in the previous iteration, the homography matrix is ​​refitted using the current inlier set, and the best fitting result is updated. After the iteration, the homography matrix H fitted using the largest number of inlier sets is the best estimate.

[0113] In this embodiment, an upper limit of the number of iterations may be set, or the iteration may be stopped when the number of inner point sets does not increase significantly in several consecutive iterations, thereby completing the entire iteration process.

[0114] It should be noted that, in this embodiment, after the homography matrix is ​​obtained, the homography matrix may be post-processed, for example, all interior points may be fitted more accurately by using the least square method.

[0115] Reference Figure 7 , Figure 7 This is a flowchart of determining matching points corresponding to feature points in a frame to be matched based on an optical flow field provided by an embodiment of the present application. The method includes but is not limited to steps S701 to S703.

[0116] Step S701: determining matching points corresponding to feature points in the frame to be matched, and obtaining multiple groups of matching point pairs.

[0117] Step S702: Screening matching point pairs.

[0118] Step S703: Generate a matching point set based on the screened matching point pairs.

[0119] In steps S701 to S703 of some embodiments, in the process of determining the matching points corresponding to the feature points in the frame to be matched based on the optical flow field, first, the matching points that match the descriptors of the feature points of the reference frame are searched in the frame to be matched, that is, the matching points corresponding to the feature points are determined, and multiple groups of matching point pairs are obtained. Each group of matching point pairs is used to characterize the correspondence between the feature points on the reference frame and the matching points on the frame to be matched. Due to possible matching errors or matching anomalies, the embodiments of the present application also need to screen the matching point pairs to verify the matching points and avoid the occurrence of incorrect pairing of matching points. Then, a matching point set is generated based on the screened matching point pairs, thereby reducing the situation of incorrect matching of pixel points, improving the accuracy of matching, and facilitating subsequent image fusion.

[0120] It should be noted that in the embodiment of the present application, in the process of determining the matching points corresponding to the feature points in the frame to be matched, the matching points can be determined by brute force search, nearest neighbor search, etc., and this embodiment does not make any specific restrictions.

[0121] It is worth noting that in the process of screening matching point pairs, outliers in the matching point pairs can be removed by detecting optical flow consistency and distance ratio tests, thereby ensuring the matching accuracy of pixels on the reference frame and the frame to be matched.

[0122] Reference Figure 8 , Figure 8 This is a flow chart of obtaining an interior point set provided by an embodiment of the present application. The method includes but is not limited to steps S801 to S803.

[0123] Step S801: For each matching point in the matching point set, project the matching point according to the test matrix to obtain a test error of the matching point.

[0124] Step S802: Compare the test error with a preset error threshold.

[0125] Step S803: Filter out all inliers whose test errors are less than the error threshold to obtain an inlier set.

[0126] In steps S801 to S803 of some embodiments, for each matching point in the matching point set, the matching point is projected according to the test matrix to obtain a test error of each matching point. The test error is then compared with a preset error threshold, and the points are classified into inliers and outliers, where inliers are points with smaller errors and outliers are points with larger errors. Finally, all inliers whose test errors are less than the error threshold are screened out to obtain an inlier set, thereby improving the fitting accuracy.

[0127] It should be noted that the error threshold in this embodiment can be set according to the model performance or the needs of the user, and this embodiment does not impose any specific limitation.

[0128] Reference Fig. 9 , Fig. 9 FIG. 1 is a flowchart of an image processing method provided by another embodiment of the present application. The method includes but is not limited to steps S901 to S902.

[0129] It should be noted that step S901 to step S902 occur before performing optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set.

[0130] Step S901: performing distortion correction on the reference frame and the frame to be matched.

[0131] In some embodiments, after obtaining the reference frame and the frame to be matched, errors introduced by lens distortion may occur. In order to avoid this situation, the embodiments of the present application will perform distortion correction on the reference frame and the frame to be matched. Specifically, the embodiments of the present application will determine the internal parameters of the device that collects the video stream, such as focal length, principal point, distortion coefficient, etc., and then select a distortion model based on the internal parameter data to describe the lens distortion of the device. Finally, each pixel point in the reference frame and the frame to be matched is inversely transformed by the distortion model and the internal parameter data, and the pixel points in the reference frame and the frame to be matched are mapped to a plane without distortion, completing the distortion correction process for the reference frame and the frame to be matched, thereby eliminating the error introduced by the lens distortion of the device, improving the image quality of the reference frame and the frame to be matched, and facilitating the subsequent improvement of the accuracy of optical flow point matching.

[0132] It should be noted that after mapping the pixel points in the reference frame and the frame to be matched to an undistorted plane, the embodiment of the present application will also resample the reference frame and the frame to be matched to obtain an undistorted image frame.

[0133] Step S902: grayscale processing is performed on the distortion-corrected reference frame and the frame to be matched.

[0134] In some embodiments, the embodiments of the present application convert the distortion-corrected reference frame and the frame to be matched into grayscale images to reduce the impact of color noise on the image processing algorithm, and because the grayscale image is a single-channel image, the computational complexity can be reduced.

[0135] It should be noted that the value of each pixel in the grayscale image represents the light intensity at that position. Compared with the color image, the grayscale image has only one channel, which greatly reduces the complexity of data processing and improves the processing speed. In addition, as the amount of data in the grayscale image is reduced, the amount of calculation of the algorithm is also reduced, and the storage space occupied is less than that of the color image, thereby enabling real-time processing of the reference frame and the frame to be matched when resources are limited.

[0136] In order to explain the above-mentioned image processing method, control device, movable robot and medium more clearly and clearly, specific examples are given below for explanation.

[0137] Example 1:

[0138] Example 1 takes the field of mobile robots as an example to illustrate the image processing method of an embodiment of the present application.

[0139] First, a series of continuous image frames are obtained from the video stream, and the current frame when the segmentation model is not called is used as the reference frame F1, and the current frame when the segmentation model is called is used as the frame to be matched F2. Then, the reference frame F1 is input into the semantic / instance segmentation model to obtain the segmentation result, where the segmentation result includes the category information of each pixel.

[0140] After obtaining the reference frame and the frame to be matched, errors introduced by lens distortion may occur. To avoid this situation, this example uses the camera internal parameters to perform distortion correction on the reference frame F1 and the frame to be matched F2 to eliminate the errors introduced by lens distortion. After that, the reference frame F1 and the frame to be matched F2 after distortion correction are converted into grayscale images to reduce the computational complexity.

[0141] After that, this example calculates the optical flow points between the reference frame F1 and the frame to be matched F2, and performs optical flow point matching to generate a matching point set. Based on the matching point set, a highly robust algorithm is used to calculate the homography matrix H, which describes the planar projection relationship from the reference frame F1 to the frame to be matched F2.

[0142] Finally, the segmentation result (i.e., mask image) on the reference frame F1 is mapped to the frame to be matched F2 using the homography matrix H, and the mapped segmentation result is fused with the frame to be matched F2 to form an image with segmentation information. The specific operations are as follows:

[0143] Step 1: Convert the coordinates (x, y) of each pixel point to homogeneous coordinates (X, Y, Z).

[0144] Step 2: Use the homography matrix H to map the homogeneous coordinates point by point to obtain new homogeneous coordinates (X', Y', Z'): (X', Y', Z') = H*(X, Y, Z).

[0145] Step 3: Convert the mapped homogeneous coordinates (X', Y', Z') back to two-dimensional coordinates (x', y'), that is, divide by the Z' coordinate: (x', y') = (X' / Z', Y' / Z').

[0146] Step 4: Use the new two-dimensional coordinates (x', y') on the frame to be matched F2 and retain the original mask value to generate the mapped segmentation result.

[0147] In this example, the results of the deep learning model on the processed frames are propagated to the unprocessed frames by calculating the homography matrix between adjacent frames. This enables real-time applications with higher frame rates without sacrificing performance. In addition, this method also takes into account camera distortion correction, further improving the accuracy of the projection results. And in real-time application scenarios such as mobile robots, the use of homography matrices can reduce the errors caused by the equation of motion and improve the stability and accuracy of the entire system.

[0148] like Fig.10 As shown, Fig.10 It is a schematic diagram of a control device provided in one embodiment of the present application.

[0149] The control device 1000 of the embodiment of the present application includes one or more processors 1001 and a memory 1002. Fig.10 In the figure, a processor 1001 and a memory 1002 are taken as an example.

[0150] The processor 1001 and the memory 1002 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0151] The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory 1002 remotely arranged relative to the processor 1001, and these remote memories may be connected to the control device 1000 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] Those skilled in the art will understand that Fig.10 The device structure shown in the figure does not constitute a limitation on the control device 1000, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0153] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network nodes. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0155] The non-transitory software program and instructions required to implement the image processing method of the above embodiment are stored in the memory, and when executed by the processor, the image processing method of the above embodiment is executed.

[0156] In some embodiments, the embodiments of the present application also provide a mobile robot. Since the mobile robot of the embodiments of the present application has the control device of the above-mentioned embodiments, and the control device of the above-mentioned embodiments can execute the image processing method of the above-mentioned embodiments, the specific implementation methods and technical effects of the mobile robot of the embodiments of the present application can refer to the specific implementation methods and technical effects of the image processing method of any of the above-mentioned embodiments.

[0157] The device embodiments or system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor or a controller.

[0159] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer-readable storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0160] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present application. These equivalent deformations or substitutions are all included in the scope defined by the present application.

Claims

1. An image processing method, characterized in that: include: Obtaining a reference frame and a frame to be matched of a video stream, and obtaining segmentation information output after the reference frame is processed by a preset segmentation model; Performing optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set; Determine a homography matrix according to the matching point set; The segmentation information is mapped to the frame to be matched based on the homography matrix to obtain a target image.

2. The image processing method according to claim 1, characterized in that: The step of obtaining a reference frame and a frame to be matched of a video stream includes: When the segmentation model is not called, the current frame of the video stream is used as a reference frame; When the segmentation model is called, the current frame of the video stream is used as a frame to be matched.

3. The image processing method according to claim 1, characterized in that: The mapping the segmentation information to the frame to be matched based on the homography matrix to obtain a target image includes: For each pixel point on the reference frame, determining the pixel coordinates of the pixel point, and converting the pixel coordinates into homogeneous coordinates; Mapping the homogeneous coordinates through the homography matrix, and performing coordinate transformation on the mapped homogeneous coordinates to obtain target coordinates; The target coordinates and the segmentation information are fused to the frame to be matched to obtain a target image.

4. The image processing method according to claim 1, characterized in that: The performing optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set includes: Determine the feature points of the reference frame, and determine the positions corresponding to the feature points on the frame to be matched to obtain an optical flow field; Based on the optical flow field, matching points corresponding to the feature points are determined in the frame to be matched to obtain a matching point set.

5. The image processing method according to claim 1, characterized in that: Determining a homography matrix according to the matching point set includes: Determine a test point set in the matching point set based on a preset iterative algorithm; Performing planar projection on the reference frame and the frame to be matched according to the test point set to obtain a homography matrix.

6. The image processing method according to claim 5, characterized in that: The performing planar projection on the reference frame and the frame to be matched according to the test point set to obtain a homography matrix includes: Fitting the test point set to obtain a test matrix; Projecting all matching points in the matching point set based on the test matrix to obtain an inlier set; The test matrix is ​​iteratively fitted according to the interior point set to obtain a homography matrix.

7. The image processing method according to claim 4, characterized in that: The step of determining matching points corresponding to the feature points in the frame to be matched based on the optical flow field to obtain a matching point set includes: Determining matching points corresponding to the feature points in the to-be-matched frame to obtain multiple groups of matching point pairs; Screening the matching point pairs; Generate a matching point set based on the filtered matching point pairs.

8. The image processing method according to claim 6, characterized in that: The projecting all matching points in the matching point set based on the test matrix to obtain an inner point set includes: For each matching point in the matching point set, projecting the matching point according to the test matrix to obtain a test error of the matching point; Comparing the test error with a preset error threshold; All inliers whose test errors are smaller than the error threshold are screened out to obtain an inlier set.

9. The image processing method according to claim 1, characterized in that: Before performing optical flow point matching on the reference frame and the frame to be matched to obtain a matching point set, the method further includes: Performing distortion correction on the reference frame and the frame to be matched; The distortion-corrected reference frame and the frame to be matched are grayed out.

10. A control device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image processing method according to any one of claims 1 to 9 when executing the computer program.

11. A mobile robot, characterized in that: The device comprises a camera and a control device as claimed in claim 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the image processing method according to any one of claims 1 to 9.

Citation Information

Cited By

  • Display screen brightness compensation method, device and system and storage medium

    CN120356444A

  • A display screen brightness compensation method, device, system and storage medium

    CN120356444B