Mobile equipment, threshold strip passing method and device thereof, electronic equipment and storage medium

Abstract

The invention provides a mobile device and a threshold strip passing method and device thereof, an electronic device and a storage medium, a threshold strip comprises a label, the mobile device comprises a visual component, and the method comprises the following steps: determining a label element in an image according to the image shot by the visual component; the label element is an element formed by the label in the image; determining the distance between the mobile device and the threshold strip according to the pixel value of the label element and the actual size of the label; under the condition that the distance is not greater than a target threshold value, determining multiple pieces of coordinate information of the label by utilizing multiple modes according to the label element; fusing the multiple pieces of coordinate information to obtain target coordinate information of the label; and according to the target coordinate information, determining a navigation path of the mobile device passing through the threshold strip.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile devices, and in particular to a mobile device, a method for a mobile device to pass through a door sill, a device for a mobile device to pass through a door sill, an electronic device, and a storage medium. Background Art

[0002] At present, when sweeping robots clean home environments, they often encounter environments with higher thresholds, which poses a challenge to the stability of home sweeping robots.

[0003] When the sweeping robot passes these thresholds, due to problems such as the passing angle, the robot will often get stuck and need corresponding accelerated escape movements to pass smoothly. When the robot is stuck, the cleaning time of the robot will be increased. When accelerating to pass the threshold, there is a possibility of colliding with other objects. Summary of the Invention

[0004] The present invention aims to at least solve or improve the technical problem in the prior art that a mobile device cannot accurately locate the position of a door sill, resulting in an inability to pass through the door sill smoothly.

[0005] To this end, a first aspect of the present invention provides a method for moving a mobile device over a threshold.

[0006] A second aspect of the present invention provides a device for moving a mobile device over a door sill.

[0007] A third aspect of the present invention provides an electronic device.

[0008] A fourth aspect of the present invention provides a storage medium.

[0009] A fifth aspect of the present invention provides a mobile device.

[0010] In view of this, according to a first aspect of the present invention, a method for a mobile device to pass through a threshold bar is proposed, wherein the threshold bar includes a label, and the mobile device includes a visual component. The method includes: determining the label element in the image based on the image captured by the visual component; the label element is the element formed by the label in the image; determining the distance between the mobile device and the threshold bar based on the pixel value of the label element and the actual size of the label; when the distance is not greater than the target threshold, determining multiple coordinate information of the label using multiple methods based on the label element; fusing the multiple coordinate information to obtain the target coordinate information of the label; and determining the navigation path of the mobile device through the threshold bar based on the target coordinate information.

[0011] The method for a mobile device to pass through a threshold bar proposed in the present invention is applicable to a mobile device, wherein a label is provided on the threshold bar. The mobile device includes a visual component, which is used to capture an image. When the visual component captures the label, the label is displayed as a label element in the image.

[0012] The method for a mobile device to pass through a threshold bar includes: determining a label element in an image captured by a visual component of the mobile device, calculating a pixel value of the label element, and since the label is determined, the size of the label is a known parameter. According to the principle of imaging, the closer the distance between the visual component and the label, the larger the pixel value of the label element, and the farther the distance between the visual component and the label, the smaller the pixel value of the label element. Therefore, the distance between the label and the visual component can be determined based on the size of the label and the pixel value of the label element. Since the position of the visual component on the mobile device can be determined, the position of the label on the threshold bar is determined, so the distance between the threshold bar and the mobile device can be determined based on the size of the label and the pixel value of the label element.

[0013] When the distance is not greater than the target distance, it means that the current mobile device is close to the threshold bar. In this case, multiple coordinate information of the label is determined in a variety of ways according to the label elements, and the multiple coordinate information is fused to obtain the target coordinate information of the label. Then, by using the fusion of multiple coordinate information, the accuracy of the target coordinate information of the label can be improved, the accuracy of the positioning of the threshold bar can be improved, and the possibility of the mobile device failing to pass the threshold bar can be reduced.

[0014] Based on the target coordinate information, the navigation path of the mobile device through the threshold bar is determined, thereby improving the success rate of the mobile device passing the threshold bar. In addition, the method has a simple positioning method for the threshold bar, simplifies the behavior of the mobile device passing the threshold bar, reduces the cost of users using the threshold bar, and enhances the stability and robustness of the mobile device in a home environment with high steps.

[0015] Furthermore, the method for a mobile device to pass through a threshold provided by the present invention determines the coordinate information of the tag only after the distance between the threshold and the mobile device is no greater than a target threshold, thereby reducing the processor's occupancy time and improving the stability of the mobile device's operation.

[0016] In addition, the method for passing a mobile device over a threshold in the above technical solution provided by the present invention may also have the following additional technical features:

[0017] In some embodiments, optionally, the step of determining the label element in the image based on the image captured by the visual component includes: extracting contour information of the element in the image; and determining the label element in the image based on the contour information and the shape of the label.

[0018] In this embodiment, the step of determining the label elements in the image based on the image captured by the visual component includes: extracting the contour information of the elements in the image, removing the lines that do not conform to the shape of the label, and retaining the lines that match the shape of the label, thereby obtaining the label elements in the image, and utilizing the contour information of the elements and the matching method with the label to improve the accuracy of label element recognition.

[0019] In some embodiments, optionally, the step of determining the distance between the mobile device and the threshold bar based on the pixel value of the tag element and the actual size of the tag includes: performing shape restoration on the tag element based on the actual size of the tag to obtain a target tag element; and determining the distance corresponding to the target tag element in the tag dictionary based on the target tag element.

[0020] In this embodiment, the step of determining the distance between the mobile device and the threshold bar based on the pixel value of the label element and the actual size of the label includes: restoring the shape of the label element based on the actual size of the label. Since the mobile device and the label on the threshold bar may not be directly opposite each other, and the size of the label element is different from the actual size of the label, the shape of the label element mounted on the label can be restored based on the perspective principle of the camera to obtain a target label element. Based on the target label element, the label dictionary is matched with the same label element to determine the distance corresponding to the target label element.

[0021] Among them, the label dictionary includes the pixel values ​​of different label elements and the corresponding distance between the mobile device and the threshold bar. Therefore, the label elements in the label dictionary can be matched according to the target label elements to determine the distance between the current mobile device and the threshold bar. This method is simple to determine, has low processor occupancy, and is highly accurate.

[0022] In some embodiments, optionally, when the distance is not greater than the target threshold, the steps of determining multiple coordinate information of the label using multiple methods based on the label elements include: determining the first coordinate information of the label in the coordinate system of the visual component based on the label elements using the inverse projection transformation method; determining the second coordinate information of the label in the coordinate system of the visual component based on the label elements using the perspective transformation method.

[0023] In this embodiment, when the distance is not greater than the target threshold, the steps of determining multiple coordinate information of the label in a plurality of ways according to the label elements include: determining the first coordinate information of the label in the coordinate system of the visual component according to the label elements by using the inverse projection transformation method, and the first coordinate information calculated by the inverse projection transformation method has a certain error due to the slight difference between the height of the threshold bar and the ground, but the error is robust and controllable; determining the second coordinate information of the label in the coordinate system of the visual component according to the label elements by using the perspective transformation method, and the perspective transformation method has high accuracy, but due to the instability of corner point recognition, the posture lacks robustness, so by fusing the first coordinate information obtained by the inverse projection transformation method and the second coordinate information obtained by the perspective transformation method, the error in the coordinate determination of the label can be reduced and the accuracy can be improved.

[0024] In some embodiments, optionally, the number of labels is multiple, and the method further includes: determining whether multiple label elements are blocked; if multiple label elements are blocked, determining multiple corner points of the multiple label elements; and merging the multiple label elements into one label element based on the multiple corner points.

[0025] In this embodiment, there are multiple labels, so there may be multiple label elements in the visually captured image. Since the labels may be blocked by other objects, the method for moving a mobile device over the threshold bar also includes: determining whether multiple label elements are blocked, and when multiple label elements are blocked, determining multiple corner points of the multiple label elements, and merging the multiple label elements into one label element. Since the setting positions of the multiple labels are also fixed, the multiple labels can be regarded as one label. By identifying the corner points of the label elements, a larger label element can be constructed based on the corner points of different label elements, thereby further improving the success rate of the mobile device passing the threshold bar.

[0026] In some embodiments, optionally, the method further includes: updating the navigation path when the distance changes.

[0027] In this embodiment, after the mobile device moves according to the navigation path, the distance between the mobile device and the threshold bar changes, and the label photographed by the visual component will be clearer. Therefore, the determined label element will be more accurate. Therefore, according to the change in distance, the navigation path is updated, and the navigation path can be gradually corrected to improve the success rate of the mobile device passing the threshold bar.

[0028] According to a second aspect of the present invention, the present invention proposes a device for a mobile device to pass through a threshold bar, wherein the threshold bar includes a label, and the mobile device includes a visual component. The device includes: a first determination module, used to determine the label element in the image based on the image taken by the visual component; the label element is the element formed by the label in the image; a second determination module, used to determine the distance between the mobile device and the threshold bar based on the pixel value of the label element and the actual size of the label; a third determination module, used to determine multiple coordinate information of the label using multiple methods based on the label element when the distance is not greater than the target threshold; a fusion module, used to fuse the multiple coordinate information to obtain the target coordinate information of the label; and a fourth determination module, used to determine the navigation path of the mobile device through the threshold bar based on the target coordinate information.

[0029] The mobile device threshold bar device proposed in the present invention is suitable for mobile devices, a label is set on the threshold bar, and the mobile device includes a visual component, which is used to capture images. When the visual component captures the label, the label is displayed as a label element in the image.

[0030] The device for allowing a mobile device to pass through a threshold bar includes: determining a label element in an image captured by a visual component of the mobile device, calculating a pixel value of the label element, and since the label is determined, the size of the label is a known parameter. According to the principle of imaging, the closer the distance between the visual component and the label, the larger the pixel value of the label element, and the farther the distance between the visual component and the label, the smaller the pixel value of the label element. Therefore, the distance between the label and the visual component can be determined based on the size of the label and the pixel value of the label element. Since the position of the visual component on the mobile device can be determined, the position of the label on the threshold bar is determined, so the distance between the threshold bar and the mobile device can be determined based on the size of the label and the pixel value of the label element.

[0031] When the distance is not greater than the target distance, it means that the current mobile device is close to the threshold bar. In this case, multiple coordinate information of the label is determined in a variety of ways according to the label elements, and the multiple coordinate information is fused to obtain the target coordinate information of the label. Then, by using the fusion of multiple coordinate information, the accuracy of the target coordinate information of the label can be improved, the accuracy of the positioning of the threshold bar can be improved, and the possibility of the mobile device failing to pass the threshold bar can be reduced.

[0032] Based on the target coordinate information, the navigation path of the mobile device through the threshold bar is determined, thereby improving the success rate of the mobile device passing the threshold bar. In addition, the device has a simple positioning method for the threshold bar, which simplifies the behavior of the mobile device passing the threshold bar, reduces the cost of users using the threshold bar, and enhances the stability and robustness of the mobile device in a home environment with high steps.

[0033] Furthermore, the device for allowing a mobile device to pass through a threshold provided by the present invention determines the coordinate information of the tag only after the distance between the threshold and the mobile device is no greater than a target threshold, thereby reducing the processor's occupancy time and improving the stability of the mobile device's operation.

[0034] According to the third aspect of the present invention, the present invention proposes an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the method for a mobile device to pass a threshold bar proposed in the embodiment of the first aspect is implemented.

[0035] The electronic device proposed in the present invention includes a processor and a memory. The memory stores programs or instructions that can be run on the processor. When the programs or instructions are executed by the processor, the method for a mobile device to pass through a threshold bar proposed in the embodiment of the first aspect is implemented. Therefore, the electronic device proposed in the present invention has all the beneficial effects of the method for a mobile device to pass through a threshold bar proposed in the embodiment of the first aspect, which will not be listed one by one here.

[0036] According to a fourth aspect of the present invention, the present invention proposes a storage medium storing a program or instruction. When the program or instruction is executed by a processor, the method for a mobile device to pass a threshold bar as proposed in the embodiment of the first aspect is implemented.

[0037] The storage medium proposed in the present invention stores a program or instruction. When the program or instruction is executed by the processor, the method for a mobile device to pass through a threshold bar proposed in the first aspect embodiment is implemented. Therefore, the storage medium proposed in the present invention has all the beneficial effects of the method for a mobile device to pass through a threshold bar proposed in the first aspect embodiment, which will not be listed one by one here.

[0038] According to the fifth aspect of the present invention, the present invention proposes a mobile device, including: a mobile device threshold bar device as proposed in the second aspect embodiment; or an electronic device as proposed in the third aspect embodiment; or a storage medium as proposed in the fourth aspect embodiment. Therefore, all the beneficial effects of the mobile device threshold bar device as proposed in the second aspect embodiment; or the electronic device as proposed in the third aspect embodiment; or the storage medium as proposed in the fourth aspect embodiment are not listed one by one here.

[0039] Additional aspects and advantages of the invention will become apparent from the description which follows, or may be learned by practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0041] Figure 1A flowchart of a method for a mobile device to pass a threshold provided by an embodiment of the present invention is shown;

[0042] Figure 2 A second flowchart of a method for a mobile device to pass a threshold provided by an embodiment of the present invention is shown;

[0043] Figure 3 A structural block diagram of a device for passing a threshold strip for a mobile device provided by an embodiment of the present invention is shown;

[0044] Figure 4 A structural block diagram of an electronic device provided by an embodiment of the present invention is shown;

[0045] Figure 5 A schematic structural diagram of a mobile device provided by an embodiment of the present invention is shown;

[0046] Figure 6 A schematic diagram of the structure of a threshold bar in a method for a mobile device to pass through a threshold bar provided by an embodiment of the present invention is shown;

[0047] Figure 7 Shown as Figure 6 A cross-sectional view of the door sill taken along the AA direction is shown.

[0048] in, Figures 5 to 7 The corresponding relationship between the reference numerals and component names is as follows:

[0049] 500 mobile devices, 510 visual components, 600 threshold strips, 610 labels. DETAILED DESCRIPTION

[0050] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0052] Refer to the following Figures 1 to 7 The following describes a method for a mobile device to pass through a threshold bar, an apparatus for a mobile device to pass through a threshold bar, an electronic device, a storage medium, and a mobile device according to some embodiments of the present invention.

[0053] The method for a mobile device to pass a threshold bar provided by the present invention is used for a mobile device to pass a threshold bar, wherein, Figure 5As shown, the mobile device 500 includes a visual component 510, which can be used for shooting. The mobile device 500 can be a sweeping robot, a floor cleaning robot, or a logistics robot. The visual component 510 can be a monocular camera or other cameras. Figure 6 and Figure 7 As shown, a label 610 is provided on the threshold bar 600, and the mobile device 500 can use the threshold bar 600 to climb stairs, pass through grooves, or other obstacles. The label 610 can be at least one, such as one, two, three, four, or five, and the front of the label 610 can be a polygonal shape, such as a rectangle or a square.

[0054] Figure 1 FIG. 1 shows one of the flow charts of a method for a mobile device to pass a threshold provided by an embodiment of the present invention. Figure 1 As shown, the specific process of the method for a mobile device to pass through a threshold provided by one embodiment of the present invention is as follows:

[0055] Step 102: determining label elements in the image based on the image captured by the visual component; the label elements are elements formed by the label in the image;

[0056] Step 104: Determine the distance between the mobile device and the threshold bar based on the pixel value of the label element and the actual size of the label;

[0057] Step 106: When the distance is not greater than the target threshold, determine multiple coordinate information of the tag using multiple methods according to the tag elements;

[0058] Step 108: Fusing multiple coordinate information to obtain target coordinate information of the tag;

[0059] Step 110: Determine a navigation path for the mobile device to pass through the threshold according to the target coordinate information.

[0060] The method for a mobile device to pass through a threshold provided by the present invention is applicable to a mobile device, wherein a label is provided on the threshold. The mobile device includes a visual component, which is used to capture an image. When the visual component captures the label, the label is displayed as a label element in the image.

[0061] The method for a mobile device to pass through a threshold bar includes: determining a label element in an image captured by a visual component of the mobile device, calculating a pixel value of the label element, and since the label is determined, the size of the label is a known parameter. According to the principle of imaging, the closer the distance between the visual component and the label, the larger the pixel value of the label element, and the farther the distance between the visual component and the label, the smaller the pixel value of the label element. Therefore, the distance between the label and the visual component can be determined based on the size of the label and the pixel value of the label element. Since the position of the visual component on the mobile device can be determined, the position of the label on the threshold bar is determined, so the distance between the threshold bar and the mobile device can be determined based on the size of the label and the pixel value of the label element.

[0062] When the distance is not greater than the target distance, it means that the current mobile device is close to the threshold bar. In this case, multiple coordinate information of the label is determined in a variety of ways according to the label elements, and the multiple coordinate information is fused to obtain the target coordinate information of the label. Then, by using the fusion of multiple coordinate information, the accuracy of the target coordinate information of the label can be improved, the accuracy of the positioning of the threshold bar can be improved, and the possibility of the mobile device failing to pass the threshold bar can be reduced.

[0063] Based on the target coordinate information, the navigation path of the mobile device through the threshold bar is determined, thereby improving the success rate of the mobile device passing the threshold bar. In addition, the method has a simple positioning method for the threshold bar, simplifies the behavior of the mobile device passing the threshold bar, reduces the cost of users using the threshold bar, and enhances the stability and robustness of the mobile device in a home environment with high steps.

[0064] Furthermore, the method for a mobile device to pass through a threshold provided by the present invention determines the coordinate information of the tag only after the distance between the threshold and the mobile device is no greater than a target threshold, thereby reducing the processor's occupancy time and improving the stability of the mobile device's operation.

[0065] In some embodiments, optionally, the step of determining the label element in the image based on the image captured by the visual component includes: extracting contour information of the element in the image; and determining the label element in the image based on the contour information and the shape of the label.

[0066] In this embodiment, the step of determining the label elements in the image based on the image captured by the visual component includes: extracting the contour information of the elements in the image, removing the lines that do not conform to the shape of the label, and retaining the lines that match the shape of the label, thereby obtaining the label elements in the image, and utilizing the contour information of the elements and the shape matching with the label to improve the accuracy of label element recognition.

[0067] Specifically, the RGB image taken by the visual component is obtained, the RGB image is converted into a grayscale image, the edge information of the image is extracted using an edge extraction algorithm, the contour information of the image is extracted using a contour extraction algorithm, the extracted contour information is fitted using a polygon fitting algorithm, and the polygons are screened and eliminated according to the distance between the shape of the label and the contour to obtain the four corner points of the label element in the image, thereby determining the label element.

[0068] In some embodiments, optionally, the step of determining the distance between the mobile device and the threshold bar based on the pixel value of the tag element and the actual size of the tag includes: performing shape restoration on the tag element based on the actual size of the tag to obtain a target tag element; and determining the distance corresponding to the target tag element in the tag dictionary based on the target tag element.

[0069] In this embodiment, the step of determining the distance between the mobile device and the threshold bar based on the pixel value of the label element and the actual size of the label includes: restoring the shape of the label element based on the actual size of the label. Since the mobile device and the label on the threshold bar may not be directly opposite each other, the shape of the label element can be restored based on the perspective principle of the camera to obtain a target label element. Based on the target label element, the label element with the same label is matched in the label dictionary to determine the distance corresponding to the target label element.

[0070] Among them, the label dictionary includes the pixel values ​​of different label elements and the corresponding distance between the mobile device and the threshold bar. Therefore, the label elements in the label dictionary can be matched according to the target label elements to determine the distance between the current mobile device and the threshold bar. This method is simple to determine, has low processor occupancy, and is highly accurate.

[0071] Specifically, the label dictionary is pre-generated and can automatically generate the most recognizable label dictionary based on the size of the label on the threshold bar and the resolution of the visual component on the mobile device, and set the labels corresponding to the label dictionary on the threshold bar in a predetermined position order.

[0072] Moreover, when generating the label dictionary, it is necessary to simultaneously calculate the range of the target threshold in the label dictionary, and take half of the maximum distance as the fuzzy target threshold for the recognition distance in the subsequent step.

[0073] The optimal label dictionary is generated by maximizing the distance corresponding to the label and maximizing the bit change of the label.

[0074] Finally, three optimal labels are generated as labels set on the threshold bar, so that there is a certain redundancy when the navigation points of the mobile device are subsequently calculated.

[0075] Taking the threshold with three labels as an example, the center points of the three labels are calculated for subsequent use of the inverse projection transformation method.

[0076] When matching tag elements, red-black trees can be used for matching, and label rotation must also be considered.

[0077] To restore the shape of the tag elements, we apply camera imaging principles to align the tag elements in the image, transforming their shape to obtain Ncm × Mcm tag element information with true scale. After image binarization, we search against a previously generated tag dictionary to calculate whether the distance between the tag and the mobile device meets the target threshold. N and M can be the same or different; N can take values ​​of 1, 2, 3, 4, or 5, and M can take values ​​of 1, 2, 3, 4, or 5. N represents the actual length of the tag, and M represents the actual width of the tag.

[0078] Among them, after the shape of the label element is restored, it can be searched and matched in the label dictionary according to the pixel value within the period.

[0079] In some embodiments, optionally, when the distance is not greater than the target threshold, the steps of determining multiple coordinate information of the label using multiple methods based on the label elements include: determining the first coordinate information of the label in the coordinate system of the visual component based on the label elements using the inverse projection transformation method; determining the second coordinate information of the label in the coordinate system of the visual component based on the label elements using the perspective transformation method.

[0080] In this embodiment, when the distance is not greater than the target threshold, the steps of determining multiple coordinate information of the label in a plurality of ways according to the label elements include: determining the first coordinate information of the label in the coordinate system of the visual component according to the label elements by using the inverse projection transformation method, and the first coordinate information calculated by the inverse projection transformation method has a certain error due to the slight difference between the height of the threshold bar and the ground, but the error is robust and controllable; determining the second coordinate information of the label in the coordinate system of the visual component according to the label elements by using the perspective transformation method, and the perspective transformation method has high accuracy, but due to the instability of corner point recognition, the posture lacks robustness, so by fusing the first coordinate information obtained by the inverse projection transformation method and the second coordinate information obtained by the perspective transformation method, the error in the coordinate determination of the label can be reduced and the accuracy can be improved.

[0081] Specifically, after matching the label elements in the label dictionary, the pixel values ​​of the corner points of the label elements are determined, the coordinate transformation between the plane where the label is located and the plane where the visual component is located is calculated, and the world coordinates of the threshold bar label are obtained according to the current posture of the mobile device.

[0082] In some embodiments, optionally, the number of labels is multiple, and the method further includes: determining whether multiple label elements are blocked; if multiple label elements are blocked, determining multiple corner points of the multiple label elements; and merging the multiple label elements into one label element based on the multiple corner points.

[0083] In this embodiment, there are multiple labels, so there may be multiple label elements in the visually captured image. Since the labels may be blocked by other objects, the method for moving a mobile device over the threshold bar also includes: determining whether multiple label elements are blocked, and when multiple label elements are blocked, determining multiple corner points of the multiple label elements, and merging the multiple label elements into one label element. Since the setting positions of the multiple labels are also fixed, the multiple labels can be regarded as one label. By identifying the corner points of the label elements, a larger label element can be constructed based on the corner points of different label elements, thereby further improving the success rate of the mobile device passing the threshold bar.

[0084] Specifically, the coordinate information obtained by the two algorithms is fused to ultimately determine the precise location of the label on the threshold bar. The navigation point of the mobile device is calculated based on the distance between the mobile device and the threshold bar.

[0085] Taking three tags as an example, when the tags are blocked, the three tags are treated as a whole. As long as the vision component captures 4 of the 12 corner points of the three tags, the distance between the threshold bar and the mobile device can be determined.

[0086] In some embodiments, optionally, the method further includes: updating the navigation path when the distance changes.

[0087] In this embodiment, after the mobile device moves according to the navigation path, the distance between the mobile device and the threshold bar changes, and the label photographed by the visual component will be clearer. Therefore, the determined label element will be more accurate. Therefore, according to the change in distance, the navigation path is updated, and the navigation path can be gradually corrected to improve the success rate of the mobile device passing the threshold bar.

[0088] That is, the distance of the navigation point of the mobile device is dynamically changed according to the distance between the mobile device and the threshold bar.

[0089] Figure 2 FIG2 shows a second flow chart of a method for a mobile device to pass a threshold provided by an embodiment of the present invention. Figure 2 As shown, the specific process of the method for a mobile device to pass through a threshold provided by one embodiment of the present invention is as follows:

[0090] Step 202: Generate a tag dictionary offline;

[0091] Step 204: Acquire an image captured by a visual component of the mobile device;

[0092] Step 206: grayscale the image and extract edges;

[0093] Step 208: extracting contour information;

[0094] Step 210: Restore the graphic of the label element using a polygon fitting algorithm and the four corner points of the label element;

[0095] Step 212: Binarize and enhance the image, match it in the label dictionary, and verify the distance;

[0096] Step 214: Determine the first coordinate information of the tag in the visual component coordinate system according to the inverse projection transformation method;

[0097] Step 216: Determine the second coordinate information of the label in the visual component coordinate system according to the perspective transformation method;

[0098] Step 218: Obtain the current position of the mobile device;

[0099] Step 220: Fusion of poses to obtain the position information of the tag in the world coordinate system;

[0100] Step 222: The navigation point of the mobile device is calculated and updated, and the mobile device moves to the navigation point.

[0101] Specifically, a label dictionary of the three most recognizable labels is generated offline, and a target threshold for label recognition is calculated. Images captured by the mobile device are captured in real time, converted to grayscale, and then edge and contour information are extracted. Polygons are then fitted, and duplicate polygons with significantly different shapes are removed. Finally, the pixel values ​​of the corner points of the three labels are obtained. Deformation caused by camera imaging is restored based on the pixel values ​​of the corner points and the actual scale of the labels. The image is then binarized, and the distance between each label element and its corresponding label element in the label dictionary is calculated, ensuring it is less than or equal to the target threshold. Based on the pixel values ​​of the three labels, two methods are used to calculate the pose of the labels in the visual component's coordinate system. One method uses a backprojection transformation based on a ground assumption, and the other uses a perspective transformation of the camera based on the known label size. This method leverages the redundant information of the three labels to address label occlusion. The two pose results are fused to determine the final pose of the threshold bar in the camera coordinate system. The pose of the threshold bar in the world coordinate system is then determined based on the pose of the mobile device in the world coordinate system. The navigation point and angle of the mobile device's movement are calculated using the three labels, and the threshold bar's movement behavior is then detected.

[0102] Optionally, three tags are set on the threshold, and different shooting states of the tags are combined into a tag dictionary and placed at fixed positions on the threshold with fixed size intervals for position and angle recognition of the threshold.

[0103] Optionally, the target threshold range is set to one third of the maximum distance in the tag dictionary for subsequent recognition verification to improve the stability of recognition.

[0104] Optionally, the image recognition of the mobile device runs in the background at a lower frequency, and when the threshold bar is captured in the image, the input frequency of the image is increased to real time.

[0105] Optionally, the specific size of the label is a 2 cm-shaped structure, the interval between adjacent labels is 2 cm, and the label is set at a lower center position of the door sill.

[0106] Optionally, the coordinate pose information of the tag in the vision component is calculated by fusing two methods. The inverse projection method has a certain error in the calculated distance due to the slight difference between the threshold bar and the ground, but the error is robust and controllable. The perspective transformation method is highly accurate, but lacks robustness due to the instability of corner point recognition. Kalman fusion of the observations from these two methods enhances the stability of threshold bar recognition.

[0107] Optionally, the navigation point of the mobile device is calculated by taking the perpendicular bisector of the three labels, and the distance between the navigation point and the mobile device is positively correlated with the distance between the mobile device and the threshold bar.

[0108] Optionally, since the closer the mobile device is to the threshold, the clearer the pixels of the image and the label elements are, a coarse-first-fine-later navigation method is adopted to ensure the mobile device's ability to accurately cross the threshold.

[0109] The present invention provides a method for enabling a mobile device to pass through a threshold using visual tags. The method employs image recognition, specifically offline generation of a threshold tag dictionary, real-time identification of tag elements in an image, and extraction of their corner points. The tag's position information is calculated using inverse projection and perspective transformations, and the mobile device's navigation point and angle are calculated to enable the mobile device to pass through the threshold smoothly. This method utilizes traditional visual methods, enabling the mobile device to successfully locate the threshold using a monocular camera. Subsequently, the mobile device only needs a threshold slightly wider than the width of the mobile device to complete the threshold crossing task, thereby enhancing the stability and overall intelligence of the mobile device during the threshold crossing process.

[0110] The above methods may be implemented in various ways depending on the specific features and / or example applications. For example, these methods may be implemented through a combination of hardware, firmware, and / or software. For example, in a hardware implementation, the processor may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, electronic devices, other device units for performing the above functions, and / or combinations thereof.

[0111] like Figure 3 As shown, according to the second aspect of the present invention, the present invention provides a device 300 for a mobile device to pass through a threshold bar, the threshold bar includes a label, the mobile device includes a visual component, and the device includes: a first determination module 302, used to determine the label element in the image based on the image taken by the visual component; the label element is the element formed by the label in the image; a second determination module 304, used to determine the distance between the mobile device and the threshold bar based on the pixel value of the label element and the actual size of the label; a third determination module 306, used to determine multiple coordinate information of the label in a plurality of ways based on the label element when the distance is not greater than the target threshold; a fusion module 308, used to fuse the multiple coordinate information to obtain the target coordinate information of the label; a fourth determination module 310, used to determine the navigation path of the mobile device through the threshold bar based on the target coordinate information.

[0112] The mobile device threshold bar device provided by the present invention is suitable for mobile devices, a label is set on the threshold bar, and the mobile device includes a visual component, which is used to capture images. When the visual component captures the label, the label is displayed as a label element in the image.

[0113] The device for allowing a mobile device to pass through a threshold bar includes: determining a label element in an image captured by a visual component of the mobile device, calculating a pixel value of the label element, and since the label is determined, the size of the label is a known parameter. According to the principle of imaging, the closer the distance between the visual component and the label, the larger the pixel value of the label element, and the farther the distance between the visual component and the label, the smaller the pixel value of the label element. Therefore, the distance between the label and the visual component can be determined based on the size of the label and the pixel value of the label element. Since the position of the visual component on the mobile device can be determined, the position of the label on the threshold bar is determined, so the distance between the threshold bar and the mobile device can be determined based on the size of the label and the pixel value of the label element.

[0114] When the distance is not greater than the target distance, it means that the current mobile device is close to the threshold bar. In this case, multiple coordinate information of the label is determined in a variety of ways according to the label elements, and the multiple coordinate information is fused to obtain the target coordinate information of the label. Then, by using the fusion of multiple coordinate information, the accuracy of the target coordinate information of the label can be improved, the accuracy of the positioning of the threshold bar can be improved, and the possibility of the mobile device failing to pass the threshold bar can be reduced.

[0115] Based on the target coordinate information, the navigation path of the mobile device through the threshold bar is determined, thereby improving the success rate of the mobile device passing the threshold bar. In addition, the device has a simple positioning method for the threshold bar, which simplifies the behavior of the mobile device passing the threshold bar, reduces the cost of users using the threshold bar, and enhances the stability and robustness of the mobile device in a home environment with high steps.

[0116] Furthermore, the device for allowing a mobile device to pass through a threshold provided by the present invention determines the coordinate information of the tag only after the distance between the threshold and the mobile device is no greater than a target threshold, thereby reducing the processor's occupancy time and improving the stability of the mobile device's operation.

[0117] In some embodiments, optionally, the first determination module includes: an extraction submodule for extracting contour information of elements in the image; and a first determination submodule for determining the label element in the image according to the contour information and the shape of the label.

[0118] In some embodiments, optionally, the second determination module includes: a restoration submodule, used to restore the shape of the label element according to the shape of the label to obtain the target label element; and a second determination submodule, used to determine the distance corresponding to the target label element in the label dictionary according to the target label element.

[0119] In some embodiments, optionally, the third determination module includes: a third determination submodule, used to determine the first coordinate information of the label in the coordinate system of the visual component based on the label element using the inverse projection transformation method; a fourth determination submodule, used to determine the second coordinate information of the label in the coordinate system of the visual component based on the label element using the perspective transformation method.

[0120] In some embodiments, optionally, the number of labels is multiple, and the device also includes: a fifth determination module, used to determine whether multiple label elements are obscured; a sixth determination module, used to determine multiple corner points of multiple label elements when multiple label elements are obscured; and a merging module, used to merge multiple label elements into one label element based on multiple corner points.

[0121] In some embodiments, optionally, it further includes: an updating module, configured to update the navigation path when the distance changes.

[0122] like Figure 4 As shown, according to the third aspect of the present invention, the present invention provides an electronic device 400, including a processor 402, a memory 404, and a program or instruction stored in the memory 404 and executable on the processor 402. When the program or instruction is executed by the processor 402, the method for a mobile device to pass a threshold bar proposed in the embodiment of the first aspect is implemented.

[0123] The electronic device provided by the present invention includes a processor and a memory. The memory stores programs or instructions that can be run on the processor. When the programs or instructions are executed by the processor, the method for a mobile device to pass through a threshold bar proposed in the embodiment of the first aspect is implemented. Therefore, the electronic device proposed by the present invention has all the beneficial effects of the method for a mobile device to pass through a threshold bar proposed in the embodiment of the first aspect, which will not be listed one by one here.

[0124] According to a fourth aspect of the present invention, the present invention provides a storage medium storing a program or instruction. When the program or instruction is executed by a processor, the method for a mobile device to pass a threshold bar provided in the embodiment of the first aspect is implemented.

[0125] The storage medium provided by the present invention stores a program or instruction. When the program or instruction is executed by the processor, the method for a mobile device to pass through a threshold bar provided in the first aspect embodiment is implemented. Therefore, the storage medium proposed by the present invention has all the beneficial effects of the method for a mobile device to pass through a threshold bar provided in the first aspect embodiment, which will not be listed one by one here.

[0126] The storage medium may be a readable storage medium.

[0127] A storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. A computer storage medium can be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer storage media includes: a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory card, a floppy disk, an encoding mechanical device (such as a punched card or a groove with a raised structure on which instructions are recorded), and any suitable combination of the foregoing. As used herein, computer storage medium should not be understood as a transmission signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium, or electrical signals transmitted through wires.

[0128] like Figure 5As shown, according to the fifth aspect of the present invention, the present invention provides a mobile device 500, including: the mobile device threshold bar device proposed in the second aspect embodiment; or the electronic device proposed in the third aspect embodiment; or the storage medium proposed in the fourth aspect embodiment. Therefore, it has all the beneficial effects of the mobile device threshold bar device proposed in the second aspect embodiment; or the electronic device proposed in the third aspect embodiment; or the storage medium proposed in the fourth aspect embodiment, which are no longer stated one by one here.

[0129] Specifically, the mobile device 500 includes a visual component 510 , and the visual component 510 can photograph the door sill.

[0130] In the present invention, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "plurality" refers to two or more, unless expressly limited otherwise. Terms such as "installed," "connected," "connected," and "fixed" should be interpreted broadly. For example, "connected" can mean a fixed connection, a detachable connection, or an integral connection; "connected" can mean a direct connection or an indirect connection through an intermediary. Those skilled in the art will understand the specific meanings of these terms in the present invention based on specific circumstances.

[0131] In the description of the present invention, it should be understood that the terms "up", "down", "left", "right", "front", "back", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the components or units referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they should not be understood as limiting the present invention.

[0132] Throughout this specification, terms such as "one embodiment," "some embodiments," and "specific embodiments" mean that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0133] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for moving a mobile device over a threshold, characterized in that: The threshold strip includes a label, the mobile device includes a visual component, and the method includes: Determining, based on an image captured by the visual component, a label element in the image; the label element is an element formed by the label in the image; determining a distance between the mobile device and the threshold bar according to a pixel value of the label element and an actual size of the label; When the distance is not greater than a target threshold, determining multiple coordinate information of the tag using multiple methods according to the tag element; Merging the plurality of coordinate information to obtain target coordinate information of the tag; A navigation path of the mobile device passing through the threshold is determined according to the target coordinate information.

2. The method for passing a mobile device through a threshold according to claim 1, characterized in that: The step of determining the label element in the image based on the image captured by the visual component includes: Extracting contour information of elements in the image; The label element in the image is determined according to the contour information and the shape of the label.

3. The method for moving a mobile device over a threshold according to claim 1 or 2, characterized in that: The step of determining the distance between the mobile device and the threshold bar according to the pixel value of the label element and the actual size of the label includes: Performing shape restoration on the label element according to the actual size of the label to obtain a target label element; According to the target tag element, the distance corresponding to the target tag element is determined in a tag dictionary.

4. The method for moving a mobile device over a threshold according to claim 1 or 2, characterized in that: The step of determining the plurality of coordinate information of the tag using a plurality of methods according to the tag element when the distance is not greater than the target threshold includes: Determine, according to the label element, first coordinate information of the label in the coordinate system of the visual component using an inverse projection transformation method; According to the label element, the second coordinate information of the label in the coordinate system of the visual component is determined by using a perspective transformation method.

5. The method for moving a mobile device over a threshold according to claim 1 or 2, characterized in that: There are multiple tags, and the method further includes: Determining whether a plurality of the label elements are obscured; In the case where a plurality of the label elements are blocked, determining a plurality of corner points of the plurality of the label elements; According to the multiple corner points, the multiple label elements are merged into one label element.

6. The method for moving a mobile device over a threshold according to claim 1 or 2, characterized in that: Also includes: When the distance changes, the navigation path is updated.

7. A device for moving a mobile device over a threshold, characterized in that: The threshold strip includes a label, the mobile device includes a visual component, and the apparatus includes: A first determining module is configured to determine, based on an image captured by the visual component, a label element in the image; the label element is an element formed by the label in the image; a second determining module, configured to determine a distance between the mobile device and the threshold bar according to a pixel value of the tag element and an actual size of the tag; a third determining module, configured to determine, in a case where the distance is not greater than a target threshold, a plurality of coordinate information of the tag using a plurality of methods according to the tag element; A fusion module, configured to fuse the plurality of coordinate information to obtain target coordinate information of the tag; The fourth determining module is configured to determine a navigation path of the mobile device passing through the threshold according to the target coordinate information.

8. An electronic device, characterized in that: The invention comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction is executed by the processor to implement the threshold crossing method for a mobile device according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the method for a mobile device to pass through a threshold bar according to any one of claims 1 to 6 is implemented.

10. A mobile device, characterized in that: include: The threshold strip device for mobile devices according to claim 7; or the electronic device according to claim 8; or The storage medium according to claim 9.

Images