Positioning system, control method and mower

Abstract

The invention provides a positioning system, a control method and a mower, the positioning system is applied to a mobile device, the positioning system comprises a visual sensor used for collecting a first environment image of the position where the mobile device is located, and the first environment image is used for obtaining state measurement of the mobile device; the speedometer is used for acquiring motion parameters of the mobile device, the motion parameters comprise a first acceleration and / or a first speed, and the motion parameters are used for predicting a first state variable of the mobile device; the main control chip is used for determining a first confidence coefficient according to a preset confidence coefficient model and the first state variable, the first confidence coefficient is used for indicating whether wheels of the mobile device slip or not, determining a second state variable according to the first confidence coefficient and the first state variable, and determining positioning information of the mobile device according to the second state variable and the state measurement. Based on the scheme, the positioning accuracy of the positioning system can be improved.

Description

Technical Field

[0001] The present application relates to the field of automatic control technology, and more specifically, to a positioning system, a control method and a lawn mower. Background Art

[0002] With the development of automatic control technology, in order to improve the efficiency of gardening or agricultural work, mowing robots (also called lawn mowers) have been proposed and applied. Generally, lawn mowers usually need to be equipped with a positioning system to achieve autonomous navigation, path planning, obstacle avoidance and other operations of the lawn mower.

[0003] However, during the operation of the lawn mower, tire slippage is a common problem, which will cause a deviation between the actual motion trajectory of the lawn mower and the expected planned path, causing the lawn mower to be unable to move along the established route, thereby deviating from the target area, reducing the working efficiency of the lawn mower, and even causing safety accidents. Summary of the invention

[0004] This application provides a positioning system, a control method and a lawn mower. Taking the lawn mower application scenario as an example, it helps to reduce the impact of lawn mower tire slippage on the actual running path of the lawn mower. The solution also introduces a mechanism for lawn mower repositioning based on markers to reduce the cumulative error of the positioning system. In addition, the operating mechanism of the solution is relatively simple, and the difficulty and cost of implementation are relatively low.

[0005] In a first aspect, a positioning system is provided, which is applied to a mobile device, and the positioning system includes: a visual sensor, used to collect a first environmental image of the location of the mobile device, the first environmental image is used to obtain a state measurement of the mobile device; an odometer, used to obtain motion parameters of the mobile device, the motion parameters include a first acceleration and / or a first speed, the motion parameters are used to predict a first state variable of the mobile device; a main control chip, used to determine a first confidence level according to a preset confidence model and the first state variable, the first confidence level is used to indicate whether a wheel of the mobile device is slipping; using the first confidence level and the first state variable to obtain a second state variable, and determining the positioning information of the mobile device according to the second state variable and the state measurement.

[0006] Based on the above technical solution, the influence of the wheel slip of the mobile device on the positioning information of the mobile device calculated by the positioning system can be effectively reduced, thereby avoiding a large deviation between the actual motion trajectory of the mobile device and the expected planned path, causing the mobile device to deviate from the target area. Furthermore, in the process of obtaining the state variables of the mobile device, a calculation method based on confidence is introduced to determine whether the state variables obtained at the current moment are obtained when the wheels of the mobile device slip, and thus adjust the fusion weight of the state variables obtained at the current moment according to the confidence, so that the subsequent positioning calculation can achieve more accurate positioning of the mobile device based on the state variables and confidence.

[0007] In a second aspect, a control method is provided, which is applied to a positioning system including any possible implementation method as in the first aspect. When the mobile device is a lawn mower, the method includes: acquiring a second environment image including a first area, and the lawn mower is located in the first area; extracting a region of interest (ROI) in the second environment image to obtain an ROI image; performing feature extraction on the ROI image to obtain first feature information, and the first feature information is used to indicate that the first area includes a lawn area and / or a non-lawn area; and determining a moving path of the lawn mower based on the first feature information.

[0008] Based on the above technical solution, the second environment image acquired by the visual sensor can accurately identify the lawn boundary and ensure that the lawn mower runs efficiently along the boundary. On the one hand, by judging the location of the non-grass area, the lawn mower can flexibly adjust its direction to avoid miscutting and repeated mowing, thereby improving work efficiency; on the other hand, based on the visual perception technology, not only the installation cost and maintenance cost of the overall lawn mower system are reduced, but also the system automatically generates the path along the edge and performs real-time trajectory tracking through the images collected by the visual perception technology, without the need for complex sensor fusion technology, while ensuring that the overall system maintains a high level of automation, reducing manual intervention and improving ease of operation.

[0009] In a third aspect, a control method is provided, which is applied to a positioning system including any possible implementation method as in the first aspect, when the mobile device is a lawn mower, the lawn mower is located in a first lawn area, and during the process of the lawn mower moving along a third path, the method includes: in response to a first instruction instructing the lawn mower to move to a second lawn area, controlling a visual sensor to collect a third environmental image of the location of the lawn mower, the third environmental image including environmental information of the location of the lawn mower; detecting whether the third environmental image includes a first indicator, when the third environmental image includes the first indicator, determining a fourth path, the first indicator is associated with the first lawn area, and the first indicator is used to indicate a first direction, the first direction is the direction of the second lawn area relative to the first indicator, and the fourth path passes through the first indicator and extends in the first direction; or, when the third environmental image does not include the first indicator, controlling the lawn mower to rotate so that the visual sensor collects the third environmental image including the first indicator.

[0010] Based on the above technical solution, by using the indicator as a reference for visual positioning information, the lawn mower is guided to complete the cross-zone task according to the corresponding path indicated by the indicator. Since the indicator has the characteristics of unique appearance, simple layout, and rich information, the random mowing lawn mower can quickly and conveniently complete the cross-zone task, and through the update of relevant data structures and states, the flexible cross-zone mowing function of multiple mowers and multiple cross-zones is realized.

[0011] In a fourth aspect, a control device is provided, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute a method in any possible implementation mode of the method design of the second aspect or the third aspect above.

[0012] In a fifth aspect, a lawn mower is provided, comprising a positioning system as in any possible implementation of the first aspect, and the control device described in the fourth aspect.

[0013] In a sixth aspect, a computer-readable storage medium is provided, storing a computer program, wherein the computer program is executed by a processor to implement a method in any possible implementation manner in the method design of the second aspect or the third aspect.

[0014] In a seventh aspect, a computer program product is provided, comprising instructions, which, when executed by a processor, enable a computer to execute a method in any possible implementation of the method design of the second aspect or the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of a framework of a positioning system 100 that is excluded from the embodiment of the present application;

[0016] Figure 2 is a flow chart of a method 200 for calculating measurement confidence proposed in an embodiment of the present application;

[0017] Figure 3 is a wheel speed change curve of a mobile device when the wheel slips, provided in an embodiment of the present application;

[0018] Figure 4 is a curve of change between power and wheel speed when a wheel of a mobile device slips, provided in an embodiment of the present application;

[0019] Figure 5 is a first-order linear relationship fitting curve between the power and wheel speed of a mobile device provided in an embodiment of the present application;

[0020] Figure 6 is an acceleration change curve of a mobile device when it slips, provided in an embodiment of the present application;

[0021] Figure 7 is a first-order linear relationship fitting curve between power and acceleration of a mobile device provided in an embodiment of the present application;

[0022] Figure 8 is a fitted relationship curve between driving power, wheel speed and acceleration provided in an embodiment of the present application;

[0023] Fig. 9 It is a schematic diagram of a real-time operating power curve and a slipping power curve of a mobile device proposed in an embodiment of the present application;

[0024] Fig.10 is a confidence curve under multiple wheel speeds provided in an embodiment of the present application;

[0025] Fig.11 is a confidence change curvature diagram under multiple wheel speeds provided in an embodiment of the present application;

[0026] Fig.12 is another confidence curve under multiple wheel speeds provided in an embodiment of the present application;

[0027] Fig.13 is another confidence change curvature diagram under multiple wheel speeds provided in an embodiment of the present application;

[0028] Fig.14 It is a confidence change curvature diagram corresponding to different values ​​of C5 provided in an embodiment of the present application;

[0029] Fig.15 It is a confidence change curvature diagram corresponding to different values ​​of C5 provided in an embodiment of the present application;

[0030] Fig.16is a schematic diagram of an application scenario of a mobile device proposed in an embodiment of the present application;

[0031] Fig.17 is a flowchart of a positioning method 1700 proposed in an embodiment of the present application;

[0032] Fig.18 is a schematic diagram of a framework of another positioning system 100 proposed in an embodiment of the present application;

[0033] Fig.19 It is a marker pattern proposed in the embodiment of the present application;

[0034] Fig. 20 is a flow chart of a control method 2000 proposed in an embodiment of the present application;

[0035] Fig.21 is a flow chart of a control method 2100 proposed in an embodiment of the present application;

[0036] Fig. 22 It is a schematic diagram of a principle for determining a third conversion relationship proposed in an embodiment of the present application;

[0037] Fig.23 It is a design diagram of an indicator proposed in an embodiment of the present application;

[0038] Fig.24 This is a schematic diagram of a multi-mower multi-span application scenario proposed in an embodiment of the present application;

[0039] Fig.25 is a schematic diagram of a control device 2500 proposed in an embodiment of the present application;

[0040] Fig.26 It is a schematic diagram of a control device 2600 proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0042] The embodiments of the present application will present various aspects, embodiments or features around a system including multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0043] In addition, in the embodiments of the present application, words such as "exemplary" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0044] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0045] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0046] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can be represented by: including the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0047] In the embodiments of the present application, the same reference numerals are used to represent the same components or parts. For the same parts in the embodiments of the present application, only one of the parts or parts may be marked with a reference numeral in the figure. It should be understood that the reference numerals are also applicable to other identical parts or parts. In addition, the various parts in the drawings are not drawn to scale, and the sizes and dimensions of the parts shown in the drawings are only exemplary and should not be understood as limiting the present application.

[0048] With the rapid development of automation control technology, people are beginning to seek more efficient and intelligent ways to complete gardening and agricultural work. In order to meet this demand, lawn mowing robots (hereinafter referred to as lawn mowers) were proposed and widely used. Based on automation technology, lawn mowers can complete mowing operations autonomously, thereby greatly improving the efficiency of gardening or agricultural work.

[0049] The positioning system is a crucial component in the operation of a lawn mower. The positioning system is mainly used to realize the functions of autonomous navigation, path planning and obstacle avoidance of the lawn mower. For example, the positioning system can obtain the location information of the lawn mower in real time and compare it with the preset map or route. The positioning system can guide the lawn mower to move along the predetermined route while avoiding possible obstacles.

[0050] Wheel odometers are widely used in mobile robots and can be integrated with other sensors such as vision and inertial measurement unit (IMU) for positioning and speed measurement. However, in actual operation, lawn mowers still encounter some problems. For example, the impact of tire slip on the positioning system, that is, when the lawn mower is driving on wet, muddy or uneven ground, the tires can easily lose grip and cause slippage. Due to slippage, although the driving distance of the lawn mower may be short, the driving distance responded by the wheel odometer may be long, resulting in a deviation between the actual motion trajectory of the lawn mower and the expected planned path. This means that the lawn mower may not be able to move along the established route, but will deviate from the target area and may even enter a dangerous area, which will not only reduce the working efficiency of the lawn mower, but may also cause safety accidents and damage the lawn mower itself or the surrounding environment.

[0051] Moreover, pure visual positioning has cumulative errors, and the longer the time, the less accurate the positioning. For positioning systems that integrate absolute measurement, due to weather, occlusion, etc., absolute measurement (such as measurement technology based on real-time kinematic (RTK)) cannot guarantee that it will always work effectively. Therefore, when there is no absolute measurement for a long time, the integrated positioning begins to accumulate positioning errors.

[0052] In addition, the lawn mower working control schemes proposed at this stage are relatively simple, and cannot realize some complex controls of the lawn mower at a relatively low cost, such as mowing along the edge, mowing across areas, and multi-machine collaborative operation.

[0053] Based on the current technical solutions, lawn mowers usually need to rely on a variety of sensors, such as ultrasonic sensors, infrared sensors or lidar, to sense the surrounding environment and mow along the edges. These devices can detect obstacles and boundaries, thereby guiding the lawn mower to mow along a predetermined path. However, these sensors are often expensive, increasing the overall cost of the lawn mower, and may affect mowing efficiency and accuracy due to the limitations of the sensors.

[0054] As for the non-path planning (random) lawn mowing robot technology that has been eliminated at this stage, a pre-buried radio frequency identification (RFID) coil solution is usually used to complete the cross-zone task. However, the RFID coil deployment process is cumbersome. First, you need to find the boundary grass that meets the requirements, then fix the RFID coil, and finally identify the RFID coil through the terminal device. Moreover, the fixed installation of the RFID coil will damage the lawn. The RFID coil is hidden in the grass, and the cross-zone indication is not obvious to the user. The lawn mower will only have a feedback signal output when it passes directly above the RFID coil to guide the lawn mower to cross the zone, which is inefficient.

[0055] Of course, the embodiments of the present application are only for lawn mowers, and take lawn mowing as an example of application scenarios, and put forward the problems to be solved in the field of robot automatic control, as well as the corresponding technical solutions, for robots in other application scenarios, such as sweeping robots, agricultural planting robots, agricultural picking robots, etc. For example, for agricultural planting robots, agricultural planting robots also need to be equipped with positioning systems, and when the tires of agricultural planting robots slip, the actual motion trajectory of agricultural planting robots will also cause deviations from the expected planned path, and the agricultural planting robots proposed at this stage cannot achieve complex control of agricultural planting robots at a relatively low cost.

[0056] In view of this, the embodiment of the present application proposes a positioning system and control method. Taking the lawn mower application scenario as an example, it helps to reduce the impact of lawn mower tire slippage on the actual running path of the lawn mower. The solution also introduces a mechanism for lawn mower repositioning based on markers. On the one hand, it can reduce the impact of tire slippage on the accuracy of subsequent lawn mower positioning, and on the other hand, it can reduce the cumulative error of the positioning system. And the operating mechanism of the scheme is relatively simple, and the difficulty and cost of implementation are low. In addition, a control method based on marker guidance is proposed, which can realize cross-regional operation of lawn mowers and collaborative operation of multiple machines.

[0057] Figure 1 1 is a schematic diagram of a positioning system 100 that is excluded from the embodiment of the present application. The positioning system 100 can be applied to a mobile device, referring to Figure 1As shown, the positioning system 100 may include: a visual sensor, used to collect a first environmental image of the location of the mobile device, the first environmental image is used to obtain a state measurement of the mobile device; an odometer, used to obtain motion parameters of the mobile device, the motion parameters include a first acceleration and / or a first speed, the motion parameters are used to predict a first state variable (also called a predicted state variable) of the mobile device; a main control chip, used to determine a first confidence level according to a preset confidence model and the first state variable, the first confidence level is used to indicate whether the wheels of the mobile device are slipping; using the first confidence level and the first state variable to obtain a second state variable, and determining the positioning information of the mobile device according to the second state variable and the state measurement.

[0058] It should be understood that the state variables involved in the embodiments of the present application are a set of parameters that describe the current state of the mobile device. In the positioning system, these parameters are usually related to physical quantities such as the position, speed, acceleration, etc. of the mobile device. The above-mentioned motion parameters are used to predict the first state variable of the mobile device, which refers to the future state of the mobile device estimated based on the motion parameters (such as the first acceleration and / or the first speed, etc.) obtained by the odometer, and these variables can be understood as mathematical representations of the dynamic behavior of the mobile device. In the embodiments of the present application, the above-mentioned state measurement may refer to the full-degree-of-freedom posture of the system output by the sensor through its respective positioning algorithm, i.e., 6 degrees of freedom (DOF), or partial degrees of freedom posture, such as 3DOF, which generally includes translation, rotation, speed, bias, etc.

[0059] The state measurement involved in the embodiment of the present application is a detailed observation value of the state variable of the mobile device. In the positioning system 100 proposed in the embodiment of the present application, the state measurement is obtained by a visual sensor. The visual sensor can be a radar, a laser scanner, a camera, etc., which can scan the application scene in real time and collect data. The above-mentioned first environmental image is used to obtain the state measurement of the mobile device, which refers to the measurement data obtained by scanning the application scene by the visual sensor, and these data reflect the actual state of the mobile device at the current moment, and are the basis for updating the first state variable predicted by the positioning system 100. Generally, the above-mentioned predicted first state variable may include translation, rotation, speed, bias, external parameters between sensors, etc.

[0060] In some possible embodiments, the visual sensor and the odometer operate simultaneously, that is, the first environment image and the motion parameters are acquired simultaneously, for example, at the kth moment.

[0061] In some possible embodiments, the sensor used to obtain the first state variable in the positioning system 100 may include at least one, and the sensor may include the visual sensor and / or the posture sensor. When the positioning system 100 includes the posture sensor, the first state variable also includes the posture information of the mobile device. The visual sensor may include a sensor with a visual function such as a camera or a laser radar that can collect images or point cloud data, and the posture sensor may include a sensor with a positioning information function such as RTK, ultra wide band (UWB) or global positioning system (GPS).

[0062] In some possible embodiments, the odometer includes an inertial odometer and / or a wheel odometer, wherein the inertial odometer is used to obtain a first acceleration, which includes a linear acceleration and / or an angular acceleration, and the wheel odometer is used to obtain a first speed, which includes a linear speed and / or an angular speed. The motion parameters recorded by the inertial odometer and / or the wheel odometer are processed to predict the first state variable of the mobile device.

[0063] In some possible embodiments, different types of odometers obtain different ways of predicting the first state variable of the fused positioning system 100. When the odometer is an inertial odometer, the above-mentioned motion parameters may include acceleration and angular acceleration. By integrating the acceleration and angular acceleration and combining the true value estimation state at time k-1, the first state variable of the fused positioning system 100 at time k can be predicted; when the odometer is a wheel odometer, the above-mentioned motion parameters may include linear velocity and angular velocity. By integrating the linear velocity and angular velocity and combining the true value estimation state at time k-1, the first state variable of the fused positioning system 100 at time k can be predicted.

[0064] In one embodiment, the main control chip may be a central processing unit (CPU), or other general-purpose processors, neural network chips, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0065] However, when the odometer is a wheel odometer, the wheels may slip due to problems such as low ground friction coefficient in the mobile device application scenario, poor contact between the robot wheels and the ground, or mismatch in wheel speed. This phenomenon may cause the first state variable predicted based on the wheel odometer to be inaccurate, and thus make the positioning information of the mobile device inaccurate.

[0066] Therefore, the embodiment of the present application proposes that the above-mentioned main control chip determines the confidence of the predicted first state variable through a confidence model, and makes a slip judgment on the first state variable based on the confidence to further obtain the second state variable; the second state variable and the previously obtained state measurement are used to determine the positioning information of the mobile device, thereby reducing the impact of slip. Accordingly, the present application provides a method for calculating the confidence of wheel odometer measurement to reduce the impact of slip; the subsequent positioning algorithm can also adjust the fusion weight of the wheel odometer according to this confidence, thereby improving the positioning accuracy; or identify that the mobile device is in a slipping state and needs to enter an escape operation state.

[0067] In some possible embodiments, the above confidence model is used to indicate the confidence corresponding to the first state variable of the mobile device at different speeds, and the confidence is used to indicate whether the wheels of the mobile device are slipping.

[0068] Figure 2 2 is a flow chart of a method 200 for calculating measurement confidence proposed in an embodiment of the present application. The method 200 can be applied to a wheel odometer, referring to Figure 2 As shown, the method 200 may include the following steps:

[0069] S210: Obtaining the static power C0 of the motor when the mobile device is stationary, the first change slope C1 of the driving power of the motor and the wheel speed V when the mobile device is moving, and the second change slope C2 of the driving power and the acceleration acc.

[0070] S220: Determine the slip power p0 of the wheel odometer when the wheels of the mobile device slip according to the above parameters and the preset linear equation.

[0071] In some possible embodiments, the above preset linear equation can be expressed by the following formula (1):

[0072] p0=C0+ C1*|V|+C2*acc (1)

[0073] S230: Obtaining the actual operating power p of the motor that satisfies the normal distribution when the mobile device is running t .

[0074] S240: Taking the above-mentioned slip power p0 as a reference value, and combining it with the actual operating power and a preset confidence model, obtain the confidence ω of the first state variable of the wheel odometer.

[0075] In some possible embodiments, in the subsequent positioning calculation process of the positioning system 100, the fusion weight of the wheel odometer may be adjusted according to the confidence level.

[0076] In some possible embodiments, the above-mentioned preset confidence model is a confidence function, which can be expressed by the following formula (2):

[0077]

[0078] Among them, f(p t ) is used to represent the confidence ω of the first state variable of the wheel odometer.

[0079] In some possible embodiments, the above preset credibility model may also be expressed by the following formulas (3) to (5):

[0080]

[0081] if P t <P0;ω=0 (4)

[0082] if V=0and acc=0;ω=1 (5)

[0083] Among them, C3, C4, C5 are constants. When the actual power p t When the actual power p is less than the slip power p0, it means that the wheels of the mobile device are in a slipping state, so the confidence of the first state variable ω=0; when the speed and acceleration of the mobile device are 0, it means that the mobile device is in a stationary state and there is no slipping at this time, so the confidence of the first state variable ω=1; when the actual power p t If it is greater than the slip power p0, the confidence ω of the first state variable is calculated according to the above formula (3).

[0084] In some possible embodiments, the above S210 can be implemented in the following manner: when the mobile device slips, the acceleration process of the wheel speed from 0 to 6 rad / s is controlled and recorded, and then the deceleration process of the wheel speed from 6 to 0 rad / s is controlled and recorded. Each time a stage is changed, the mobile device is controlled to run at a constant speed for a period of time. Figure 3 This is a wheel speed change curve of a mobile device when the wheel slips, provided in an embodiment of the present application. Based on the above process, the driving power of the motor during the change of the wheel speed can be further obtained, and a first curve can be fitted according to the wheel speed and the driving power, and the first curve is a change curve between the power and the wheel speed when the wheel slips.

[0085] Figure 4 This is a curve of the change between power and wheel speed when the wheels of a mobile device slip, provided in an embodiment of the present application. Figure 4 As shown, Figure 4 The horizontal axis is the wheel speed, and the vertical axis is the motor drive power of the mobile device. Figure 4 It can be seen that the speed of the mobile device is proportional to the power, and when the speed and acceleration are 0, the power value at this time is recorded as the static power value.

[0086] Further, according to Figure 4 The change curve shown fits the first linear equation of driving power and wheel speed, and the second linear equation of driving power and acceleration is obtained based on the first linear equation. By combining the first linear equation and the second linear equation, a preset linear equation can be obtained.

[0087] In some possible embodiments, Figure 4 By fitting the change curve shown in FIG. 1 , the static power C0 of the motor when the mobile device is stationary and the first change slope C1 of the driving power of the motor and the wheel speed when the mobile device is moving can be obtained.

[0088] Figure 5 is a first-order linear relationship fitting curve of power and wheel speed when the wheels of a mobile device slip, provided in an embodiment of the present application. Figure 5 The curve shown can establish the first linear equation P = C0 + C1 * |V|. Then calculate by the following formula: acc = (V t -V t-1 ) / Δt, where V t is the wheel speed obtained at time t, V t-1 It is the wheel speed obtained at the t-1th moment (i.e., the moment before the tth moment), and the acceleration change curve over time can be obtained (corresponding to the red curve).

[0089] Figure 6 is an acceleration change curve of a mobile device when it slips, provided in an embodiment of the present application. Figure 7 is a first-order linear relationship fitting curve between power and acceleration when a mobile device is slipping, provided in an embodiment of the present application. Figure 6 and Figure 7 As shown, Figure 7 By fitting the curve, the second change slope C2 of the driving power and the acceleration acc can be obtained, and then the second linear equation P=C2*acc is constructed. Combining the first linear equation and the second linear equation, the correlation between the driving power and the wheel speed, and the correlation between the driving power and the acceleration can be obtained. Figure 7 The blue curve is the curve of power and wheel speed change during slipping, and the red curve is the change curve fitted by the above formula.

[0090] Figure 8 is a fitted relationship curve between driving power, wheel speed and acceleration provided in an embodiment of the present application. Figure 8 As shown in the figure, the curve is used to represent the above preset linear equation. Figure 8 In the figure, the blue curve is the curve of power and speed change during slipping, and the red curve is the curve obtained by fitting the above preset linear equation.

[0091] In some possible embodiments, the power during slipping can be used as a position parameter in a normal distribution function. When the mobile device is operating normally, the driving power of the mobile device should satisfy the normal distribution function, that is, the above formula (2). Further, f(p t ) is negated to obtain a preset confidence function of the first state variable of the wheel odometer, which can be expressed by the following formulas (6) to (8):

[0092]

[0093] if P t <P0;ω=0 (7)

[0094] if V=0and acc=0;ω=1 (8)

[0095] Wherein, formula (7) and formula (8) are the same as the above formula (4) and formula (5), and the meanings expressed by formula (7) and formula (8) are not repeatedly explained here.

[0096] Furthermore, the confidence function is evaluated, that is, the mobile device is controlled to collect actual power (or real-time operating power) at different speeds during actual operation.

[0097] Fig. 9 Schematic diagram of a real-time operating power curve (corresponding to the blue curve) and a slipping power curve (corresponding to the red curve) of a mobile device proposed in an embodiment of the present application. Fig. 9 As shown, the blue curve is used to represent the real-time operating power of the mobile device during operation, and the red curve is used to represent the slipping power curve between the power and speed during the slipping.

[0098] Fig.10 is a confidence curve under multiple wheel speeds provided in an embodiment of the present application. Fig.10 By fitting the coordinate points in , we can determine the following Fig.11 The curve shown is used to indicate the change in the curvature of the confidence level; Fig.11is a confidence change curvature diagram under multiple wheel speeds provided by an embodiment of the present application. The confidence of the wheel odometer under different wheel speeds is calculated based on the above confidence function, such as Fig.10 and Fig.11 Based on Fig.11 It can be seen that when the mobile device is running at a low speed (for example, the wheel speed is less than 0.2 rad / s), the running power and the slip power are very close, resulting in a very low confidence at low speed, and the curvature of the change of the confidence is single. In view of this, the above-mentioned preset confidence function (see formulas (6) to (8)) can be improved so that the confidence can have different curvatures at different speeds. The modified formulas are shown in formulas (3) to (5) above.

[0099] Fig.12 is another confidence curve under multiple wheel speeds provided in an embodiment of the present application. Fig.13 This is another confidence change curvature diagram under multiple wheel speeds provided by an embodiment of the present application, wherein multiple wheel speeds v are quantized to 1 to 6, and C4=0, C5=3.5. Fig.12 and Fig.13 It is obtained based on the functional expressions represented by the above formulas (3) to (5), referring to Fig.12 and Fig.13 As shown, the smaller the wheel speed of the mobile device, the higher the confidence gradient of the wheel odometer is, thereby avoiding the situation where the operating power and the slip power are very close when the mobile device is running at a low speed, resulting in a very low confidence at a low speed and an inability to accurately judge whether the wheel is slipping.

[0100] In some possible embodiments, in order to ensure the accuracy of confidence calculation, constant parameters C4 and C5 are introduced into the above formula (3), thereby affecting the confidence finally obtained.

[0101] Fig.14 : This is a confidence change curvature diagram corresponding to different values ​​of C5 provided in an embodiment of the present application, wherein C4 is set to 0, and C5 is set to 0.3 (corresponding to the red curve) and 0.6 (corresponding to the blue curve). Fig.14 As shown, when C4 is fixed, the confidence change curvature of different C5 values ​​at different wheel speeds is obtained, and it can be determined that the larger the C5 parameter is, the lower the confidence will be. Therefore, the C5 value can have a greater impact on the confidence under high-speed conditions.

[0102] Fig.15 : This is a confidence change curvature diagram corresponding to different values ​​of C4 provided in an embodiment of the present application, wherein C5 is set to 0.35, and C4 is set to 0.1 (corresponding to the red curve) and 1.0 (corresponding to the blue curve). Fig.15As shown, when C5 is fixed, the confidence change curvature of different C4 values ​​at different wheel speeds is obtained, and it can be determined that the larger the C4 parameter, the lower the confidence, so the C4 value can have a greater impact on the confidence under low-speed conditions.

[0103] It can be seen that when the mobile device is running outdoors, the confidence at low speed can be increased (i.e., the C4 value is reduced) and the confidence at high speed can be reduced (i.e., the C5 value is increased), thereby improving the accuracy of detecting wheel slippage and subsequent positioning accuracy as a whole. Similarly, when the mobile device moves in different application scenarios, the values ​​of C4 and C5 can be adaptively adjusted, thereby helping to increase the positioning accuracy of the positioning system 100.

[0104] Based on the above technical solution, the influence of the wheel slip of the mobile device on the positioning information of the mobile device calculated by the positioning system can be effectively reduced, thereby avoiding a large deviation between the actual motion trajectory of the mobile device and the expected planned path, causing the mobile device to deviate from the target area. Furthermore, in the process of obtaining the state variables of the mobile device, a calculation method based on confidence is introduced to determine whether the state variables obtained at the current moment are obtained when the wheels of the mobile device slip, and thus adjust the fusion weight of the state variables obtained at the current moment according to the confidence, so that the subsequent positioning calculation can achieve more accurate positioning of the mobile device based on the state variables and confidence.

[0105] In addition, the positioning system 100 can also improve the accuracy of the positioning information obtained by fusing data obtained from multiple sensors (such as the above-mentioned visual sensor and / or posture sensor, and odometer). However, considering the weather, obstacle occlusion, and the existence of cumulative errors, the visual posture sensor may not be guaranteed to work effectively all the time. Therefore, when there is no absolute image data from the posture sensor for a long time, it will only rely on pure visual sensors and odometers, and positioning errors will occur. In this regard, the present application uses the following scheme to achieve repositioning of the positioning system, thereby reducing the measurement errors that only rely on visual sensors and odometers, and further improving the positioning accuracy of the positioning system 100.

[0106] Fig.16 is a schematic diagram of an application scenario of a mobile device proposed in an embodiment of the present application. Assume that the mobile device is equipped with a positioning system, which includes a visual sensor and a posture sensor. Fig.16 As shown, before the mobile device works, Fig.16Corresponding visual signs (or markers) need to be deployed in the application environment site, and the mobile device is controlled to move along a preset path, and the visual sensor and the posture sensor are simultaneously turned on to collect data; during the movement of the mobile device, the main control chip can be based on the positioning method provided in the present application, especially combined with the repositioning mechanism of the visual sensor and the posture sensor, that is, control the mobile device to move along the repositioning path, and configure the positioning system 100 to enter the corresponding working mode, so as to obtain high-precision positioning information of the mobile device.

[0107] Fig.17 1700 is a flowchart of a positioning method 1700 proposed in an embodiment of the present application. Fig.17 As shown, the method 1700 includes the following steps: S1710: obtaining the working status of the visual sensor and the posture sensor; S1720: configuring the positioning system to enter the corresponding working mode according to the working status of the visual sensor and the posture sensor; when the working mode is the normal fusion mode, go to S1730; when the working mode is the repositioning mode, go to S1740.

[0108] Among them, in normal fusion mode, the visual sensor and posture sensor work normally, and the error is within the tolerance range; in repositioning mode, the visual sensor and posture sensor are in an abnormal working state, or the error between the visual sensor and posture sensor exceeds the tolerance range.

[0109] In some possible embodiments, the above working state includes a valid working state and a failed working state. When the visual sensor is in a valid working state, if the working state of the posture sensor is a valid working state, the positioning system is configured to enter a normal fusion mode to obtain positioning information; if the working state of the posture sensor is a failed working state, the positioning system is configured to enter a repositioning mode to obtain positioning information. When the visual sensor is in a failed working state, the visual sensor or the positioning system needs to be restarted until the visual sensor can collect images in the application scene; if the visual sensor still does not work after multiple restarts, it means that the positioning system has failed and needs to be replaced.

[0110] In some possible embodiments, whether the visual sensor is in an effective working state can be determined by the following methods:

[0111] By setting a time threshold to evaluate whether the image data of the visual sensor can be obtained, it is determined whether the data is lost. That is, if the positioning system does not receive the data from the visual sensor within a certain period of time, it is determined that the data is lost. At this time, the visual sensor is in an invalid working state and the visual sensor or positioning system needs to be restarted. Otherwise, the visual sensor is in a valid working state.

[0112] Alternatively, the visual sensor can be judged whether it is in an effective working state based on the difference in data obtained under different states of the visual sensor. By tracking whether the number of stable feature data obtained by the visual sensor at time k is too small, the image data corresponding to the sensor is judged to be invalid, that is, the visual sensor is judged to be in an invalid working state, and the visual sensor or positioning system needs to be restarted; if there are weak textures, moving objects, etc., the number of features that the visual sensor can stably track at time k is too small, which makes the current image data corresponding to the visual sensor invalid, that is, the visual sensor is judged to be in an invalid working state, and the visual sensor or positioning system needs to be restarted.

[0113] It should be noted that, in this application, a stable feature is a feature whose required information can be collected by a visual sensor over a continuous period of time. For example, a feature with depth information over a continuous period of time is considered a stable feature.

[0114] In some possible embodiments, whether the posture sensor is in a valid working state can be determined by the following methods:

[0115] The main control chip can determine whether the posture sensor is in a valid working state or a failed working state at time n based on the state measurement of the posture sensor. The detailed operation is as follows: by setting a time threshold (such as more than 2 minutes) to evaluate whether the state measurement of the posture sensor can be obtained to determine whether the data is lost, that is, if the main control chip does not receive the data of the posture sensor within a certain period of time, it is determined that the data is lost, and the posture sensor is in a failed working state at this time, otherwise the posture sensor is in a valid working state. If the posture sensor cannot output the state measurement normally due to feature tracking failure in the case of bad weather or occlusion by houses or trees, the posture sensor is considered to be in a failed working state at this time.

[0116] Alternatively, the main control chip can also determine whether the posture sensor is in a valid working state or a failed working state at time n based on the state measurement of the posture sensor. The detailed operation is as follows: according to the difference in data obtained under different states of the posture sensor, determine whether the posture sensor is in a valid working state. For example, the covariance of the state variables of the mobile device at the current moment can be compared, or it can be determined whether the difference between the state variables of the mobile device at the previous moment and the state variables at the current moment is significantly different from the preset threshold (that is, greater than the preset threshold), so as to determine whether the posture sensor is in a valid working state.

[0117] It should be noted that the normal fusion mode and repositioning mode in the present application are two separate threads, which can configure the positioning system to enter the corresponding working mode according to the working status of the visual sensor and the posture sensor, thereby reducing the computing power requirements of the positioning system while ensuring that the positioning system obtains high-precision positioning information.

[0118] S1730: Obtain image data collected by the visual sensor and state measurements obtained by the posture sensor, determine predicted state variables based on the image data, and use the state measurements and predicted state variables to obtain positioning information.

[0119] The image data may be the first environment image mentioned in the aforementioned embodiment, and the predicted state variable may be the first state variable predicted by the positioning system mentioned in the aforementioned embodiment, which is a predicted quantity.

[0120] In some possible embodiments, in the process of determining positioning information based on state measurements and predicted state variables, the confidence of the predicted state variables proposed in the aforementioned embodiments can be obtained, and the predicted state variables can be corrected based on the confidence, and finally the positioning information can be determined based on the state measurements and the corrected predicted state variables.

[0121] In some possible embodiments, the above-mentioned predicted state variables can be implemented in the following manner: predict the state variables of the positioning system at time n based on the positioning information of the positioning system at time m and the image data of the visual sensor at time n, so as to obtain the predicted state variables of the positioning system at time n.

[0122] In some possible embodiments, the predicted state variable x of the positioning system at time n can be predicted by the nominal state motion equation, and the nominal state motion equation can be expressed by the following formula (9):

[0123]

[0124] Among them, p n represents the predicted translation of the positioning system at time n in the global coordinate system, p m R represents the translation of the true value estimation state in the positioning information of the positioning system at time m in the global coordinate system, and time m is before time n; m Represents the true value estimated rotation matrix from the visual sensor to the global coordinate system at time m, which can be obtained through the quaternion q m Calculated; represents the relative translation between the visual sensors from time n to time m in the visual sensor coordinate system, which can be obtained based on the images corresponding to time n and time m; q n Represents the quaternion corresponding to the predicted rotation matrix of the positioning system at time n, q m Represents the true value estimated quaternion corresponding to the positioning system at time m, It represents the relative transformation of the four elements between the visual sensors from time n to time m in the visual sensor coordinate system, which can be obtained based on the images corresponding to time n and time m.

[0125] It should be noted that the global coordinate system may be a world coordinate system relative to the site where the application scenario is located, and the visual sensor coordinate system refers to a coordinate system established based on the visual sensor.

[0126] In some possible embodiments, the predicted state variables of the positioning system at time n can be predicted using a preset neural network model, that is, the positioning information of the positioning system at time m and the image data of the visual sensor at time n are input into the preset neural network model for processing to obtain the predicted state variables of the positioning system at time n.

[0127] After obtaining the predicted state variables of the positioning system, the state measurement of the posture sensor can be further used to update the predicted state variables of the positioning system to obtain the positioning information of the positioning system. The detailed steps of this operation are as follows: based on the state measurement of the posture sensor at time n, the observation quantity of the posture sensor is obtained; according to the predicted state variables of the positioning system at time n and the observation quantity of the posture sensor at time n, the error state variables of the positioning system at time n are determined; the predicted state variables of the positioning system are optimized using the error state variables to obtain the positioning information of the positioning system at time n.

[0128] S1740: In the repositioning mode, image data captured by the visual sensor is obtained and it is detected whether there is a visual mark in the image. If so, repositioning is performed according to the position of the visual mark in the image to obtain positioning information of the positioning system; if not, the mobile device is controlled to rotate until the visual sensor captures an image including the visual mark.

[0129] In some possible embodiments, the above S1740 can be implemented by the following detailed operations: detecting whether there is a visual mark in the image by using a preset detection algorithm. For example, graying the image to obtain a grayscale image, detecting the edge contour in the grayscale image to obtain a candidate area, matching the candidate area with the pre-stored visual mark, if the match is successful, it means that there is a visual mark in the current image, otherwise there is no visual mark, at this time, it is necessary to control the visual sensor to rotate in place (it can also be controlling the mobile device to rotate in place), and repeatedly collect images until the visual mark is detected through the image.

[0130] In some possible embodiments, the thread of the above-mentioned repositioning mode can be started by determining whether the positioning system data fusion has timed out. The fusion timeout (for example, greater than 2 minutes) can be used to indicate that the visual sensor is currently in a failed working state, and then enters the following process.

[0131] S1741: Control the mobile device equipped with the positioning system to move near a reference position. The reference position may be a charging pile in the field of the application scenario, etc., which can be used as a reference, and the visual mark is installed at the reference position.

[0132] S1742: Control the visual sensor to collect images and determine whether there is a visual mark in the image collected by the visual sensor. If yes, go to S1744; if no, go to S1743.

[0133] S1743: Rotate the mobile device body equipped with the positioning system until the visual sensor captures an image including the visual mark.

[0134] S1744: Control the mobile device to drive directly toward the visual sign, and execute a repositioning mode for the mobile device based on the position of the visual sign in the previous and next two sets of image data.

[0135] In some possible embodiments, after the presence of a visual mark in an image is detected for the first time, a first initial position of a visual sensor in the positioning system in a world coordinate system constructed based on the positioning system (i.e., a mobile device) and a second initial position of the visual sensor in a world coordinate system constructed based on the visual mark are obtained, and the first initial position and the second initial position are combined to obtain a relative position in the world coordinate system of the positioning system and the world coordinate system of the visual mark; the mobile device is controlled to continuously move toward the visual mark and obtain a moving position of the visual sensor in the world coordinate system of the visual mark, and the distance between the mobile device and the visual mark is obtained according to the moving position; if the distance is within a preset distance threshold range, the moving position is used to obtain a relative position in the world coordinate system of the positioning system and the world coordinate system of the visual mark. The corrected posture of the positioning system in the world coordinate system of the positioning system is calculated by using the mobile posture and the relative posture in the world coordinate system of the positioning system and the world coordinate system of the visual mark, and the positioning information of the mobile device in the application scenario is obtained, thereby realizing repositioning; conversely, if the distance is not within the preset distance threshold range, the mobile device is continued to be controlled to move toward the visual mark until the distance between the mobile device and the visual mark is within the preset distance threshold range, so as to repeat the step of "if the distance is within the preset distance threshold range, the corrected posture of the positioning system in the world coordinate system of the positioning system is calculated by using the mobile posture and the relative posture in the world coordinate system of the positioning system and the world coordinate system of the visual mark" to obtain the positioning information of the mobile device.

[0136] In some possible embodiments, the first initial posture can be obtained as follows: while the visual sensor is collecting images, the odometer of the positioning system synchronously records the motion parameters of the mobile device; assuming that the image in which the visual mark is first detected is the current frame image, if the positioning system can obtain the image and motion parameters (such as angular velocity or linear velocity, etc.), the motion parameters corresponding to the previous frame image recorded by the odometer are obtained and processed to obtain the predicted posture of the current frame image; according to the feature points extracted from the current frame image, feature matching is performed with the previous frame image to obtain matching features matching the previous frame image, and the predicted posture is updated using the matching features as observation constraints to obtain the first initial posture of the positioning system when collecting the current frame image, and the first initial posture represents the relative transformation relationship of the current frame image from the visual sensor coordinate system to the world coordinate system of the positioning system. Wherein, when the current frame image is the first frame image, the previous frame image of the first frame image can be the first frame image itself, and the corresponding first initial posture can be the initial value recorded by the odometer.

[0137] In some possible embodiments, the predicted position and posture of the current frame image can be obtained as follows: input the motion parameters corresponding to the previous frame image into a preset displacement prediction model, and use the preset displacement prediction model to predict the predicted displacement increment of the positioning system; according to the predicted displacement increment of the image acquisition device and the position and posture corresponding to the previous frame image acquired by the image acquisition device, the predicted position and posture when the positioning system acquires the current frame image is obtained. It should be noted that the predicted position and posture is the initial estimate of the current frame image.

[0138] In some possible embodiments, taking into account the existence of drift and other errors in the odometer, after obtaining the predicted posture, the following operations can be further performed to obtain the above-mentioned first initial posture: feature matching can be performed based on the feature points extracted from the current frame image with the previous frame image to obtain matching features that match the previous frame image, and the matching features are used as observation constraints to update the predicted posture to obtain the above-mentioned first initial posture.

[0139] In some possible embodiments, the above-mentioned updating of the predicted posture to obtain the above-mentioned first initial posture can be achieved through the following detailed operations: using the chi-square distribution to check whether there are outliers between the feature matching between each image feature, so as to eliminate the outliers in the predicted posture and obtain the first initial posture corresponding to each image.

[0140] For example, the three-dimensional points corresponding to the collected objects are projected to each image through the internal and external parameters of the visual sensor to obtain the predicted position value of the feature point. The chi-square value is constructed based on the difference between the predicted position value of the feature point and the observed position value matching it in the image. The chi-square value is used to measure the degree of difference between the predicted position value of the feature point and the observed position value matching it in the image; the corresponding critical value is found in the chi-square distribution table according to the degrees of freedom and the significance level, and the calculated chi-square value is compared with the critical value. If the chi-square value is less than the critical value, it means that there are no abnormal values ​​in the data. If the chi-square value is greater than the critical value, it means that there are abnormalities in the data and the corresponding data needs to be eliminated. Among them, the degrees of freedom are determined by the number of different groups divided into which the data for the chi-square test is performed; the significance level is a custom preset threshold, which indicates the degree of error acceptance allowed during the test process; the values ​​in the chi-square distribution table are statistically calculated and pre-calibrated, and different degrees of freedom and significance levels correspond to different critical values.

[0141] Furthermore, if the chi-square value is greater than the critical value, it means that there are anomalies in the data. In this case, it is necessary to eliminate the abnormal observation values ​​of the three-dimensional feature points of the collected object in the corresponding image, and only use the normal observation values ​​corresponding to the three-dimensional feature points in two adjacent frames of images to calculate the relative pose, so as to ensure a more accurate first initial pose.

[0142] In some possible embodiments, when the visual sensor in the positioning system only collects image data, the above-mentioned first initial pose can be obtained in the following manner: according to the feature points extracted from the current frame image, feature matching is performed with the previous frame image, and the calculated pose of the current frame image is calculated based on the matching features of the current frame image and the previous frame image; the three-dimensional feature points corresponding to the collected object in the world coordinate system are projected to each image through the internal and external parameters of the camera to obtain the predicted position value of the feature point, and the feature matching of the current frame image and the previous frame image is used as the observation constraint, and the difference between the predicted position value based on the feature point and the observed position value matching therewith in the image is calculated through a preset optimization function to obtain the error value; it is determined whether the error value is less than a preset threshold, and if it is less than the preset threshold, the calculated pose is used as the first initial pose of the current image; if the error value is greater than the preset threshold, the observation value of the feature point is eliminated, and the calculated pose of the current frame image is iteratively optimized, and the above steps are repeated until the error value obtained is less than the preset threshold, and the calculated pose corresponding to the error value is used as the first initial pose of the current frame image.

[0143] In some possible embodiments, the second initial pose can be obtained as follows: when there is a visual marker in the image acquired by the visual sensor, the corner points of the visual marker in the image in the visual marker camera coordinate system are extracted as 2D observation points. Since the world coordinates of each corner point of the visual marker in the visual marker world coordinate system are known, the second initial pose of the visual sensor in the visual marker world coordinate system G can be calculated through the perspective-n-point (PNP) algorithm. Among them, t0 represents the initial time corresponding to the first detection of the visual sign.

[0144] After obtaining the first initial pose of the visual sensor And the second initial pose After that, the relative position of the visual landmark coordinate system and the world coordinate system of the positioning system can be calculated based on the initial position of the visual sensor in different world coordinate systems. Right now

[0145] Get the relative position of the visual mark world coordinate system and the positioning system world coordinate system Then, the mobile device is controlled to move toward the visual mark. During the movement, the mobile position of the visual sensor in the world coordinate system of the visual mark is continuously obtained. For example, the mobile position obtained at time t is The distance between the visual sensor and the visual marker is calculated based on the mobile posture. If the distance is within the preset distance threshold range (such as the distance on the z-axis is within the range of 0.45 meters), the relative posture of the visual marker world coordinate system G and the positioning system world coordinate system W is used. Calculate the corrected position of the visual sensor in the world coordinate system of the positioning system at time t The positioning information of the positioning system at time t is obtained to achieve repositioning; if the distance is not within the preset distance threshold range, the positioning system is continued to be controlled to move until the distance between the visual mark and the visual sensor is within the preset distance threshold range.

[0146] Furthermore, in the process of controlling the positioning system to move toward the visual mark, it is also possible to judge whether the acquisition field of view of the visual sensor in the positioning system is facing the visual mark based on the initial position of the visual sensor in the positioning system in the world coordinate system of the visual mark. If the two are not facing each other, the angle that the positioning system needs to rotate is calculated based on the initial position, so as to control the positioning system to rotate according to the rotation angle so that the acquisition field of view of the visual sensor in the positioning system faces the visual mark, and then control the positioning system to move directly toward the visual mark, that is, rotate the positioning system so that the optical axis of the visual sensor coincides with any coordinate axis in the world coordinate system of the visual mark, so that the distance between the visual mark and the visual sensor can be directly obtained through the coordinates of the coordinate axis, which is beneficial to reduce the amount of calculation.

[0147] Based on the repositioning mechanism proposed in the embodiment of the present application, the embodiment of the present application further proposes another positioning system 100, which can implement the above-mentioned repositioning mechanism.

[0148] Fig.18 It is a schematic diagram of the framework of another positioning system 100 proposed in the embodiment of the present application. The premise of realizing the repositioning function of the following functional modules is that the first environment image acquired by the visual sensor should include at least one marker, which can be any object with a marking function, such as a signboard, a sign post, etc., or an object including a special mark, such as a triangular cone, a crash bucket, a charging post, etc. including a mark.

[0149] refer to Fig.18 As shown, compared to Figure 1 The positioning system 100 shown, Fig.18 The positioning system 100 shown also includes: a posture sensor; wherein the posture sensor is used to record the state measurement of the mobile device equipped with the positioning system while the visual sensor collects each frame of image data; the main control chip is used to obtain the working status of the visual sensor and the posture sensor, and configure the positioning system to enter the corresponding working mode according to the working status of the visual sensor and the posture sensor, and the working mode includes a normal fusion mode or a repositioning mode.

[0150] When the working mode is the normal fusion mode, the first environment image captured by the visual sensor and the state measurement obtained by the posture sensor are obtained, the predicted state variables are determined according to the first environment image, and the positioning information is obtained using the state measurement and the predicted state variables; when the working mode is the repositioning mode, the first environment image captured by the visual sensor is obtained and it is detected whether there is a marker in the first environment image. If so, repositioning is performed according to the position of the marker in the first environment image to obtain the positioning information of the mobile device; if not, the mobile device is controlled to rotate until the visual sensor captures an image including the marker.

[0151] Furthermore, the main control chip is also used to obtain a first initial posture of the visual sensor in the positioning system in the world coordinate system of the positioning system, and a second initial posture of the visual sensor in the world coordinate system of the marker, and to obtain a relative posture in the world coordinate system of the positioning system and the world coordinate system of the visual marker by combining the first initial posture and the second initial posture; to control the mobile device to continuously move toward the marker and obtain the moving posture of the visual sensor in the world coordinate system of the marker, and to obtain the distance between the mobile device and the marker according to the moving posture; if the distance is within a preset distance threshold range, the corrected posture of the positioning system in the world coordinate system of the positioning system is calculated using the moving posture and the relative posture in the world coordinate system of the positioning system and the world coordinate system of the marker, and the positioning information of the mobile device is obtained to achieve repositioning; if the distance is not within the preset distance threshold range, the mobile device is continued to be controlled to move toward the marker until the distance between the mobile device and the marker is within the preset distance threshold range to obtain the positioning information of the mobile device.

[0152] In one embodiment, a marker is installed at a reference position. When the visual sensor is in a failed working state, the positioning system is configured to enter a repositioning mode. At this time, the main control chip is also used to control the mobile device to move near the reference position, and control the visual sensor to capture images to detect whether there is a marker in the image captured by the visual sensor; if there is a marker in the image, the posture of the mobile device is changed so that the lens of the visual sensor is facing the marker; and the mobile device is controlled to drive straight toward the marker, and based on the position of the marker in the front and rear sets of images, the repositioning mode is executed on the mobile device; if there is no marker in the image, the mobile device body is rotated until the visual sensor captures an image including the marker.

[0153] It should be noted that the corresponding operations of the above functional modules can be performed under the condition that the fusion positioning timeout is met; the specific execution method of the main control chip is detailed in Fig.17 The contents described in the corresponding embodiments will not be described in detail in this embodiment.

[0154] Fig.19 It is a marker pattern proposed in the embodiment of the present application. It should be noted that the marker pattern in the embodiment of the present application can be as follows: Fig.19 The QR code or other iconic graphics shown in the figure can be installed on the markers in the venue, such as charging piles, triangular cones, etc., or can be spread around the application environment. When there are multiple visual signs, the positioning system can return the corresponding marker position based on the principle of proximity.

[0155] Based on the above technical solution, the repositioning function of the positioning system 100 is realized. By deploying markers on the site, when the fusion positioning times out during the operation of the mobile device, the mobile device can be autonomously controlled to return to the vicinity of the marker and perform visual repositioning correction to achieve the purpose of reducing the cumulative positioning error, thereby effectively avoiding the cumulative error generated by the visual sensor during use, or the failure of the visual sensor due to factors such as weather and obstructions, which leads to inaccurate positioning of the positioning system 100, thereby further improving the positioning accuracy of the positioning system 100.

[0156] When the mobile device is a lawn mower and the positioning system 100 is applied to the lawn mower to realize the perception of the surrounding environment and mowing along the edge, the existing technical solutions usually rely on other sensors, such as ultrasonic sensors, infrared sensors or laser radars, to detect obstacles and boundaries in the surrounding environment, so as to guide the lawn mower robot to mow along the predetermined planned path. However, these sensors are expensive and large in size, which not only increases the overall cost of the lawn mower, but also makes it difficult to integrate into small or portable lawn mowers, limiting the flexibility of equipment design; in addition, the performance of some sensors decreases under certain environmental conditions (such as rainy days or low-light environments), thereby affecting the efficiency and accuracy of mowing.

[0157] In view of this, an embodiment of the present application proposes a control method, which is applied to a lawn mower equipped with the above-mentioned positioning system 100 to achieve complex control of the lawn mower, such as achieving complex functions such as mowing along the edge and mowing across areas.

[0158] Fig. 20 2000 is a flow chart of a control method 2000 proposed in an embodiment of the present application. Fig. 20 As shown, the method 2000 may include the following steps:

[0159] S2010: Acquire a second environment image, the second environment image including a first area, and the lawn mower is located in the first area. In some possible embodiments, the second environment image may be a top view image from a bird's eye view perspective.

[0160] S2020: Extract the ROI in the second environment image to obtain a ROI image. In some possible embodiments, the ROI area may be an area within a preset range in front of the lawn mower.

[0161] S2030: Extract features from the ROI image to obtain first feature information, where the first feature information is used to indicate that the first area includes a lawn area and / or a non-lawn area.

[0162] In some possible embodiments, the non-lawn area may include an obstacle area protruding into the lawn boundary or an area outside the lawn boundary, wherein the obstacle area protruding into the lawn boundary may include an area where stones, flower beds or other types of obstacles are located.

[0163] S2040: Determine a moving path of the lawn mower according to the first feature information.

[0164] In some possible embodiments, when the first characteristic information is used to indicate an obstacle area, the lawn mower can be controlled to rotate in place to bypass the non-lawn area based on the above moving path. When the first characteristic information is used to indicate an area outside the lawn boundary, the lawn mower can be controlled to walk along the boundary diameter based on the above moving path. In summary, the moving path can be an edge path along the lawn boundary and avoiding boundary obstacles.

[0165] In some possible embodiments, when the first characteristic information is used to indicate that the first area includes a non-lawn area, the method 2000 may further perform the following operations:

[0166] S2050: Acquire multiple contour points of the non-lawn area.

[0167] S2055: Select any two points from the plurality of contour points as the starting point and the ending point to construct a first straight line.

[0168] S2060: Determine a first maximum vertical distance between the remaining contour points in the plurality of contour points and the first straight line.

[0169] S2065: When the first maximum vertical distance is less than or equal to the first distance threshold, determine multiple contour points for indicating the lawn boundary, and go to S2080; or, when the first maximum vertical distance is greater than the first distance threshold, determine multiple contour points for indicating obstacles, and go to S2070.

[0170] S2070: Construct a second straight line using a first contour point having a maximum vertical distance to the first straight line among the plurality of contour points and a starting point.

[0171] S2071: Determine the second maximum vertical distance between the remaining contour points in the plurality of contour points and the second straight line.

[0172] S2072: When the second maximum vertical distance is less than or equal to the first distance threshold, determine the outline of the obstacle according to the plurality of outline points, the second straight line, and the vertical distances between the plurality of outline points and the second straight line.

[0173] S2073: Determine a first path according to the outline of the obstacle, where a minimum vertical distance between the first path and the outline of the obstacle is greater than or equal to a second distance threshold.

[0174] In some possible embodiments, if the above-mentioned first maximum vertical distance is greater than a preset distance threshold, the contour point corresponding to the first maximum vertical distance is used as a new endpoint, and is used together with the starting point to form a new straight line segment, i.e., a second straight line. The vertical distances between the remaining contour points and the second straight line are repeatedly calculated to obtain the second maximum vertical distance. The second maximum vertical distance is compared with the preset distance threshold, and an approximate straight line segment is obtained or a new straight line segment is reorganized based on the comparison result until the straight line distances from all contour points to the straight line segment are less than the preset distance threshold, so as to obtain the obstacle contour, and the current non-lawn local area is regarded as an obstacle area protruding into the lawn boundary, thereby forming a moving path along the obstacle boundary.

[0175] It should be understood that this embodiment can reduce the number of contour points and generate a moving path through polygonal approximation processing, which can not only improve the efficiency of generating the moving path, but also ensure the accurate matching of the moving path with the actual boundary.

[0176] S2080: Determine the contour of the lawn boundary according to the plurality of contour points.

[0177] S2081: Determine a second path according to the contour of the lawn boundary, where the second path extends along the lawn boundary, and a minimum vertical distance between the second path and the contour of the lawn boundary is greater than or equal to a second distance threshold.

[0178] In some possible embodiments, in order to ensure the safety of the lawn mower during movement, after obtaining a moving path along a lawn boundary or a moving path along an obstacle boundary, a safety distance can be added on the basis of the moving path, so that the lawn mower can move as close to the boundary as possible while avoiding hitting obstacles during the movement.

[0179] In some possible embodiments, after the moving path of the lawn mower is determined, the following operations may be further performed:

[0180] S2090: Determine a target point according to the second environment image, where the position of the target point is related to the viewing distance of the visual sensor.

[0181] S2091: Determine the steering angle and moving speed of the lawn mower according to the positioning information and the target point.

[0182] S2092: Control the movement of the lawn mower based on the steering angle and the moving speed.

[0183] In some possible embodiments, the above positioning information indicates that the distance between the current position of the lawn mower and the position of the target point is positively correlated with the moving speed. For example, when the lawn mower moves along the moving path, it will first determine the target point at the next moment on the moving path, that is, the position point that the lawn mower needs to reach at the next moment. When the distance between the current position of the lawn mower and the target point is too close, the moving speed is slowed down; when the distance between the current position of the lawn mower and the target point is too far, the moving speed is accelerated.

[0184] In some possible embodiments, the target point at the next moment can be obtained in the following manner: obtain a preset sight distance, which is a fixed distance forward from the current position of the lawn mower; construct a circle with the current position as the center and the preset sight distance as the radius, and calculate the intersection of the circle and the moving path to obtain the target position point at the next moment. When the lawn mower moves along the moving path of the boundary, the target position point will be updated frequently. In order to prevent the lawn mower from suddenly changing direction when moving and to ensure that the lawn mower always moves in the direction of the specified moving path, the target position points in the two frames of images before and after the adjacent moments can be subjected to sliding window filtering, that is, the target position points in the two frames of images before and after the adjacent moments are averaged to smooth the jumping changes of the target position points at different moments, so that the visual sensor can work stably under different lighting conditions, is not significantly affected by weather changes, and adapts to a variety of environmental conditions, thereby enabling the lawn mower to maintain stable path tracking performance in a complex environment and reduce path deviations caused by external interference.

[0185] In some possible embodiments, after the target position point of the lawn mower at the next moment is obtained, the deviation (such as lateral deviation and heading deviation) between the current position of the lawn mower and the target position point is calculated according to the coordinates of the current position of the lawn mower and the coordinates of the target position point to obtain the steering angle required for the lawn mower wheels, and a control signal is generated based on the steering angle and the moving speed and sent to the lawn mower wheels to control the lawn mower wheels to move at the set speed and steering direction, thereby facilitating the lawn mower to move along the target position point.

[0186] It should be understood that the above-mentioned safety distance and sight distance can be dynamically adjusted according to factors such as the moving speed of the lawn mower and the complexity of the environment, thereby flexibly adjusting the path tracking points to improve the smoothness and accuracy of the path tracking.

[0187] Based on the above technical solution, the second environment image acquired by the visual sensor can accurately identify the lawn boundary and ensure that the lawn mower runs efficiently along the boundary. On the one hand, by judging the location of the non-grass area, the lawn mower can flexibly adjust its direction to avoid miscutting and repeated mowing, thereby improving work efficiency; on the other hand, based on the visual perception technology, not only the installation cost and maintenance cost of the overall lawn mower system are reduced, but also the system automatically generates the path along the edge and performs real-time trajectory tracking through the images collected by the visual perception technology, without the need for complex sensor fusion technology, while ensuring that the overall system maintains a high level of automation, reducing manual intervention and improving ease of operation.

[0188] In practical applications, the lawn area can be distributed in patches, or it can be divided into multiple areas by non-lawn areas, and the multiple areas are not adjacent to each other, that is, when the lawn mower completes work in one of the adjacent lawn areas, it needs to move to another lawn area to continue working. However, in the existing cross-area mowing technical solutions, the RFID coil solution is usually used to complete the cross-area task. For example, the RFID coil is buried in the area where the lawn mower actually needs to be mowed in advance, and a sensor specifically used to identify the RFID is set on the lawn mower. When in use, the sensor identifies the wire to determine different lawn areas. This solution of burying RFID coils and using sensors to identify the mowing area is not only cumbersome and labor-intensive, but also requires a large number of wires to be deployed, and requires strong professionalism of the deployment personnel. In addition, the installation and fixation of RFID will damage the lawn and cost a lot. In addition, the RFID is hidden in the grass, and the cross-area indication is not obvious to the user. When the sensor passes directly above the RFID, there will be a feedback signal output to guide the lawn mower to cross the area. The efficiency is low and cannot meet the intelligent needs of the robot equipment.

[0189] In view of this, an embodiment of the present application proposes another control method, which indicates the subsequent moving direction of the lawn mower based on a pre-set indicator, thereby realizing a cross-zone mowing function.

[0190] Fig.21 2 is a flow chart of a control method 2100 proposed in an embodiment of the present application. In the application scenario of the control method 2100, the lawn mower also includes a visual sensor, and the lawn mower is located in the first lawn area where the first indicator is placed, and the lawn mower is moving along the third path, that is, the method 2100 is executed when the lawn mower moves along the third path. It should be understood that the third path can be the first path or the second path, or other paths different from the first path and the second path.

[0191] refer to Fig.21 As shown, the method 2100 may include the following steps:

[0192] S2110: In response to a first instruction instructing the lawn mower to move to a second lawn area, controlling the visual sensor to collect a third environmental image of the location of the lawn mower, where the third environmental image includes environmental information of the location of the lawn mower.

[0193] In some possible embodiments, when the first instruction is received, the visual sensor may be controlled, or the mobile device may be controlled to rotate in situ to obtain the third environment image.

[0194] S2120: Detect whether the third environmental image includes the first indicator. When the third environmental image includes the first indicator, obtain the fourth path according to the first indicator; the first indicator is associated with the first lawn area and is used to indicate a first direction, the first direction is the direction of the second lawn area relative to the first indicator, and the fourth path passes through the first indicator and extends in the first direction; or, when the third environmental image does not include the first indicator, control the lawn mower to rotate so that the visual sensor collects the third environmental image including the first indicator, thereby obtaining the fourth path to achieve cross-area movement.

[0195] In some possible embodiments, when multiple lawn areas are separated by cross-area passages, multiple indicators (including the first indicator) with unique identifications may be set in the cross-area passages, and further, may be located at the edges of the lawn areas at the same time.

[0196] In some possible embodiments, when the third environmental image includes a first indicator, the third environmental image can be processed and analyzed so as to obtain the position of the first indicator relative to the lawn mower, and based on the position of the first indicator relative to the lawn mower, guide the lawn mower to move to the starting point of the cross-zone where the first indicator is located, so as to pass through the non-lawn area and enter another lawn area according to the indication of the first indicator, that is, control the lawn mower to move along the fourth path mentioned above.

[0197] In some possible embodiments, when it is determined that the third environmental image does not include the first indicator, the lawn mower can be controlled to continue to perform the edge mowing task, that is, continue to move along the third path until the first indicator or other indicators are detected in the image captured by the visual sensor, and then use the captured image to obtain the position of the indicator relative to the lawn mower to guide the lawn mower to move to the starting point of the cross-zone where the indicator is located, so as to pass through the non-lawn area and enter another lawn area according to the indication of the indicator.

[0198] In some possible embodiments, the fourth path may be determined in the following manner:

[0199] S2121: Acquire a first conversion relationship between a third coordinate system established based on the visual sensor and a fourth coordinate system established based on the lawn mower.

[0200] S2122: Obtain a first coordinate of the first indicator in a third coordinate system corresponding to the third environment image and a second coordinate of the first indicator in a world coordinate system constructed based on the first indicator.

[0201] S2123: Determine a second conversion relationship between the third coordinate system and the world coordinate system according to the first coordinate and the second coordinate of the first indicator.

[0202] In some possible embodiments, the first indicator includes a first graphic code, and the first graphic code is used to indicate the second coordinate and the first direction.

[0203] In some possible embodiments, when the first indicator includes a first graphic code, the second coordinate may refer to the coordinate of the first indicator graphic code in the world coordinate system.

[0204] S2124: Determine a third transformation relationship between the fourth coordinate system and the world coordinate system according to the first transformation relationship and the second transformation relationship.

[0205] S2125: Determine a fourth path for instructing the lawn mower to move across zones according to the third conversion relationship.

[0206] Fig. 22 It is a schematic diagram of a principle for determining a third conversion relationship proposed in an embodiment of the present application.

[0207] refer to Fig. 22 As shown, it is assumed that the visual sensor is fixedly installed on the lawn mower, that is, the relative position between the visual sensor and the lawn mower should be fixed and known, so the first conversion relationship T_C2R between the third coordinate system and the fourth coordinate system is known.

[0208] When the visual sensor captures an image including the first indicator, by identifying the first coordinate of the first graphic code in the first indicator in the image in the third coordinate system, and the world coordinate of the first graphic code in the world coordinate system (i.e., the above-mentioned second coordinate) is also known, the second conversion relationship, i.e., T_M2C, can be determined based on the first coordinate of the first graphic code and the above-mentioned second coordinate.

[0209] Furthermore, based on the fact that the first transformation relationship T_C2R between the third coordinate system and the fourth coordinate system is known, combined with the second transformation relationship T_M2C, the third transformation relationship T_M2R between the fourth coordinate system and the world coordinate system can be determined, and then the relative position of the first indicator relative to the visual sensor can be determined based on the third transformation relationship.

[0210] exist Fig. 22In the figure, {M} system represents the world coordinate system, the direction indicated by the arrow is the X-axis, the Z-axis is perpendicular to the arrow plane and faces upward, the Y-axis is parallel to the arrow plane and perpendicular to the direction of the arrow, and the origin is the center of the arrow plane; {C} system represents the third coordinate system. From the rear of the visual sensor, it can be observed that the XYZ three-axis direction of the third coordinate system is "lower right front", and the origin is the structural center of the visual sensor; {R} system represents the fourth coordinate system, the forward direction of the lawn mower is set to the X-axis, the Z-axis is perpendicular to the lawn mower wheel surface and faces upward, the Y-axis is obtained by the right-hand rule, and the origin is the center of the rear wheelbase of the lawn mower; the final visual positioning output posture information is "T_M2R", which represents the posture of the {M} system in the {R} system, the translation part is output as three vectors "xyz", unit: m (meter), and the rotation part is output as quaternion "wxyz".

[0211] Fig.23 This is a design diagram of an indicator proposed in an embodiment of the present application.

[0212] refer to Fig.23 As shown, a single indicator may include four detection surfaces, each surface having a unique coding pattern. The appearance of the indicator is not limited to a cube, and may also be set to a single-sided, double-sided, or other three-dimensional shape.

[0213] In some possible embodiments, the above-mentioned coding pattern may use AprilTag (a visual positioning marker), or a QR code or a barcode, or any other coding pattern with a unique identifier (ID), such as a binary circular random coding pattern, a four-value circular random coding pattern, etc.; in addition, identifiable coding patterns such as checkerboards and circular arrays may also be used.

[0214] For example, taking the AprilTag 36h11 family pattern as an example, there are 587 pattern ids (0-586), which support the design of 587 / 4≈146 unique indicators, indicator ids (0-145), which are arranged on each surface of the indicator using AprilTag according to certain design rules; among them, if the indicator id is known, the ids of the coded patterns on each surface of the indicator can be expressed as: id*4, id*4+1, id*4+2, id*4+3;

[0215] The arrow on the top surface of the indicator indicates the direction of motion of the lawn mower. Looking in the direction of the arrow on the top surface, the front, right, back, and left correspond to the ids of the above-mentioned surface coding patterns in sequence. For example, when the visual sensor collects the id*4 coding pattern of a certain indicator, according to the mapping relationship between the coding pattern and the top surface arrow, it indicates forward, so the lawn mower is controlled to move forward into another lawn area; when the visual sensor collects the id*4+1 coding pattern of the indicator, according to the mapping relationship between the coding pattern and the top surface arrow, it indicates left, so the lawn mower is controlled to move left into another lawn area, and so on.

[0216] In some possible embodiments, the second lawn area includes a second indicator. During the process of the lawn mower moving along the fourth path, the method 2100 further includes the following steps: Fig.21 Not shown):

[0217] S2130: Control the visual sensor to collect a fourth environment image of the location of the lawn mower.

[0218] S2140: Detect whether the fourth environmental image includes the second indicator. When the fourth environmental image includes the second indicator, obtain a fifth path according to the second indicator, the second indicator is associated with the second lawn area, the starting point of the fifth path is located on the fourth path, the end point is located on the second lawn area, and the fifth path passes through the second indicator; or, when the fourth environmental image does not include the second indicator, control the lawn mower to continue moving along the fourth path until the second indicator is included in the fourth environmental image.

[0219] In some possible embodiments, when it is determined that the lawn mower has reached the second lawn area based on the images collected by the visual sensor, the working state of the lawn mower moving across regions may be reset to wait for the next instruction to start the cross-region task.

[0220] The above embodiment only describes the scenario of dual-zone cross-zone. In addition, the embodiment of the present application also proposes a cross-zone solution for multiple areas and multiple entry and exit of lawn mowers.

[0221] In some possible embodiments, the first lawn area may further include a third indicator, which is associated with the third lawn area and is used to indicate a second direction, which is a direction of the third lawn area relative to the third indicator.

[0222] In some possible embodiments, before controlling the visual sensor to collect the third environmental image of the location of the lawn mower, the following operations can also be performed: setting the number of lawn areas to be worked on, the number of indicators, and the number of indicators included in each lawn area, and initializing the lawn areas and indicators, that is, assigning ID numbers to the lawn areas and indicators, and assigning indicators to each lawn area, that is, associating the indicator numbers with the lawn area codes, and setting the earlier numbered (for example, the first) indicator in each area as the indicator of the default working path; defining the lawn area where the lawn mower is located, the ID of the indicator to be detected, the forward / reverse flag, and the default cross-zone path, and updating the forward / reverse flag in the cross-zone status maintenance according to the current area where the lawn mower is located, the ID of the indicator to be detected, and the default cross-zone path.

[0223] Fig.24 It is a schematic diagram of a multi-span application scenario of a lawn mower proposed in an embodiment of the present application.

[0224] In some possible embodiments, when the lawn mower performs a lawn mowing task, the next area and the ID of the indicator to be detected may be determined according to the forward / reverse flag of the indicator and the default cross-area path until all areas are traversed.

[0225] refer to Fig.24 As shown, taking the four lawn areas ABCD as an example, if the four areas need to be mowed, the default cross-area path is A>B>C>D>A, ensuring that the order of the areas and indicators is correct. When the lawn mower starts working from area A, the forward flag is true, and the id of the indicator to be detected is the first indicator id of area A (the indicator with the first number), that is, the forward cross-area is completed according to the default path sequence, and the next area is B. There are 10 indicators in this scene, distributed in four lawn areas, and the ids of the indicators are 0 to 9; if the lawn mower is in area B, the forward flag is false, the id of the indicator to be detected is the first indicator id of area B, and the reverse cross-area is completed according to the reverse default path (that is, B>A>D>C>B), and the next area is A.

[0226] It should be noted that, when the lawn mower completes cross-zone work in the forward direction or the reverse direction, the same indicator can be used as an indicator for exiting the area or entering the area; the default cross-zone path includes a closed-loop cross-zone path and a non-closed-loop cross-zone path. If the path involves all areas and the path forms a closed loop, such as the default cross-zone path is A>B>C>D>A, it is a closed-loop cross-zone path; if the path involves some areas, such as only areas A, B, and C, and the default cross-zone path is A>B>C>B>A, it is a non-closed-loop cross-zone path.

[0227] In some possible embodiments, the execution of the above multi-area, multi-input and multi-output cross-area control process needs to meet the following two conditions: the indicator number (id) is unique, and there is only one channel between the areas.

[0228] Based on the above technical solution, by using the indicator as a reference for visual positioning information, the lawn mower is guided to reach the cross-zone starting point and end point in sequence to complete the cross-zone task. Since the indicator has the characteristics of unique appearance, simple layout, and rich information, the random mowing lawn mower can quickly and conveniently complete the cross-zone task, and through the update of relevant data structures and states, the flexible cross-zone mowing function of multiple mowers and multiple cross-zones is realized.

[0229] The embodiment of the present application also proposes a control device, which includes a module or unit for executing any one of the control methods 2000 proposed in the embodiment of the present application.

[0230] Fig.25 2500 is a schematic diagram of a control device 2500 proposed in an embodiment of the present application. The device 2500 is applied to a lawn mower including any positioning system 100 proposed in an embodiment of the present application, Fig.25 As shown, the device 2500 includes: an acquisition unit 2510, used to acquire a second environment image, the second environment image includes a first area, and the lawn mower is located in the first area; an image processing unit 2520, used to extract a region of interest ROI in the second environment image to determine the ROI image; a determination unit 2530, used to perform feature extraction on the ROI image to determine first feature information, the first feature information is used to indicate that the first area includes a lawn area and / or a non-lawn area; and based on the first feature information, determine the moving path of the lawn mower.

[0231] In some possible embodiments, when the first feature information is used to indicate that the first area includes a non-lawn area, the acquisition unit 2510 is further used to: acquire coordinate information of multiple contour points of the non-lawn area; the determination unit 2530 is further used to: select any two of the multiple contour points as the starting point and the ending point and construct a first straight line; determine the first maximum vertical distance between the remaining contour points of the multiple contour points and the first straight line; when the first maximum vertical distance is less than or equal to the first distance threshold, determine that the multiple contour points are used to indicate a lawn boundary; or, when the first maximum vertical distance is greater than the first distance threshold, determine that the multiple contour points are used to indicate an obstacle.

[0232] In some possible embodiments, when the first maximum vertical distance is greater than the first distance threshold, the above-mentioned determination unit 2530 is further used to: construct a second straight line using a first contour point among multiple contour points that has a maximum vertical distance with the first straight line and a starting point; determine a second maximum vertical distance between the remaining contour points among the multiple contour points and the second straight line; when the second maximum vertical distance is less than or equal to the first distance threshold, determine the contour of the obstacle based on the multiple contour points, the second straight line, and the vertical distances between the multiple contour points and the second straight line; determine a first path based on the contour of the obstacle, and the minimum vertical distance between the first path and the contour of the obstacle is greater than or equal to the second distance threshold.

[0233] In some possible embodiments, when the first maximum vertical distance is less than or equal to the first distance threshold, the above-mentioned determination unit 2530 is also used to: determine the contour of the lawn boundary based on multiple contour points; determine a second path based on the contour of the lawn boundary, the second path extends along the lawn boundary, and the minimum vertical distance between the second path and the contour of the lawn boundary is greater than or equal to the second distance threshold.

[0234] In some possible embodiments, the determination unit 2530 is further used to: determine a target point according to the second environment image, wherein the position of the target point is related to the visual distance of the visual sensor; and determine the steering angle and moving speed of the mobile device according to the positioning information and the target point. The device 2500 further includes a control unit 2540, which is used to control the movement of the lawn mower based on the steering angle and the moving speed.

[0235] In some possible embodiments, the distance between the current position of the lawn mower indicated by the positioning information and the position of the target point is positively correlated with the moving speed.

[0236] In addition, the embodiment of the present application also proposes another control device, which includes a module or unit for executing any one of the control methods 2100 proposed in the embodiment of the present application. The device no longer elaborates on the implementation of each unit. The detailed implementation functions of each module can be found in the embodiment of the above-mentioned control method 2100.

[0237] Fig.26 2600 is a schematic diagram of a control device 2600 proposed in an embodiment of the present application. The device 2600 is applied to a lawn mower including any one of the positioning systems 100 proposed in an embodiment of the present application, the lawn mower includes a visual sensor, the lawn mower is located in a first lawn area, and the device 2600 performs corresponding functional actions during the lawn mower moves along a third path, referring to Fig.26As shown, the device 2600 includes: a control unit 2610, which is used to control the visual sensor to collect a third environment image of the location of the lawn mower in response to a first instruction instructing the lawn mower to move to the second lawn area, and the third environment image includes environmental information of the location of the lawn mower; a determination unit 2620, which is used to detect whether the third environment image includes a first indicator, when the third environment image includes the first indicator, obtain a fourth path according to the first indicator, the first indicator is associated with the first lawn area, and the first indicator is used to indicate a first direction, the first direction is the direction of the second lawn area relative to the first indicator, and the fourth path passes through the first indicator and extends in the first direction; or, when the third environment image does not include the first indicator, the control unit 2610 controls the lawn mower to continue moving along the third path.

[0238] An embodiment of the present application also proposes a control device, which includes a processor and a memory, where the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute any one of the control methods proposed in the embodiment of the present application.

[0239] In one embodiment, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0240] In one embodiment, the memory may be an internal storage unit of the control device, such as a hard disk or a memory, or an external storage device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory may include both an internal storage unit and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, etc., and may also be used to temporarily store data that has been output or is to be output.

[0241] The embodiment of the present application also proposes a lawn mower, including any one of the positioning systems proposed in the embodiment of the present application, and any one of the control devices proposed in the embodiment of the present application.

[0242] The present application also provides a computer-readable storage medium storing a computer program, which is executed by a processor to perform any one of the control methods proposed in the embodiments of the present application.

[0243] The present application also provides a computer program product, including instructions, which, when executed by a processor, enable a computer to execute any one of the control methods proposed in the embodiments of the present application.

[0244] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0245] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0246] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0247] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0248] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0249] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0250] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A positioning system, characterized in that: The positioning system is applied to a mobile device, and the positioning system comprises: A visual sensor, used to collect a first environment image of the location of the mobile device, wherein the first environment image is used to obtain a state measurement of the mobile device; an odometer, used to obtain motion parameters of the mobile device, the motion parameters including a first acceleration and / or a first speed, and the motion parameters are used to predict a first state variable of the mobile device; A main control chip is used to determine a first confidence level according to a preset confidence model and the first state variable, wherein the first confidence level is used to indicate whether the wheels of the mobile device are slipping; obtain a second state variable using the first confidence level and the first state variable, and determine the positioning information of the mobile device according to the second state variable and the state measurement.

2. The positioning system according to claim 1, characterized in that: The confidence model is used to indicate the confidence corresponding to the first state variable obtained by the mobile device at different speeds.

3. The positioning system according to claim 1 or 2, characterized in that: The odometer includes an inertial odometer and / or a wheel odometer, wherein the inertial odometer is used to obtain the first acceleration, the first acceleration includes linear acceleration and / or angular acceleration, and the wheel odometer is used to obtain the first speed, the first speed includes linear speed and / or angular speed.

4. The positioning system according to claim 3, characterized in that: The determining the first confidence according to the preset confidence model and the first state variable includes: Acquire the static power of the motor when the mobile device is stationary, the first change slope of the driving power of the motor and the wheel speed when the mobile device is moving, and the second change slope of the driving power and the acceleration; Determine the slipping power when the wheels of the mobile device slip according to the static power, the driving power, the first change slope, the second change slope and a preset linear equation; Acquire the actual operating power of the motor satisfying the normal distribution when the mobile device is running; The slip power is used as a reference value, and the confidence of the first state variable is obtained in combination with the actual operating power and the confidence model.

5. The positioning system according to claim 1 or 2, characterized in that: Applied in an environment including a marker, the positioning system is mounted on a mobile device, and the positioning system also includes a posture sensor, wherein the main control chip is also used for: Acquire the working status of the visual sensor and the posture sensor, and configure the positioning system to enter a corresponding working mode according to the working status of the visual sensor and the posture sensor, wherein the working mode includes a normal fusion mode or a repositioning mode; When the working mode is the normal fusion mode, a first environment image collected by a visual sensor and a state measurement obtained by a posture sensor are obtained, a predicted state variable is determined according to the first environment image, and positioning information is obtained using the state measurement and the predicted state variable; When the working mode is the relocation mode, acquiring a first environment image captured by a visual sensor and detecting whether there is a marker in the first environment image; If it exists, repositioning is performed according to the position of the marker in the first environment image to obtain positioning information of the mobile device; if it does not exist, the mobile device is controlled to rotate until the visual sensor captures an image including the marker.

6. The positioning system according to claim 4, characterized in that: The repositioning according to the position of the marker in the first environment image to obtain positioning information of the positioning system includes: Acquire a first initial pose of the visual sensor in the positioning system in the world coordinate system of the positioning system, and a second initial pose of the visual sensor in the world coordinate system of the marker, and combine the first initial pose and the second initial pose to obtain a relative pose in the world coordinate system of the positioning system and the world coordinate system of the visual marker; Controlling the mobile device to continuously move toward the marker and obtaining a moving posture of the visual sensor in a world coordinate system of the marker, and obtaining a distance between the mobile device and the marker according to the moving posture; If the distance is within a preset distance threshold range, the corrected posture of the positioning system in the world coordinate system of the positioning system is calculated by using the mobile posture and the relative posture in the world coordinate system of the positioning system and the world coordinate system of the marker, and the positioning information of the mobile device is obtained to achieve repositioning; If the distance is not within the preset distance threshold range, the mobile device is continuously controlled to move toward the marker until the distance between the mobile device and the marker is within the preset distance threshold range to obtain the positioning information of the mobile device.

7. The positioning system according to claim 6, characterized in that: Acquiring a first initial position and posture of the visual sensor in the positioning system in the world coordinate system of the positioning system includes: Obtaining motion parameters corresponding to the previous frame image recorded by the odometer and processing the motion parameters to obtain a predicted position and posture of the current frame image; The feature points extracted from the current frame image are matched with the previous frame image to obtain matching features that match the previous frame image. The matching features are used as observation constraints to update the predicted posture to obtain the first initial posture of the positioning system when the current frame image is collected.

8. The positioning system according to any one of claims 5 to 7, characterized in that: The marker is installed at the reference position. When the visual sensor is in a failed working state, the main control chip is also used for: Controlling the moving device to move near the reference position, and controlling the visual sensor to collect images to detect whether there is a marker in the image collected by the visual sensor; If there is a marker in the image, the posture of the mobile device is changed so that the lens of the visual sensor is facing the marker; and the mobile device is controlled to drive directly toward the marker, and a repositioning mode is performed on the mobile device based on the position of the marker in the previous and next two sets of images; If there is no marker in the image, the mobile device body is rotated until the visual sensor captures an image including the marker.

9. A control method, characterized in that: Applied in the positioning system according to any one of claims 1 to 8, when the mobile device is a lawn mower, the method comprises: Acquire a second environment image including a first area, wherein the lawn mower is located in the first area; Extracting a region of interest ROI in the second environment image to obtain a ROI image; Extracting features from the ROI image to obtain first feature information, where the first feature information is used to indicate that the first area includes a lawn area and / or a non-lawn area; A moving path of the lawn mower is determined according to the first feature information.

10. The method according to claim 9, characterized in that In a case where the first characteristic information is used to indicate that the first area includes the non-lawn area, the method further includes: Acquire a plurality of contour points of the non-lawn area; Select any two points from the multiple contour points as the starting point and the ending point to construct the first straight line; Determine a first maximum vertical distance between the remaining contour points in the plurality of contour points and the first straight line; When the first maximum vertical distance is less than or equal to a first distance threshold, the plurality of contour points are determined to indicate a lawn boundary; or when the first maximum vertical distance is greater than the first distance threshold, the plurality of contour points are determined to indicate an obstacle.

11. The method according to claim 10, characterized in that When the first maximum vertical distance is greater than the first distance threshold, the method further includes: Constructing a second straight line by using a first contour point having a maximum vertical distance to the first straight line among the plurality of contour points and the starting point; determining a second maximum vertical distance between the remaining contour points in the plurality of contour points and the second straight line; When the second maximum vertical distance is less than or equal to the first distance threshold, determining the outline of the obstacle according to the plurality of outline points, the second straight line, and the vertical distances between the plurality of outline points and the second straight line; A first path is determined according to the outline of the obstacle, wherein a minimum vertical distance between the first path and the outline of the obstacle is greater than or equal to a second distance threshold.

12. The method according to claim 10, characterized in that When the first maximum vertical distance is less than or equal to the first distance threshold, the method further includes: Determining the contour of the lawn boundary according to the plurality of contour points; A second path is determined according to the contour of the lawn boundary, wherein the second path extends along the lawn boundary, and a minimum vertical distance between the second path and the contour of the lawn boundary is greater than or equal to a second distance threshold.

13. The method according to any one of claims 9 to 11, characterized in that The method further comprises: Determine a target point according to the second environment image, wherein the position of the target point is related to the viewing distance of the visual sensor; Determining a steering angle and a moving speed of the lawn mower according to the positioning information and the target point; The movement of the lawn mower is controlled based on the steering angle and the moving speed.

14. The method according to claim 13, characterized in that The positioning information indicates that the distance between the current position of the lawn mower and the position of the target point is positively correlated with the moving speed.

15. A control method, characterized in that: Applied to the positioning system according to any one of claims 1 to 8, when the mobile device is a lawn mower, the lawn mower is located in a first lawn area where a first indicator is placed, and during the movement of the lawn mower along a third path, the method comprises: In response to a first instruction instructing the lawn mower to move to a second lawn area, controlling the visual sensor to collect a third environmental image of the location of the lawn mower, wherein the third environmental image includes environmental information of the location of the lawn mower; detecting whether the third environment image includes a first indicator, and when the third environment image includes the first indicator, acquiring a fourth path according to the first indicator, wherein the first indicator is associated with the first lawn area and the first indicator is used to indicate a first direction, the first direction is a direction of the second lawn area relative to the first indicator, and the fourth path passes through the first indicator and extends toward the first direction; or, When the third environment image does not include the first indicator, the lawn mower is controlled to rotate so that the visual sensor collects a third environment image including the first indicator.

16. The method according to claim 15, characterized in that The acquiring the fourth path according to the first indicator comprises: Acquire a first conversion relationship between a third coordinate system established based on the visual sensor and a fourth coordinate system established based on the lawn mower; Acquire a first coordinate of the first indicator in a third coordinate system corresponding to the third environment image and a second coordinate of the first indicator in a world coordinate system constructed based on the first indicator; determining a second conversion relationship between the third coordinate system and the world coordinate system according to the first coordinate and the second coordinate of the first indicator; Determining a third conversion relationship between the fourth coordinate system and the world coordinate system according to the first conversion relationship and the second conversion relationship; The fourth path for instructing the lawn mower to move across zones is determined according to the third conversion relationship.

17. The method according to claim 16, characterized in that The first indicator includes a first graphic code, and the first graphic code is used to indicate the second coordinate and the first direction.

18. The method according to any one of claims 15 to 17, characterized in that The second lawn area includes a second indicator, and during the movement of the lawn mower along the fourth path, the method further includes: Controlling the visual sensor to collect a fourth environment image of the location of the lawn mower; detecting whether the fourth environment image includes the second indicator, and when the fourth environment image includes the second indicator, acquiring a fifth path according to the second indicator, wherein the second indicator is associated with the second lawn area, the starting point of the fifth path is located on the fourth path, the ending point is located on the second lawn area, and the fifth path passes through the second indicator; or, When the fourth environment image does not include the second indicator, the lawn mower is controlled to continue moving along the fourth path until the fourth environment image includes the second indicator.

19. The method according to any one of claims 15 to 18, characterized in that The first lawn area further includes a third indicator, the third indicator is associated with the third lawn area, and the third indicator is used to indicate a second direction, and the second direction is a direction of the third lawn area relative to the third indicator.

20. The method according to claim 19, characterized in that Before controlling the visual sensor to collect the third environment image at the location of the lawn mower, the method further includes: Setting the number of lawn areas to be worked on, the number of indicators, and the number of indicators included in each lawn area, and assigning ID numbers to the lawn areas and the indicators, and associating the numbers of the indicators with the codes of the lawn areas; Set the first numbered indicator in each area as the indicator of the default working path; Define the lawn area where the lawn mower is located, the ID of the indicator to be detected, the forward / reverse flag and the default cross-area path, and update the forward / reverse flag in the cross-area status maintenance according to the current area where the lawn mower is located, the ID of the indicator to be detected and the default cross-area path.

21. A control device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program stored in the memory, so that the apparatus performs the method according to any one of claims 9 to 20.

22. A lawn mower, characterized in that: The method comprises the positioning system according to any one of claims 1 to 8, and the control device according to claim 21.

23. A computer-readable storage medium, characterized in that: Instructions are stored thereon, and when the instructions are executed by a processor, the processor implements the method according to any one of claims 9 to 20.

Images