Method and apparatus for initializing VIO system, and self-moving device

Abstract

A method (200) and apparatus for initializing a VIO system, and a system, which relate to the technical field of visual positioning navigation. The method (200) for initializing a VIO system comprises: S210, acquiring state information of a VIO system; S220, sending instruction information to a control apparatus of a self-moving device according to the state information, wherein the instruction information is used for instructing the control apparatus to control the self-moving device to be stationary first and then to start; and S230, when it is determined that the self-moving device has been stationary, performing a static initialization operation on the VIO system, so as to determine an initial value of the VIO system, wherein the state information comprises at least one piece of the following content: the VIO system initiating a first signal, which is used for indicating that the VIO system is about to initially start, or a data anomaly occurring during data processing by the VIO system, or a preset duration having elapsed since the last time when the static initialization operation was completed. In this way, static initialization conditions can be provided for a VIO system, so as to implement a static initialization operation for the VIO system, thereby facilitating the improvement in the efficiency of initialization.

Description

Method, device and self-mobile equipment for initializing VIO system

[0001] This application claims priority to the Chinese patent application with application number 202310897775.2 filed with the State Intellectual Property Office of China on July 20, 2023, and priority to the Chinese patent application entitled “VIO system initialization method, device and self-mobile device”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of visual positioning and navigation technology, and more specifically, to a method and device for initializing a VIO system and a self-moving device. Background Art

[0003] Visual-inertial odometry (VIO) can be divided into two categories: loosely coupled and tightly coupled. Tight coupling refers to combining the states of the image acquisition device and the inertial measurement unit (IMU) to construct the motion equations and observation equations, and then performing motion estimation in a unified manner. However, a tightly coupled VIO system is highly nonlinear, especially when the image acquisition device is a monocular camera. In applications such as vision-based real-time positioning and mapping, the detection scale and rotation relative to gravity of the monocular camera are unobservable. Therefore, a tightly coupled VIO system requires a good initial value when fusing visual data with IMU data. This initial value can be determined by initializing the VIO system.

[0004] In view of this, a method to implement VIO system initialization needs to be developed urgently.

[0005] Summary of the Invention

[0006] The present application provides a method, apparatus, and self-mobile device for initializing a VIO system, which can provide conditions for static initialization of the VIO system, enabling the VIO system to complete static initialization operations.

[0007] In a first aspect, a method for initializing a VIO system is provided, the method comprising: obtaining first status information of the VIO system; sending first indication information to a control device of a self-mobile device according to the first status information, the first indication information being used to instruct the control device to control the self-mobile device to first remain stationary for a first period of time and then start; when it is determined that the self-mobile device is stationary, performing a static initialization operation on the VIO system to determine an initial value of the VIO system; wherein the above-mentioned first status information includes at least one of the following contents: the VIO system initiates a first signal, the first signal being used to indicate that the VIO system is about to be initially started, or a data anomaly occurs during data processing by the VIO system, or a preset time has elapsed since the last time the static initialization operation was completed.

[0008] For example, the first time period is a preset reasonable time period, and the length of the time period is required to ensure that some operations for static initialization of the VIO system can be completed normally.

[0009] For example, during the initialization of the VIO system, if any of the following situations occurs, it is considered that a data anomaly has occurred:

[0010] 1. In the process of fusing visual information and IMU data, there is a situation where the estimated state diverges;

[0011] 2. The VIO system fails to track a map point within the preset time period;

[0012] 3. Determine based on the auxiliary sensor that the visual positioning error of the VIO system is greater than or equal to the error threshold.

[0013] The auxiliary sensor may be a global positioning system (GPS).

[0014] For example, the preset duration can be a reasonable preset duration, such as half an hour. After the VIO system is initialized, it can be considered that the accumulated error of the VIO system has been eliminated. However, during subsequent operation of the VIO system, the error will accumulate again. Therefore, a timer trigger mechanism can be used to instruct the VIO system to initialize every preset duration starting from the moment the last static initialization is completed.

[0015] Based on the above technical solution, it is possible to determine whether the current VIO system needs to be initialized by obtaining the status information of the VIO system. When it is determined that the VIO system meets the initialization conditions, the mobile device is actively controlled to remain stationary for a first period of time and then start up, thereby creating conditions for static initialization, which helps to improve the efficiency, success rate and robustness of initialization of the VIO system.

[0016] In combination with the first aspect, in certain implementations of the first aspect, when the image acquisition device in the above-mentioned VIO system is a monocular camera, a first key frame and a second key frame are determined in sequence, wherein the first key frame is obtained by the monocular camera when it is determined that the self-moving device is stationary, and the disparity between the first key frame and the second key frame is greater than or equal to the second disparity threshold; the second time period is determined based on the first moment of obtaining the first key frame and the second moment of obtaining the second key frame; the second time period is evenly divided into a first sub-period and a second sub-period, and the first sub-period is located before the second sub-period in time sequence; based on the inertial measurement unit IMU data obtained by the VIO system in the first sub-period, the acceleration mean and the gyroscope mean are determined, and the acceleration mean is a vector; based on the acceleration mean, the gyroscope mean, the first key frame and the second key frame, the initial value of the VIO system is determined.

[0017] For example, the first, second, and third image frames are acquired by an image acquisition device, and the three image frames are not necessarily acquired consecutively. However, the first image frame is acquired first, followed by the second image frame, and finally the third image frame. The first image frame can be referred to as an initial frame, which is the first image frame acquired after the first indication information is sent.

[0018] For example, the disparity between image frames can be determined by extracting and matching feature points between the image frames. It should be understood that feature points generally refer to corner points or representative regions and edges in an image, which have the characteristic of remaining unchanged even after a small change in the camera's perspective. Typically, corner points in an image can be selected as feature points.

[0019] For example, the IMU data may include acceleration data and gyroscope data.

[0020] For example, the acceleration mean is used to determine the acceleration bias and gravity direction in the initial value of the VIO system, the gyroscope mean is used to determine the gyroscope bias in the initial value of the VIO system, and the second image frame and the third image frame are used to determine the feature point depth in the initial value of the VIO system.

[0021] Based on the above technical solution, it is possible to reasonably obtain detection data from the period when the mobile device undergoes motion jumps, and complete the static initialization of the VIO system based on the monocular camera based on the detection data. This initialization method is highly efficient, has a high success rate, and is highly robust.

[0022] In combination with the first aspect, in certain implementations of the first aspect, a first image frame and a second image frame are determined in sequence. When the disparity between the first image frame and the second image frame is less than or equal to a first disparity threshold, the first image frame or the second image frame is determined to be a first key frame, and the first disparity threshold is less than the second disparity threshold. A third image frame is determined. When the disparity between the third image frame and the first key frame is greater than or equal to the second disparity threshold, the third image frame is determined to be a second key frame.

[0023] For example, the first disparity threshold is a preset threshold value, such as 0, in which case the disparity between the first and second image frames should be equal to the first disparity threshold value. Considering possible sensor detection errors, the first disparity threshold can be preset to a value close to 0, in which case the disparity between the first and second image frames should be less than the first disparity threshold value. Correspondingly, the second disparity threshold can also be a preset threshold value, specifically a reasonable threshold value preset based on the application scenario and motion performance of the mobile device.

[0024] Based on the above technical solution, it is possible to determine whether the self-moving device is in a stationary state, so as to ensure that the static initialization of the VIO system can be performed under the condition that the self-moving device is already stationary.

[0025] In combination with the first aspect, in some implementations of the first aspect, when the image acquisition device in the above-mentioned VIO system is a monocular camera, the acceleration mean in each third time period is determined according to the IMU data obtained by the VIO system in each third time period, and the acceleration mean is a vector; the first acceleration mean and the second acceleration mean are determined in sequence, wherein the first acceleration mean is less than or equal to the first acceleration threshold, the second acceleration mean is greater than or equal to the second acceleration threshold, the first acceleration threshold is less than the second acceleration threshold, and the third time period corresponding to the first acceleration mean and the third time period corresponding to the second acceleration mean are adjacent time periods; in the first acceleration mean corresponding When the disparity between the fourth image frame and the fifth image frame obtained in the third time period corresponding to the second acceleration average is less than or equal to the first disparity threshold, and the disparity between the sixth image frame and the seventh image frame obtained in the third time period corresponding to the second acceleration average is greater than or equal to the second disparity threshold, the fourth image frame or the fifth image frame is determined as the first key frame, and the seventh image frame is determined as the second key frame, and the first gyroscope average is determined according to the IMU data obtained by the VIO system in the third time period corresponding to the first acceleration average, wherein the first disparity threshold is less than the second disparity threshold; and the initial value of the VIO system is determined according to the first acceleration average, the first gyroscope average, the first key frame, and the second key frame.

[0026] For example, the first acceleration threshold is a preset threshold. Ideally, the first acceleration threshold should be equal to 0, but in actual applications, the sensor may have detection errors, so the first acceleration threshold can be a value close to 0, such as 0.1m / s. 2 , 0.2m / s 2 Correspondingly, the second acceleration threshold is also a preset threshold, which can be a reasonable threshold determined based on the application scenario of the mobile device and its own motion performance, for example, 1m / s 2 , 2m / s 2 wait.

[0027] Based on the above technical solution, it is possible to determine whether the mobile device is in a stationary state, and reasonably obtain detection data during the period when the mobile device undergoes motion jumps. Based on the detection data, the static initialization of the VIO system based on the monocular camera is completed. This initialization method is highly efficient, has a high success rate and is highly robust.

[0028] In combination with the first aspect, in certain implementations of the first aspect, after the first indication information is sent, if it is still not determined whether the mobile device has become stationary after a fifth time period, the first indication information is resent to the control device.

[0029] Based on the above technical solution, a retransmission mechanism for the first indication information is introduced, which can effectively avoid the problem of failure to receive the first indication information, thereby ensuring that the initialization of the VIO system can proceed normally and increasing the reliability of static initialization of the VIO system.

[0030] In combination with the first aspect, in certain implementations of the first aspect, within the above-mentioned first time period, when the first key frame has been determined, a second indication message is sent to the control device of the self-mobile device, and the second indication message is used to instruct the control device to control the self-mobile device to start in advance.

[0031] Based on the above technical solution, considering that the first time period is sufficient time reserved for the VIO system to collect relevant data from the mobile device in a stationary state, the first time period may have the problem of too long reserved time. When the above-mentioned collection process is completed in advance, the second indication information can be used to start the mobile device in advance, so that the static initialization enters the subsequent process, thereby increasing the execution efficiency of the static initialization.

[0032] In a second aspect, a device for initializing a VIO system is provided, which is applied to a self-moving device equipped with a VIO system. The device includes: an acquisition unit for acquiring first status information of the VIO system; a sending unit for sending first indication information to a control device of the self-moving device according to the first status information, wherein the first indication information is used to instruct the control device to control the self-moving device to remain stationary for a first time period and then start; a processing unit for performing a static initialization operation on the VIO system to determine the initial value of the VIO system when it is determined that the self-moving device has become stationary.

[0033] Among them, the above-mentioned first status information includes at least one of the following contents: the VIO system initiates a first signal, the first signal is used to indicate that the VIO system is about to be initially started, or a data abnormality occurs during the process of the VIO system processing data, or a preset time has passed since the last static initialization operation was completed.

[0034] In combination with the second aspect, in certain implementations of the second aspect, when the image acquisition device in the above-mentioned VIO system is a monocular camera, the above-mentioned processing unit is specifically used to: determine the first key frame and the second key frame in sequence, wherein the first key frame is obtained by the monocular camera when it is determined that the self-mobile device is stationary, and the disparity between the first key frame and the second key frame is greater than or equal to the second disparity threshold; determine the second time period based on the first moment of obtaining the first key frame and the second moment of obtaining the second key frame; divide the second time period into a first sub-period and a second sub-period on average, and the first sub-period is located before the second sub-period in time sequence; determine the acceleration mean and the gyroscope mean based on the inertial measurement unit IMU data obtained by the VIO system in the first sub-period, and the acceleration mean is a vector; determine the initial value of the VIO system based on the acceleration mean, the gyroscope mean, the first key frame and the second key frame.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the above-mentioned processing unit is specifically used to: determine the first image frame and the second image frame in sequence, and when the disparity between the first image frame and the second image frame is less than or equal to the first disparity threshold, determine the first image frame or the second image frame as the first key frame, and the first disparity threshold is less than the second disparity threshold; determine the third image frame, and when the disparity between the third image frame and the first key frame is greater than or equal to the second disparity threshold, determine the third image frame as the second key frame.

[0036] In combination with the second aspect, in some implementations of the second aspect, when the image acquisition device in the above-mentioned VIO system is a monocular camera, the above-mentioned processing unit is specifically used to: determine the acceleration mean in each third time period according to the IMU data obtained by the VIO system in each third time period, and the acceleration mean is a vector; determine the first acceleration mean and the second acceleration mean in sequence, wherein the first acceleration mean is less than or equal to the first acceleration threshold, the second acceleration mean is greater than or equal to the second acceleration threshold, the first acceleration threshold is less than the second acceleration threshold, and the third time period corresponding to the first acceleration mean and the third time period corresponding to the second acceleration mean are adjacent time periods; in the first acceleration When the disparity between the fourth image frame and the fifth image frame obtained in the third time period corresponding to the velocity average is less than or equal to the first disparity threshold, and the disparity between the sixth image frame and the seventh image frame obtained in the third time period corresponding to the second acceleration average is greater than or equal to the second disparity threshold, the fourth image frame or the fifth image frame is determined as the first key frame, and the seventh image frame is determined as the second key frame, and the first gyroscope average is determined based on the IMU data obtained by the VIO system in the third time period corresponding to the first acceleration average, wherein the first disparity threshold is less than the second disparity threshold; and the initial value of the VIO system is determined based on the first acceleration average, the first gyroscope average, the first key frame, and the second key frame.

[0037] In combination with the second aspect, in certain implementations of the second aspect, after the above-mentioned sending unit sends the above-mentioned first indication information, the above-mentioned sending unit is also used to: if it is still not determined whether the above-mentioned mobile device has become stationary after the fifth time period, re-send the above-mentioned first indication information to the above-mentioned control device.

[0038] In combination with the second aspect, in certain implementations of the second aspect, within the above-mentioned first time period, the above-mentioned sending unit is also used to: when the above-mentioned processing unit has determined the above-mentioned first key frame, send a second indication information to the control device of the self-mobile device, and the second indication information is used to instruct the control device to control the self-mobile device to start in advance.

[0039] In a third aspect, a device for initializing a VIO system is provided, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute a method in any possible implementation of the method design of the first aspect above.

[0040] In a fourth aspect, a chip system is provided, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to echo a signal from the memory of the electronic device and send a signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes a method in any possible implementation of the method design of the first aspect above.

[0041] In a fifth aspect, a computer-readable storage medium is provided, storing a computer program or instruction, which is used to implement the method in any possible implementation manner in the method design of the first aspect.

[0042] In a sixth aspect, a computer program product is provided. When the computer program code or instructions are executed on a computer, the computer executes a method in any possible implementation of the method design of the first aspect.

[0043] In a seventh aspect, a self-propelled device is provided, comprising: a processor, a controller, and a VIO system, wherein the VIO system includes an image acquisition device and an IMU;

[0044] The above-mentioned processor is used to obtain first status information of the VIO system; send first indication information to the controller according to the first status information, the first indication information is used to instruct the controller to control the self-moving device to remain stationary for a first time period and then start; the above-mentioned image acquisition device is used to acquire image frames at a first frame rate; the above-mentioned IMU is used to acquire IMU data at a second frame rate, wherein the second frame rate is greater than the first frame rate; the above-mentioned processor is also used to determine the motion state of the self-moving device according to the image frames acquired by the image acquisition device, and when it is determined that the self-moving device has stopped, perform a static initialization operation on the VIO system according to the data acquired by the image acquisition device and the IMU to determine the initial value of the VIO system; wherein the above-mentioned first status information includes at least one of the following contents: the VIO system initiates a first signal, the first signal is used to indicate that the VIO system is about to start up initially, or a data abnormality occurs during the process of the VIO system processing data, or a preset time has passed since the last time the static initialization operation was completed.

[0045] In an eighth aspect, a self-moving device is provided, which includes the device in any possible implementation of the second aspect or the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG1 is a schematic block diagram of a VIO system 100 provided in an embodiment of the present application;

[0047] FIG2 is a flow chart of a method 200 for initializing a VIO system provided in an embodiment of the present application;

[0048] FIG3 is a flow chart of a static initialization method 300 provided in an embodiment of the present application;

[0049] FIG4 is a schematic diagram of determining a first key frame and a second key frame according to an embodiment of the present application;

[0050] FIG5 is a flow chart of another static initialization method 500 provided in an embodiment of the present application;

[0051] FIG6 is a flow chart of another static initialization method 600 provided in an embodiment of the present application;

[0052] FIG7 is a schematic block diagram of a VIO system initialization apparatus 700 provided in an embodiment of the present application;

[0053] FIG8 is a schematic block diagram of a computer device 800 provided in an embodiment of the present application;

[0054] FIG9 is a schematic block diagram of a computer-readable storage medium 900 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is a kind of association relationship that describes associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In this application, "at least one" refers to one or more, and "more than one" refers to two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0056] In the embodiments of this application, prefixes such as "first" and "second" are used only to distinguish different description objects and have no limiting effect on the position, order, priority, quantity, or content of the described objects. The use of prefixes such as ordinal numbers in the embodiments of this application to distinguish description objects does not constitute a limitation on the described objects. For a statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.

[0057] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0058] VIO is typically installed on self-driving vehicles, providing them with real-time mapping and positioning information. This information is used to fuse data from image acquisition devices and IMUs. While the camera pose is calculated from two keyframes acquired by the image acquisition device, IMU measurement data is also used as a model constraint to achieve real-time positioning and map construction. Furthermore, because the IMU can output a variety of navigation parameters, it can also provide the camera with detailed motion information for each frame.

[0059] In some possible embodiments, the self-moving device proposed in this application can be any mechanical device that can move in space with a high degree of autonomy, for example, a mobile robot, a mobile purifier, a drone or an unmanned vehicle, etc. Among them, the mobile robot can include a sweeping robot, a lawn mowing robot, a companion robot or a guide robot, etc.

[0060] Visual induction (VIO) systems installed on self-propelled devices can be categorized into two types: loosely coupled and tightly coupled. Loosely coupled VIO systems perform motion estimation based on image frames captured by an image acquisition device and motion calculation based on the self-propelled device's motion data acquired by an IMU. These two results are then combined. While the algorithms in loosely coupled VIO systems are relatively simple, the image acquisition device and IMU data calculation processes are independent of each other. Therefore, these systems struggle to fully utilize the data from both sources, ensuring good detection results. Tightly coupled VIO systems, on the other hand, combine the data from the image acquisition device and IMU to construct the motion and observation equations, and then perform unified motion estimation on the self-propelled device. While the algorithms in tightly coupled VIO systems are more complex, they fully utilize the data from both the image acquisition device and IMU, achieving better detection results and enabling application in more complex scenarios. These systems are a research focus in the field of visual positioning and navigation technology.

[0061] However, the tightly coupled VIO system is a highly nonlinear system, especially the VIO system whose image acquisition device is a monocular camera. In the application scenario of vision-based real-time positioning and map construction, the detection scale and rotation relative to the gravity direction of the monocular camera are unmeasurable. Therefore, the tightly coupled VIO system needs a relatively good initial value when fusing visual data and IMU data, and this initial value can be determined by initializing the VIO system.

[0062] Generally speaking, the initialization methods of the VIO system include dynamic initialization and static initialization. Among them, dynamic initialization refers to the method of initializing the VIO system while the mobile device is moving. Although the mobile device is in a mobile state most of the time during operation, it is usually difficult for the mobile device to fully meet the motion conditions of dynamic initialization during movement, and the computational efficiency of dynamic initialization is low, and the computational results are difficult to converge. In contrast, the computational complexity of static initialization is low, and the computational results are easy to converge and the computational efficiency is high. Similar to dynamic initialization, the execution of static initialization also requires the mobile device to meet the corresponding motion conditions, that is, the mobile device needs to remain stationary for a period of time. However, compared with the motion conditions of dynamic initialization, the motion conditions of static initialization are clearer and easier to achieve. In summary, static initialization has lower requirements, takes less time, has a higher success rate, and is more robust.

[0063] However, based on an existing method for initializing a VIO system, it can be seen that the static initialization of the VIO system is a passive execution process, that is, the VIO system performs static initialization only when it is determined that the self-moving device is in a stationary state. If the self-moving device is in motion for a long time, the VIO will not be initialized for a long time, which will cause the accumulated error of the VIO system to gradually increase, reducing the accuracy of the VIO system detection. Alternatively, based on another existing method for initializing a VIO system, it can be seen that the initialization of the VIO system is also a passive execution process, that is, when it is found that the system needs to be initialized, the current motion state of the self-moving device is first determined. If the self-moving device is in a stationary state, static initialization is performed. If the self-moving device is in motion, dynamic initialization is performed. However, the dynamic initialization of this method also has the corresponding problems mentioned in the above description, so the overall initialization efficiency is poor and the robustness is difficult to guarantee.

[0064] In view of this, an embodiment of the present application proposes a method for initializing a VIO system, by obtaining status information of the VIO system and determining whether the VIO system meets the initialization conditions based on the status information. If the VIO system meets the initialization conditions, the control device of the mobile device is instructing the mobile device to stop, thereby providing conditions for performing a static initialization operation on the VIO system.

[0065] FIG1 is a schematic block diagram of a VIO system 100 provided in an embodiment of the present application.

[0066] In some possible embodiments, the VIO system may include: an image acquisition device, an IMU, and a processing device.

[0067] In some possible embodiments, the image acquisition device may be a monocular camera or a binocular camera. Specifically, it may be an optical three primary colors (red-green-blue, RGB) camera, an infrared radiation (IR) camera, or a depth camera, which is not limited in the embodiments of the present application. The image acquisition device in the embodiments of the present application is used to collect image data from the surroundings of the mobile device.

[0068] In some possible embodiments, an IMU is used to collect motion data from a mobile device. The IMU includes an accelerometer and a gyroscope, which are used to collect acceleration data and angular velocity data from the mobile device, respectively.

[0069] In some possible embodiments, the processing device is used to obtain status information of the VIO system, determine whether the VIO system meets the initialization conditions, and instruct the control device of the mobile device to make the mobile device still, providing conditions for performing static initialization operations on the VIO system.

[0070] In some possible embodiments, the VIO system may be mounted on a self-mobile device. In the system architecture, the control device belongs to the self-mobile device, and the processing device may be a specific functional module in the central processing unit (CPU) of the self-mobile device.

[0071] In some possible embodiments, the processing device may also be a processor independent of the CPU of the self-moving device, which can establish a connection with the CPU of the self-moving device and further instruct the control device to stop the self-moving device.

[0072] In some possible embodiments, the above-mentioned processing device can also be installed in a server. When a communication connection is established between the mobile device and the server, the processing device can send corresponding instruction information to the control device of the mobile device through the server when it determines that the VIO system needs to perform static initialization, instructing the control device to put the mobile device to rest.

[0073] Based on the above-mentioned VIO system, an embodiment of the present application proposes a method for initializing the VIO system, so that when the VIO system needs to perform a static initialization operation, the mobile device is actively stationary, thereby providing conditions for performing a static initialization operation on the VIO system.

[0074] FIG2 is a flow chart of a method 200 for initializing a VIO system according to an embodiment of the present application. The method 200 is applied to a self-mobile device equipped with a VIO system.

[0075] S210: Acquire first status information of the VIO system.

[0076] The first status information is used to indicate the operating status of the VIO system, and may include at least one of the following:

[0077] The VIO system initiates a first signal, where the first signal is used to indicate that the VIO system is about to be initially started, or,

[0078] During the VIO system processing data, data anomalies occur, or

[0079] The preset time has elapsed since the last static initialization operation was completed.

[0080] S220: Sending first indication information to a control device of the mobile device according to the first status information.

[0081] S230: When it is determined that the mobile device is stationary, a static initialization operation is performed on the VIO system to determine an initial value of the VIO system.

[0082] It should be understood that if the VIO system is not initialized before each startup, the cumulative error of the VIO system will gradually increase over time, affecting the accuracy of the VIO system's detection. Furthermore, the VIO system and the mobile device are typically started simultaneously, so the motion state of the mobile device must change from static to dynamic, which directly meets the requirements for static initialization of the VIO system. Therefore, it is appropriate and necessary to perform a VIO system initialization before each startup.

[0083] In some possible embodiments, the first signal may be a signal sent to the global system or to the processing device of the VIO system 100 before the VIO system is started.

[0084] In some possible embodiments, during the initialization of the VIO system, if any of the following situations occurs, it is considered that a data anomaly has occurred:

[0085] 1. In the process of fusing visual information and IMU data, there is a situation where the estimated state diverges.

[0086] The VIO system includes an image acquisition device and an IMU. During the operation of the VIO system, it is necessary to fuse the sensor data obtained by the image acquisition device and the sensor data obtained by the IMU. For example, state estimation is performed through the Falman filter algorithm. If the covariance of the estimated state diverges, it means that the VIO system has a data anomaly, also known as "runaway".

[0087] 2. The VIO system fails to track a map point within the preset time period.

[0088] Tracking map points is the primary function of the VIO system. If the VIO system cannot continuously and normally track map points, it means that the VIO system is not operating normally. This situation also falls into the "runaway" situation mentioned above, and the VIO system needs to be initialized to resolve this abnormality.

[0089] The preset time period can be a reasonable time interval, such as 2 seconds. Furthermore, the preset time period can be adjusted at any time based on the application scenario of the self-mobile device. If necessary, such as when the self-mobile device is used in more complex scenarios, the preset time period can be equal to the unit cycle length of the image acquisition device. That is, map point detection is performed on each frame of the image. If a map point is not tracked in a frame of the image, it indicates that a data anomaly has occurred in the VIO system.

[0090] 3. Determine based on the auxiliary sensor that the visual positioning error of the VIO system is greater than or equal to the error threshold.

[0091] Self-mobile devices are usually also equipped with auxiliary sensors, such as GPS. While the VIO system is performing visual positioning, the GPS will also collect the location information of the self-mobile device accordingly. Therefore, after the VIO system outputs the visual positioning result, it can be verified with the GPS positioning result to determine the visual positioning error of the VIO system. When the visual positioning error is greater than or equal to the error threshold, it means that the cumulative error of the VIO system is too large, which also indicates that the VIO system has data anomalies.

[0092] In some possible embodiments, the preset duration may be a preset reasonable duration, such as half an hour. After the initialization operation is completed for the VIO system, it can be considered that the accumulated error of the VIO system has been eliminated. However, during the subsequent operation of the VIO system, the error will be accumulated again. Therefore, a timer trigger mechanism can be used to instruct the VIO system to perform an initialization operation every unit time. The specific process is as follows: before starting the timer trigger mechanism, determine the preset duration. Then, after completing this static initialization operation, record the current moment, and then start the timer. After the preset duration has passed, the timer will trigger a second signal. The second signal is used to indicate that a preset duration has passed since the last time the static initialization operation was completed. The information carried by the second signal belongs to the above-mentioned first state information.

[0093] Based on the existing VIO system initialization method mentioned above, if any of the three situations mentioned in S220 above occurs in the VIO system, it can only relatively passively judge the current motion state of the mobile device. Assuming that the VIO system has a problem with data processing abnormality during the movement of the mobile device, the movement state of the mobile device obviously will not meet the conditions for static initialization, and the VIO system will not perform static initialization in time, so this abnormal state will continue until it is detected that the mobile device is stationary, and then static initialization can be performed. Even if the VIO system can perform dynamic initialization, based on the above-mentioned known defects of dynamic initialization, the execution efficiency of the dynamic initialization is relatively low.

[0094] In some possible embodiments, when the control device receives the first instruction information, the motion state of the self-moving device may be stationary or moving, and this embodiment of the present application is not limited thereto. Even if the self-moving device is about to start when the control device receives the first instruction information, the control device will control the self-moving device to remain stationary for a first period of time and then start it, rather than controlling the self-moving device to start immediately.

[0095] In some possible embodiments, the first period is a preset, reasonable time period. The length of this period is sufficient to ensure that some operations for static initialization of the VIO system can be completed normally. It is typically less than 2 seconds and should not be too long or too short. Some of these operations may include the subsequent acquisition of two frames of images by the VIO system's monocular camera to confirm that the mobile device is in a stationary state. If the first period is too long, the efficiency of static initialization will be reduced. If the first period is too short, the VIO system will not be able to collect detection data that meets the requirements for static initialization in a timely manner, causing static initialization to fail.

[0096] In some possible embodiments, the first time period may also be determined based on the image acquisition frame rate of the VIO system. For example, the higher the image acquisition frame rate, the shorter the first time period.

[0097] In some possible embodiments, the static initialization includes initialization of IMU data and initialization of visual data. The initialization objects of IMU data include gyroscope bias, acceleration bias, and gravity direction, and the initialization objects of visual data include feature point depth. Therefore, the initial values ​​of the VIO system include gyroscope bias, acceleration bias, gravity direction, and feature point depth. The gyroscope bias, acceleration bias, and gravity direction can be determined based on IMU data obtained from a mobile device in a stationary state, while the feature point depth can be determined based on two frames of images obtained from a mobile device in motion.

[0098] Based on the above technical solution, it is possible to determine whether the current VIO system needs to be initialized by obtaining the status information of the VIO system. When it is determined that the VIO system meets the initialization conditions, the mobile device is actively controlled to remain stationary for a first period of time and then start up, thereby creating conditions for static initialization, which helps to improve the efficiency, success rate and robustness of initialization of the VIO system.

[0099] In some possible embodiments, when the image acquisition device in the VIO system is a monocular camera, the above S240 can be implemented in the following manner.

[0100] FIG3 is a flow chart of a static initialization method 300 provided in an embodiment of the present application.

[0101] S310: Determine a first key frame and a second key frame in sequence, wherein the first key frame is acquired by a monocular camera when it is determined that the mobile device is stationary, and a disparity between the first key frame and the second key frame is greater than or equal to a second disparity threshold.

[0102] In some possible embodiments, the disparity between image frames can be determined by extracting and matching feature points between the image frames. It should be understood that feature points generally refer to corner points or representative regions and edges in an image, which have the characteristic of remaining unchanged even after a small change in the camera's perspective. Typically, corner points in an image can be selected as feature points.

[0103] In some possible embodiments, the corner extraction of the image frame may be achieved by using a Moravec corner detection algorithm, a Harris corner detection algorithm, or a Shi-Tomasi algorithm.

[0104] In some possible embodiments, feature matching between image frames includes a feature point method and a direct method. The former is based on descriptor matching and is optimized by minimizing the reprojection error, while the latter is based on optical flow tracking and minimizes the image pixel luminosity error and then directly uses the image pixel grayscale information for matching.

[0105] Taking the above-mentioned Harris corner detection algorithm as an example, the Harris corner measurement formula can be used to select the feature point with the largest corner response value as the final selection result. Then, the fast (FAST) feature point detection algorithm is used to detect the first grayscale difference between the specified pixel and the pixels in its surrounding area. If there are enough pixels corresponding to the first grayscale value difference exceeding the preset grayscale threshold, it indicates that the specified pixel may be a corner point. Then, it is measured based on the Harris corner response value to determine whether it is a corner point. After identifying multiple groups of feature points between two frames of images, the relative inter-frame pose of the two frames can be calculated by minimizing the projection error between 3D and 2D. Common methods include EPnP method, UPnP method or P3P method.

[0106] S320: Determine a second time period according to the first moment of obtaining the first key frame and the second moment of obtaining the second key frame.

[0107] FIG4 is a schematic diagram of determining a first key frame and a second key frame proposed in an embodiment of the present application.

[0108] Referring to Figure 4, it can be seen that after determining that the self-moving device is stationary, the first frame of image or the nth frame of image acquired when it is stationary can be used as the first key frame. However, in the process of initializing the depth of feature points for a monocular camera, it is not possible to achieve it only with two frames of image frames with a parallax of 0 or close to 0. Therefore, two frames of image frames with a parallax greater than or equal to the second parallax threshold are required to achieve feature point depth initialization. These two frames of image are the first key frame and the second key frame. The second period between these two key frames is the period when the self-moving device undergoes a motion jump. This motion jump refers to the jump process of the self-moving device from a stationary state to a moving state.

[0109] As can be seen from the above description, when determining the second key frame, simply finding a parallax between a certain image frame and the first key frame does not necessarily mean that the image frame is the second key frame. Because the parallax between the two frames used for feature point depth initialization must be large enough to meet the requirements for feature point depth initialization, the second key frame may not be the first image frame acquired after the mobile device moves, but may be an image frame acquired later.

[0110] Since the second period between the first key frame and the second key frame is the period when the mobile device undergoes a motion jump, the second period must include a period when the mobile device remains stationary. Therefore, the IMU also needs to obtain IMU data within the second period to initialize the acceleration bias, gyroscope bias, and gravity direction. The specific process of extracting the IMU data for initialization from the IMU data within the second period is as follows:

[0111] S330: Divide the second time period into a first sub-time period and a second sub-time period on an equal basis, wherein the first sub-time period is located before the second sub-time period in terms of time sequence.

[0112] S340: Determine an acceleration mean and a gyroscope mean based on the IMU data acquired by the VIO system in the first sub-period, where the acceleration mean is a vector.

[0113] It should be understood that during the second period, the self-moving device experienced a jump in motion. Therefore, it can be assumed that the self-moving device remained essentially stationary during the first sub-period, while it must have moved during the second sub-period. Therefore, the acceleration value acquired by the VIO system during the first sub-period can be used to determine the initial value of the VIO system.

[0114] In some possible embodiments, the above-mentioned IMU data may include acceleration data and gyroscope data.

[0115] S350: Determine an initial value of the VIO system according to the acceleration mean, the gyroscope mean, the first key frame, and the second key frame.

[0116] In some possible embodiments, the acceleration mean is used to determine the acceleration bias and gravity direction in the initial value of the VIO system, the gyroscope mean is used to determine the gyroscope bias in the initial value of the VIO system, and the second image frame and the third image frame are used to determine the feature point depth in the initial value of the VIO system.

[0117] In some possible embodiments, the acceleration bias, gyroscope bias, and gravity direction in the initial value may be determined in the following manner.

[0118] For the direction of gravity: the relative direction of gravity can be determined based on the acceleration value obtained by the VIO system in the first sub-period. For example, if the acceleration value obtained in the first sub-period is Then assume that the reference gravity vector Perpendicular to the horizontal plane of the self-moving device; finally based on the formula: Confirm the above and The rotation amount R between them can be used to determine the direction of gravity in the initial value.

[0119] For the gyroscope bias: the gyroscope mean value corresponding to the first sub-period should be equal to 0, so the gyroscope bias is equal to the gyroscope mean value.

[0120] For acceleration bias: the average acceleration corresponding to the first sub-period should be equal to the acceleration due to gravity, so the acceleration bias is equal to the difference between the average acceleration corresponding to the first sub-period and the acceleration due to gravity.

[0121] Regarding the depth of the feature points: the depth of the feature points may be initialized based on the second image frame and the third image frame through a triangulation method.

[0122] Based on the above technical solution, it is possible to reasonably obtain detection data from the mobile device during the period of motion jump, and complete the static initialization of the VIO system based on the monocular camera based on the detection data. This initialization method is highly efficient, has a high success rate and is highly robust.

[0123] In some possible embodiments, the first key frame and the second key frame in the above method 300 may be obtained in the following manner.

[0124] sequentially determining a first image frame and a second image frame, and determining the first image frame or the second image frame as a first key frame when a disparity between the first image frame and the second image frame is less than or equal to a first disparity threshold, wherein the first disparity threshold is less than the second disparity threshold;

[0125] A third image frame is determined, and when a disparity between the third image frame and the first key frame is greater than or equal to a second disparity threshold, the third image frame is determined to be a second key frame.

[0126] It should be understood that the above-mentioned first image frame, second image frame and third image frame are acquired through an image acquisition device, and these three image frames are not necessarily continuous in the acquisition sequence, but the first image frame is acquired, then the second image frame is acquired, and finally the third image frame is acquired.

[0127] In some possible embodiments, the first image frame may be referred to as an initial frame, which is the first image frame acquired after the first indication information is sent.

[0128] In some possible embodiments, the first disparity threshold is a preset threshold value, for example, it may be equal to 0, in which case the disparity between the first image frame and the second image frame should be equal to the first disparity threshold value. Considering possible sensor detection errors, the first disparity threshold value may be preset to a value close to 0, in which case the disparity between the first image frame and the second image frame should be less than the first disparity threshold value. Correspondingly, the second disparity threshold value may also be a preset threshold value, specifically a reasonable threshold value preset based on the application scenario and motion performance of the mobile device.

[0129] It should be understood that when the disparity between the first image frame and the second image frame is less than or equal to the first disparity threshold, it can be determined that the mobile device is in a stationary state during the period between acquiring the first image frame and acquiring the second image frame.

[0130] In some possible embodiments, the above-mentioned second disparity threshold may also be a preset reasonable threshold, for example, it may be 30. It should be understood that the greater the disparity between the two frames of images obtained from the mobile device as the main perspective, the faster the mobile device moves. Then, taking the second image frame as the first key frame as an example, when the disparity between the second image frame and the third image frame is greater than or equal to the second disparity threshold, it can be determined that the mobile device is in motion during the period between obtaining the second image frame and obtaining the third image frame, so the third image frame is the second key frame.

[0131] Based on the above technical solution, it is possible to determine whether the self-moving device is in a stationary state, so as to ensure that the static initialization of the VIO system can be performed under the condition that the self-moving device is already stationary.

[0132] In some possible embodiments, the second key frame and the corresponding second time period may be determined in the following manner:

[0133] S3201: Determine the first image frame as a first key frame, where the first image frame is acquired by a monocular camera when the mobile device is determined to be stationary.

[0134] S3202: Determine a first disparity between each subsequently acquired image frame and the first key frame.

[0135] S3203: When it is determined that the first disparity between the i-th image frame and the first key frame is greater than the second disparity threshold, the i-th image frame is set as the second key frame.

[0136] In some possible embodiments, the second key frame may be the third image frame, and the second image frame may be an image frame between the first key frame and the second key frame.

[0137] S3204: Determine a time period from the first key frame to the acquisition of the second key frame as a first time window, and slide a first sliding window in the first time window.

[0138] S3205: During the period when the first sliding window slides in the first time window, determine the corresponding acceleration mean value according to the acceleration values ​​obtained by the VIO system in the time period corresponding to the first sliding window at each moment.

[0139] S3206: At two adjacent moments, if the acceleration mean corresponding to the first sliding window changes from being less than or equal to the first threshold to being greater than the second threshold, the time period corresponding to the first sliding window at the latter moment is the second time period.

[0140] It should be understood that the first threshold value may be a value close to 0, for example, 0.1 m / s 2 .

[0141] In some possible embodiments, the second key frame and the corresponding second time period may be determined in the following manner:

[0142] S3211: Determine the first image frame as a first key frame, where the first image frame is acquired by a monocular camera when the mobile device is determined to be stationary.

[0143] S3212: Determine a first disparity between each subsequently acquired image frame and the first key frame.

[0144] S3213: If the first disparity between each image frame acquired within the preset window period and the first key frame is smaller than the first disparity threshold, determine the last image frame acquired within the preset window period as the new first key frame.

[0145] S3214: If it is determined that the first disparity between the i-th image frame and the first key frame is greater than the second disparity threshold within the preset window period, the i-th image frame is determined to be the second key frame.

[0146] In some possible embodiments, the second key frame may be the third image frame, and the second image frame may be the first key frame determined within the preset window period.

[0147] S3215: Determine the time period corresponding to the moment when the first key frame is obtained and the moment when the second key frame is obtained as the second time period.

[0148] Based on the above technical solution, the second key frame and the second time period can be accurately determined to ensure that subsequent static initialization operations can proceed normally.

[0149] In some possible embodiments, when the image acquisition device in the VIO system is a monocular camera, the above S240 can also be implemented in the following manner.

[0150] FIG5 is a flow chart of another static initialization method 500 provided in an embodiment of the present application.

[0151] S510: Determine an acceleration mean value in each third time period based on the acceleration value obtained by the VIO system in each third time period, where the acceleration mean value is a vector.

[0152] S520: Determine the first acceleration mean and the second acceleration mean in sequence, wherein the first acceleration mean is less than or equal to the first acceleration threshold, and the second acceleration mean is greater than or equal to the second acceleration threshold, wherein the first acceleration threshold is less than the second acceleration threshold, and the third time period corresponding to the first acceleration mean and the third time period corresponding to the second acceleration mean are adjacent time periods.

[0153] In some possible embodiments, the first acceleration threshold is a preset threshold. Ideally, the first acceleration threshold should be equal to 0, but in actual applications, the sensor may have detection errors, so the first acceleration threshold can be a value close to 0, such as 0.1 m / s. 2 , 0.2m / s 2 Correspondingly, the second acceleration threshold is also a preset threshold, which can be a reasonable threshold determined based on the application scenario of the mobile device and its own motion performance, for example, 1m / s 2 , 2m / s 2 wait.

[0154] It should be understood that if the acceleration is detected to be greater than or equal to the second acceleration threshold during two adjacent time periods, it means that the self-moving device may have experienced a motion jump during these two adjacent time periods. However, since the first acceleration average value will also be equal to 0 when the self-moving device is moving at a constant speed, in order to determine that the self-moving device is stationary during the third time period corresponding to the first acceleration average value, the following steps are required:

[0155] S530: When the disparity between the fourth image frame and the fifth image frame obtained in the third time period corresponding to the first acceleration mean is less than or equal to the first disparity threshold, and the disparity between the sixth image frame and the seventh image frame obtained in the third time period corresponding to the second acceleration mean is greater than or equal to the second disparity threshold, the fourth image frame or the fifth image frame is determined as the first key frame, and the seventh image frame is determined as the second key frame.

[0156] The first disparity threshold is smaller than the second disparity threshold.

[0157] S540: Determine a first gyroscope mean value based on IMU data acquired by the VIO system within a third time period corresponding to the first acceleration mean value.

[0158] S550: Determine an initial value of the VIO system according to the first acceleration mean, the first gyroscope mean, the first key frame, and the second key frame.

[0159] In some possible embodiments, if the disparity between the fourth image frame and the fifth image frame is not less than or equal to the first disparity threshold, it indicates that the mobile device is moving at a constant speed during the third period of time. In this scenario, the process can return to S510 and be repeated.

[0160] It should be understood that the specific process of determining the initial value of the VIO system is similar to the process of determining the initial value of the VIO system in the above method 300, and will not be repeated here.

[0161] Based on the above technical solution, it is also possible to determine whether the mobile device is in a stationary state, and reasonably obtain detection data during the period when the mobile device undergoes motion jumps. According to the detection data, the static initialization of the VIO system based on the monocular camera is completed. This initialization method is highly efficient, has a high success rate and is highly robust.

[0162] In some possible embodiments, during the first time period, once the first key frame has been determined, a second instruction message may be sent to the control device of the self-mobile device, instructing the control device to start the self-mobile device in advance. This solution can be applied in the following scenario: after the VIO system completes the collection of relevant data from the stationary self-mobile device during static initialization, if the stationary duration of the self-mobile device has not yet reached the first time period, the control device may start the self-mobile device in advance based on the second instruction message, allowing the static initialization to proceed to subsequent steps.

[0163] Based on the above technical solution, considering that the first time period is reserved for the VIO system to be sufficient for collecting relevant data of the self-mobile device in a stationary state, there may be a problem of too long reserved time in the first time period. When the above-mentioned collection process is completed in advance, the second indication information can be used to start the self-mobile device in advance, so that the static initialization enters the subsequent process, thereby increasing the execution efficiency of the static initialization.

[0164] In some possible embodiments, the process of obtaining the first key frame and the second key frame and initializing the feature point depth based on the first key frame and the second key frame can also be completed before the mobile device comes to rest. In this process, when the disparity between the two image frames is greater than or equal to the second disparity threshold, the two image frames can be determined to be the first key frame and the second key frame, respectively.

[0165] In some possible embodiments, when the image acquisition device in the VIO system is a binocular camera, the above S240 can also be implemented in the following manner.

[0166] FIG6 is a flow chart of another static initialization method 600 provided in an embodiment of the present application.

[0167] S610: Determine an acceleration mean value in each fourth time period based on the acceleration value obtained by the VIO system in each fourth time period, where the acceleration mean value is a vector.

[0168] S620: Determine the third acceleration mean and the fourth acceleration mean in sequence, wherein the third acceleration mean is less than or equal to the first acceleration threshold, and the fourth acceleration mean is greater than or equal to the second acceleration threshold, wherein the first acceleration threshold is less than the second acceleration threshold, and the fourth time period corresponding to the third acceleration mean and the fourth time period corresponding to the fourth acceleration mean are adjacent time periods.

[0169] S630: Determine a second gyroscope mean value based on IMU data acquired by the VIO system in a fourth time period corresponding to the third acceleration mean value.

[0170] S640: Determine an initial value of the VIO system according to the third acceleration mean value and the second gyroscope mean value.

[0171] It should be understood that when the image acquisition device is a binocular camera, due to its special structure, it is not necessary to obtain image frames corresponding to the transition from the mobile device from stationary to moving in order to initialize the depth of the feature points. Therefore, in this embodiment, the depth of the feature points can be initialized based on the first image frame and the second image frame.

[0172] Based on the above technical solution, it is possible to determine whether the mobile device is in a stationary state, and reasonably obtain the corresponding detection data within a specified time period of the mobile device. According to the detection data, the static initialization of the VIO system based on the binocular camera is completed. This initialization method is efficient, has a high success rate and is highly robust.

[0173] In some possible embodiments, after sending the first indication information, if it is still not determined whether the mobile device has become stationary after a fifth period of time, the first indication information is resent to the control device.

[0174] In some possible embodiments, the fifth time period is a preset time period, for example, 10 seconds.

[0175] In some possible embodiments, the reasons for not determining whether the mobile device has been stationary after the fifth time period include: failure to send the first indication information, so that the control device does not receive the first indication information after the fifth time period; or, the control device fails to parse the first indication information and does not perform the corresponding control operation within the fifth time period.

[0176] Based on the above technical solution, a retransmission mechanism for the first indication information is introduced, which can effectively avoid the problem of failure to receive the first indication information, thereby ensuring that the initialization of the VIO system can proceed normally and increasing the reliability of static initialization of the VIO system.

[0177] In some possible embodiments, when it is determined that a stationary initialization operation is required for the VIO system, it is not necessary to immediately instruct the control device to control the self-moving device to be stationary first and then start, because the self-moving device is usually equipped with other auxiliary sensors to enable the positioning of the self-moving device to be maintained for a period of time. Therefore, the timing of the control device controlling the self-moving device to be stationary can be combined with the operating trajectory of the self-moving device. For example, if the self-moving device is executing an I-shaped operating trajectory, you can wait until the self-moving device turns, and then instruct the control device to control the self-moving device to be stationary first and then start.

[0178] Based on the above technical solution, the impact of the above static initialization operation on the operation of the mobile device can be reduced.

[0179] In addition, an embodiment of the present application also provides a device for implementing any of the above methods. For example, a device for initializing a VIO system is provided, which includes a unit (or means) for implementing any of the above methods for initializing a VIO system.

[0180] FIG7 is a schematic block diagram of a device 700 for initializing a VIO system according to an embodiment of the present application. The device is applied to a self-propelled device equipped with a VIO system. As shown in FIG7 , the device 700 includes:

[0181] The acquiring unit 710 is configured to acquire first status information of the VIO system.

[0182] The sending unit 720 is configured to send first instruction information to a control device of the mobile device according to the first state information, wherein the first instruction information is used to instruct the control device to control the mobile device to remain stationary for a first period of time and then start.

[0183] The processing unit 730 is configured to perform a static initialization operation on the VIO system to determine an initial value of the VIO system when it is determined that the mobile device is stationary.

[0184] The first status information includes at least one of the following:

[0185] The VIO system initiates a first signal, where the first signal is used to indicate that the VIO system is about to be initially started, or,

[0186] During the VIO system processing data, data anomalies occur, or

[0187] The preset time has elapsed since the last static initialization operation was completed.

[0188] In some possible embodiments, when the image acquisition device in the VIO system is a monocular camera, the processing unit 730 is specifically configured to:

[0189] Determining a first key frame and a second key frame in sequence, wherein the first key frame is acquired by a monocular camera when it is determined that the mobile device is stationary, and a disparity between the first key frame and the second key frame is greater than or equal to a second disparity threshold;

[0190] determining a second time period according to a first moment of acquiring the first key frame and a second moment of acquiring the second key frame;

[0191] The second period is evenly divided into a first sub-period and a second sub-period, wherein the first sub-period is located before the second sub-period in time sequence;

[0192] Determine the acceleration mean and the gyroscope mean based on the inertial measurement unit (IMU) data acquired by the VIO system in the first sub-period, where the acceleration mean is a vector.

[0193] The initial value of the VIO system is determined based on the acceleration mean, the gyroscope mean, the first key frame, and the second key frame.

[0194] In some possible embodiments, the processing unit 730 is specifically configured to:

[0195] sequentially determining a first image frame and a second image frame, and determining the first image frame or the second image frame as a first key frame when a disparity between the first image frame and the second image frame is less than or equal to a first disparity threshold, and the first disparity threshold is less than the second disparity threshold;

[0196] A third image frame is determined, and when a disparity between the third image frame and the first key frame is greater than or equal to a second disparity threshold, the third image frame is determined to be a second key frame.

[0197] In some possible embodiments, when the image acquisition device in the VIO system is a monocular camera, the processing unit 730 is specifically configured to:

[0198] Determine, based on the IMU data acquired by the VIO system in each third period, an acceleration mean in each third period, where the acceleration mean is a vector;

[0199] Determining a first acceleration average and a second acceleration average in sequence, wherein the first acceleration average is less than or equal to a first acceleration threshold, the second acceleration average is greater than or equal to the second acceleration threshold, the first acceleration threshold is less than the second acceleration threshold, and a third time period corresponding to the first acceleration average and a third time period corresponding to the second acceleration average are adjacent time periods;

[0200] When the disparity between the fourth image frame and the fifth image frame obtained in the third time period corresponding to the first acceleration average value is less than or equal to the first disparity threshold, and the disparity between the sixth image frame and the seventh image frame obtained in the third time period corresponding to the second acceleration average value is greater than or equal to the second disparity threshold, respectively determine the fourth image frame or the fifth image frame as the first key frame and the seventh image frame as the second key frame, and determine the first gyroscope average value based on the IMU data obtained by the VIO system in the third time period corresponding to the first acceleration average value, wherein the first disparity threshold is less than the second disparity threshold;

[0201] An initial value of the VIO system is determined according to the first acceleration mean, the first gyroscope mean, the first key frame, and the second key frame.

[0202] In some possible embodiments, after the sending unit 720 sends the first indication information, the sending unit 720 is further configured to:

[0203] If it is still not determined whether the self-moving device has come to rest after the fifth period of time, the first indication information is resent to the control device.

[0204] In some possible embodiments, during the first time period, the sending unit 720 is further used to: when the processing unit 730 has determined the first key frame, send a second indication message to the control device of the self-mobile device, and the second indication message is used to instruct the control device to control the self-mobile device to start in advance.

[0205] In some possible embodiments, the present application proposes a self-propelled device, comprising: a processor, a controller, and a VIO system, wherein the VIO system includes an image acquisition device and an IMU;

[0206] The processor is configured to obtain first status information of the VIO system; and send first indication information to the controller according to the first status information, wherein the first indication information is configured to instruct the controller to control the mobile device to remain stationary for a first period of time and then start.

[0207] The image acquisition device is used to acquire image frames at a first frame rate;

[0208] The IMU is configured to acquire IMU data at a second frame rate, wherein the second frame rate is greater than the first frame rate;

[0209] The processor is further configured to determine a motion state of the self-moving device based on image frames acquired by the image acquisition device, and when it is determined that the self-moving device is stationary, perform a static initialization operation on the VIO system based on data acquired by the image acquisition device and the IMU to determine an initial value of the VIO system;

[0210] The first status information includes at least one of the following:

[0211] The VIO system initiates a first signal, where the first signal is used to indicate that the VIO system is about to be initially started, or,

[0212] During the VIO system processing data, data anomalies occur, or

[0213] The preset time has elapsed since the last static initialization operation was completed.

[0214] In some possible embodiments, the present application further proposes a self-moving device, which includes the apparatus 700 described in any of the above embodiments.

[0215] FIG8 is a schematic block diagram of a computer device 800 according to an embodiment of the present application. The computer device 800 shown in FIG8 includes a memory 810, a processor 820, and a bus 840. Optionally, the computer device 800 also includes a communication interface 830. The memory 810, processor 820, and communication interface 830 are interconnected via bus 840.

[0216] The memory 810 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 810 can store programs. When the program stored in the memory 810 is executed by the processor 820, the processor 820 is used to perform the various steps of the VIO system initialization method of the embodiment of the present application. For example, the processor 820 can execute the method shown in Figures 2 to 6 above.

[0217] The processor 820 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the VIO system initialization method proposed in the method embodiment of the present application.

[0218] The processor 820 can also be an integrated circuit chip with signal processing capabilities. During the implementation process, the various steps of the VIO system initialization method of the present application can be completed through the hardware integrated logic circuit in the processor 820 or software instructions.

[0219] The processor 820 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods involved in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or the like. The storage medium is located in the memory 810, and the processor 820 reads the information in the memory 810 and, in combination with its hardware, completes the functions required to be executed by the units included in the apparatus shown in FIG. 7 , or executes the methods shown in FIG. 2 to FIG. 6 of the method embodiments of the present application.

[0220] The communication interface 830 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 800 and other devices or a communication network.

[0221] The bus 840 may include a path for transmitting information between various components of the device 800 (eg, the memory 800 , the processor 820 , and the communication interface 830 ).

[0222] It should be understood that although the above-mentioned device 800 only shows a memory, a processor, and a communication interface, during the specific implementation process, those skilled in the art will understand that the device 800 may also include other components necessary for normal operation. At the same time, according to specific needs, those skilled in the art will understand that the device 800 may also include hardware components that implement other additional functions. In addition, those skilled in the art will understand that the device 800 may also include only the components necessary to implement the embodiments of the present application, and does not necessarily include all the components shown in Figure 8.

[0223] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0224] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0225] Figure 9 is a schematic block diagram of a computer-readable storage medium 900 provided in an embodiment of the present application. The computer-readable storage medium 900 shown in Figure 9 stores computer instructions 910. When the computer instructions 910 are executed by a processor, the method corresponding to the above embodiment can be implemented.

[0226] In some possible embodiments, the computer-readable storage medium 900 can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes a collection of one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, or a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0227] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0228] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0229] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0231] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0232] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0233] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for initializing a visual inertial odometry (VIO) system, characterized in that: Applied to a self-moving device equipped with a VIO system, the method comprises: Acquire first status information of the VIO system; Sending first indication information to a control device of the self-moving device according to the first state information, wherein the first indication information is used to instruct the control device to control the self-moving device to first remain stationary within a first period of time and then start; When it is determined that the mobile device is stationary, performing a static initialization operation on the VIO system to determine an initial value of the VIO system; The first status information includes at least one of the following: The VIO system initiates a first signal, where the first signal is used to indicate that the VIO system is to be initially started, or, During the process of data processing by the VIO system, data anomalies occur, or, A preset time has elapsed since the last time the static initialization operation was completed.

2. The method according to claim 1, characterized in that: The image acquisition device in the VIO system is a monocular camera, and when it is determined that the mobile device is stationary, performing a static initialization operation on the VIO system includes: Determining a first key frame and a second key frame in sequence, wherein the first key frame is acquired by the monocular camera when it is determined that the mobile device is stationary, and a disparity between the first key frame and the second key frame is greater than or equal to a second disparity threshold; Determining a second time period according to a first time instant of acquiring the first key frame and a second time instant of acquiring the second key frame; Divide the second time period into a first sub-time period and a second sub-time period on an even basis, wherein the first sub-time period is located before the second sub-time period in terms of time sequence; Determine an acceleration mean and a gyroscope mean according to the inertial measurement unit IMU data acquired by the VIO system in the first sub-period, wherein the acceleration mean is a vector; An initial value of the VIO system is determined according to the acceleration mean, the gyroscope mean, the first key frame, and the second key frame.

3. The method according to claim 2, characterized in that The sequentially determining the first key frame and the second key frame comprises: Determining a first image frame and a second image frame in sequence, and determining the first image frame or the second image frame as the first key frame when the disparity between the first image frame and the second image frame is less than or equal to a first disparity threshold, and the first disparity threshold is less than the second disparity threshold; A third image frame is determined, and when the disparity between the third image frame and the first key frame is greater than or equal to the second disparity threshold, the third image frame is determined to be the second key frame.

4. The method according to claim 1, characterized in that The image acquisition device in the VIO system is a monocular camera, and when it is determined that the mobile device is stationary, performing a static initialization operation on the VIO system includes: Determine, according to the IMU data acquired by the VIO system in each third time period, a mean acceleration value in each third time period, wherein the mean acceleration value is a vector; Determine a first acceleration mean and a second acceleration mean in sequence, wherein the first acceleration mean is less than or equal to a first acceleration threshold, the second acceleration mean is greater than or equal to a second acceleration threshold, the first acceleration threshold is less than the second acceleration threshold, and a third time period corresponding to the first acceleration mean and a third time period corresponding to the second acceleration mean are adjacent time periods; When the disparity between the fourth image frame and the fifth image frame obtained in the third time period corresponding to the first acceleration mean value is less than or equal to the first disparity threshold, and the disparity between the sixth image frame and the seventh image frame obtained in the third time period corresponding to the second acceleration mean value is greater than or equal to the second disparity threshold, the fourth image frame or the fifth image frame is respectively determined as the first key frame, and the seventh image frame is determined as the second key frame, and the first gyroscope mean is determined according to the IMU data obtained by the VIO system in the third time period corresponding to the first acceleration mean value, wherein the first disparity threshold is less than the second disparity threshold; An initial value of the VIO system is determined according to the first acceleration mean, the first gyroscope mean, the first key frame, and the second key frame.

5. The method according to any one of claims 1 to 4, characterized in that After sending the first indication information, the method further includes: If it is still not determined whether the self-moving device has come to a standstill after the fifth period of time, the first indication information is resent to the control device.

6. The method according to any one of claims 2 to 4, characterized in that During the first period, the method further includes: In the case where the first key frame has been determined, second indication information is sent to the control device of the self-moving device, where the second indication information is used to instruct the control device to control the self-moving device to start in advance.

7. A device for initializing a VIO system, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program codes, and the processor is used to call the program codes to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program code or instruction is executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

10. A self-propelled device, characterized in that: include: A processor, a controller and a VIO system, wherein the VIO system includes an image acquisition device and an IMU; The processor is used to obtain first state information of the VIO system; send first indication information to the controller according to the first state information, wherein the first indication information is used to instruct the controller to control the self-moving device to remain still for a first period of time and then start; The image acquisition device is used to acquire image frames according to a first frame rate; The IMU is used to obtain IMU data according to a second frame rate, and the second frame rate is greater than the first frame rate; The processor is further configured to determine the motion state of the self-moving device according to the image frames acquired by the image acquisition device, and when it is determined that the self-moving device is stationary, determine the motion state of the self-moving device according to the image acquisition device and the IMU The acquired data is used to perform a static initialization operation on the VIO system to determine an initial value of the VIO system; The first status information includes at least one of the following: The VIO system initiates a first signal, where the first signal is used to indicate that the VIO system is to be initially started, or, During the process of data processing by the VIO system, data anomalies occur, or, A preset time has elapsed since the last time the static initialization operation was completed.

Images