Method, apparatus, and self-moving device for vio system initialization

Abstract

The method, device and system for initializing a VIO system relate to the technical field of visual positioning and navigation, and the method comprises the following steps: obtaining state information of the VIO system; sending indication information to a control device of a self-moving device according to the state information, the indication information being used for instructing the control device to control the self-moving device to first be stationary and then be started; and when it is determined that the self-moving device has been stationary, performing a static initialization operation on the VIO system to determine an initial value of the VIO system, the state information comprising at least one of the following: the VIO system initiating a first signal, the first signal being used for indicating that the VIO system is about to be initially started, or data abnormality occurring in a data processing process of the VIO system, or a preset time duration being experienced since a time point at which a last static initialization operation is completed. Based on the scheme, the VIO system can be provided with a condition for static initialization, so as to implement a static initialization operation on the VIO system, and the efficiency of initialization can be improved.

Description

Technical Field

[0001] This application relates to the field of visual positioning and navigation technology, and more specifically, to a method, apparatus, and self-moving device for initializing a VIO system. Background Technology

[0002] Visual-inertial odometry (VIO) can be divided into two categories: loosely coupled and tightly coupled. Tightly coupled VIO refers to merging the states of the image acquisition device and the inertial measurement unit (IMU) to construct motion and observation equations, and then performing unified motion estimation. However, tightly coupled VIO systems are highly nonlinear systems, especially those using monocular cameras. In vision-based real-time localization and mapping applications, the detection scale and rotation relative to gravity of the monocular camera are insignificant. Therefore, tightly coupled VIO systems require a good initial value when fusing visual and IMU data, which can be determined by initializing the VIO system.

[0003] Therefore, a method for initializing a VIO system is urgently needed. Summary of the Invention

[0004] This application provides a method, apparatus, and self-moving device for initializing a VIO system, which can provide the conditions for static initialization of the VIO system, enabling the VIO system to complete static initialization operations.

[0005] In a first aspect, a method for initializing a VIO system is provided. The method includes: acquiring first state information of the VIO system; sending first indication information to a control device of a self-moving device based on the first state information, the first indication information being used to instruct the control device to control the self-moving device to remain stationary for a first time period and then start; and, when it is determined that the self-moving device is stationary, performing a static initialization operation on the VIO system to determine the initial value of the VIO system; wherein the first state information includes at least one of the following: the VIO system initiates a first signal, the first signal being used to indicate that the VIO system is about to start initially, or, during the VIO system's data processing, a data anomaly occurs, or, a preset time period has elapsed since the last completion of the static initialization operation.

[0006] For example, the first time period mentioned above is a preset reasonable time period. The length of this time period is sufficient to ensure that the static initialization operations for the VIO system can be completed normally.

[0007] For example, during the initialization of the VIO system, if any of the following situations occur, a data anomaly is considered to have occurred:

[0008] 1. In the process of fusing visual information and IMU data, there is a situation where the estimated state diverges.

[0009] 2. The VIO system failed to track map points for an extended period of time.

[0010] 3. Based on auxiliary sensors, determine that the visual positioning error of the VIO system is greater than or equal to the error threshold.

[0011] Among them, the auxiliary sensor can be the Global Positioning System (GPS).

[0012] For example, the preset duration can be a reasonable preset duration, such as half an hour. After the initialization operation of the VIO system is completed, it can be considered that the accumulated error of the VIO system has been eliminated. However, during the subsequent operation of the VIO system, the error will accumulate again. Therefore, based on a timer triggering mechanism, starting from the time when the last static initialization is completed, the VIO system can be instructed to perform an initialization operation every preset duration.

[0013] Based on the above technical solution, it is possible to determine whether the current VIO system needs to be initialized by obtaining the status information of the VIO system. When it is determined that the VIO system meets the initialization conditions, the self-moving device is actively controlled to remain still for a first time period before starting up, thereby creating conditions for performing static initialization. This helps to improve the efficiency, success rate and robustness of initializing the VIO system.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, when the image acquisition device in the aforementioned VIO system is a monocular camera, a first keyframe and a second keyframe are determined sequentially. The first keyframe is acquired by the monocular camera when the self-moving device is determined to be stationary, and the disparity between the first keyframe and the second keyframe is greater than or equal to a second disparity threshold. A second time period is determined based on the first moment of acquiring the first keyframe and the second moment of acquiring the second keyframe. The second time period is divided into a first sub-time period and a second sub-time period, with the first sub-time period preceding the second sub-time period in time sequence. The average acceleration and the average gyroscope value are determined based on the IMU data acquired by the VIO system during the first sub-time period, with the average acceleration being a vector. The initial values ​​of the VIO system are determined based on the average acceleration, the average gyroscope value, the first keyframe, and the second keyframe.

[0015] For example, the first, second, and third image frames mentioned above are acquired by an image acquisition device, and the acquisition sequence of these three image frames is not necessarily consecutive. However, the first image frame is acquired first, followed by the second image frame, and the third image frame is acquired last. The first image frame can be referred to as the initial frame, which is the first image frame acquired after the first indication information is sent.

[0016] For example, disparity between image frames can be determined by feature point extraction and feature matching performed between image frames. It should be understood that feature points typically refer to corner points or representative blocks and edges in an image, possessing the property of remaining unchanged after slight changes in camera viewpoint. Typically, corner points in an image are chosen as feature points.

[0017] For example, the IMU data mentioned above may include acceleration data and gyroscope data.

[0018] For example, the aforementioned acceleration mean is used to determine the acceleration zero bias and gravity direction in the initial values ​​of the VIO system, the aforementioned gyroscope mean is used to determine the gyroscope zero bias in the initial values ​​of the VIO system, and the aforementioned second image frame and the aforementioned third image frame are used to determine the feature point depth in the initial values ​​of the VIO system.

[0019] Based on the above technical solution, detection data during the period when the self-moving device undergoes a motion jump can be reasonably obtained. Static initialization of the VIO system based on a monocular camera can be completed based on the detection data. This initialization method is efficient, has a high success rate, and is robust.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, a first image frame and a second image frame are determined sequentially. When the disparity between the first image frame and the second image frame is less than or equal to a first disparity threshold, the first image frame or the second image frame is determined as a first key frame, and the first disparity threshold is less than a second disparity threshold. A third image frame is determined, and when the disparity between the third image frame and the first key frame is greater than or equal to a second disparity threshold, the third image frame is determined as a second key frame.

[0021] For example, the aforementioned first disparity threshold is a preset threshold, such as being equal to 0. In this case, the disparity between the first image frame and the second image frame should be equal to the first disparity threshold. Considering that the sensor may have detection errors, the aforementioned first disparity threshold can be preset to a value close to 0. In this case, the disparity between the first image frame and the second image frame should be less than the first disparity threshold. Correspondingly, the second disparity threshold can also be a preset threshold, specifically a reasonable threshold preset based on the application scenario and motion performance of the mobile device.

[0022] Based on the above technical solution, it is possible to determine whether the self-moving device is in a stationary state, so as to ensure that the static initialization of the VIO system can be performed under the condition that the self-moving device is already stationary.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, when the image acquisition device in the aforementioned VIO system is a monocular camera, the average acceleration value within each third time period is determined based on the IMU data acquired by the VIO system in each third time period. This average acceleration value is a vector. A first average acceleration value and a second average acceleration value are determined sequentially, wherein the first average acceleration value is less than or equal to a first acceleration threshold, the second average acceleration value is greater than or equal to a second acceleration threshold, the first acceleration threshold is less than the second acceleration threshold, and the third time period corresponding to the first average acceleration value and the third time period corresponding to the second average acceleration value are adjacent time periods. When the disparity between the fourth and fifth image frames acquired in the third time period is less than or equal to the first disparity threshold, and the disparity between the sixth and seventh image frames acquired in the third time period corresponding to the second acceleration mean is greater than or equal to the second disparity threshold, the fourth or fifth image frame is determined as the first keyframe, and the seventh image frame is determined as the second keyframe. Based on the IMU data acquired by the VIO system in the third time period corresponding to the first acceleration mean, the first gyroscope mean is determined, wherein the first disparity threshold is less than the second disparity threshold. Based on the first acceleration mean, the first gyroscope mean, the first keyframe, and the second keyframe, the initial value of the VIO system is determined.

[0024] For example, the first acceleration threshold mentioned above is a preset threshold. Ideally, the first acceleration threshold should be equal to 0, but in practical applications, sensors may have detection errors, so the first acceleration threshold can be a value close to 0, such as 0.1 m / s². 2 0.2m / s 2 Etc. Correspondingly, the aforementioned second acceleration threshold is also a preset threshold, which can be a reasonable threshold determined based on the application scenario and motion performance of the mobile device, for example, 1 m / s². 2 2m / s 2 wait.

[0025] Based on the above technical solution, it is possible to determine whether the self-moving device is stationary and to reasonably obtain the detection data during the period when the self-moving device undergoes a motion change. The static initialization of the VIO system based on the monocular camera is completed based on the detection data. This initialization method is efficient, has a high success rate, and is robust.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, after sending the first instruction information, and if it is still not determined whether the self-moving device has stopped after a fifth time period, the first instruction information is resent to the control device.

[0027] Based on the above technical solution, a retransmission mechanism for the first indication information is introduced, which can effectively avoid the problem of failure to receive the first indication information, thereby ensuring that the initialization of the VIO system can proceed normally. This increases the reliability of static initialization of the VIO system.

[0028] In conjunction with the first aspect, in some implementations of the first aspect, during the aforementioned first time period, when the first keyframe has been determined, second instruction information is sent to the control device of the self-moving device, the second instruction information being used to instruct the control device to control the self-moving device to start in advance.

[0029] Based on the above technical solution, considering that the first time period is reserved for the VIO system to collect relevant data of the self-moving device in a stationary state, the reserved time in the first time period may be too long. When the above collection process is completed in advance, the self-moving device can be started in advance by the second indication information, so that the static initialization can enter the subsequent process, thereby increasing the execution efficiency of static initialization.

[0030] Secondly, a device for initializing a VIO system is provided. This device is applied to a self-moving device equipped with a VIO system. The device includes: an acquisition unit for acquiring first state information of the VIO system; a transmission unit for transmitting first indication information to a control device of the self-moving device based on the first state information, wherein the first indication information is used to instruct the control device to control the self-moving device to remain stationary for a first time period and then start; and a processing unit for performing a static initialization operation on the VIO system when it is determined that the self-moving device is stationary, so as to determine the initial value of the VIO system.

[0031] The aforementioned first state information includes at least one of the following: the VIO system initiates a first signal, which indicates that the VIO system is about to start up; or, a data anomaly occurs during the VIO system's data processing; or, a preset time has elapsed since the last completion of the static initialization operation.

[0032] In conjunction with the second aspect, in some implementations of the second aspect, when the image acquisition device in the aforementioned VIO system is a monocular camera, the aforementioned processing unit is specifically used to: sequentially determine a first keyframe and a second keyframe, wherein the first keyframe is acquired by the monocular camera when the self-moving device is determined to be stationary, and the disparity between the first keyframe and the second keyframe is greater than or equal to a second disparity threshold; determine a second time period based on the first time of acquiring the first keyframe and the second time of acquiring the second keyframe; divide the second time period into a first sub-time period and a second sub-time period, wherein the first sub-time period is sequentially preceding the second sub-time period; determine the average acceleration value and the average gyroscope value based on the inertial measurement unit (IMU) data acquired by the VIO system during the first sub-time period, wherein the average acceleration value is a vector; and determine the initial values ​​of the VIO system based on the average acceleration value, the average gyroscope value, the first keyframe, and the second keyframe.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the above processing unit is specifically used to: sequentially determine a first image frame and a second image frame; when the disparity between the first image frame and the second image frame is less than or equal to a first disparity threshold, determine the first image frame or the second image frame as a first key frame, wherein the first disparity threshold is less than a second disparity threshold; determine a third image frame; when the disparity between the third image frame and the first key frame is greater than or equal to the second disparity threshold, determine the third image frame as a second key frame.

[0034] In conjunction with the second aspect, in some implementations of the second aspect, when the image acquisition device in the aforementioned VIO system is a monocular camera, the aforementioned processing unit is specifically used to: determine the average acceleration value within each third time period based on the IMU data acquired by the VIO system within each third time period, wherein the average acceleration value is a vector; sequentially determine the first average acceleration value and the second average acceleration value, wherein the first average acceleration value is less than or equal to a first acceleration threshold, the second average acceleration value is greater than or equal to a second acceleration threshold, the first acceleration threshold is less than the second acceleration threshold, and the third time period corresponding to the first average acceleration value and the third time period corresponding to the second average acceleration value are adjacent time periods; in the first acceleration... When the disparity between the fourth and fifth image frames acquired in the third time period corresponding to the average velocity is less than or equal to the first disparity threshold, and the disparity between the sixth and seventh image frames acquired in the third time period corresponding to the average acceleration is greater than or equal to the second disparity threshold, the fourth or fifth image frame is determined as the first keyframe, and the seventh image frame is determined as the second keyframe. Based on the IMU data acquired by the VIO system in the third time period corresponding to the average acceleration, the first gyroscope average is determined, wherein the first disparity threshold is less than the second disparity threshold. Based on the first average acceleration, the first gyroscope average, the first keyframe, and the second keyframe, the initial values ​​of the VIO system are determined.

[0035] In conjunction with the second aspect, in some implementations of the second aspect, after the sending unit sends the first instruction information, the sending unit is further configured to: if it is still not determined whether the self-moving device has stopped after the fifth time period, resend the first instruction information to the control device.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, during the first time period, the sending unit is further configured to: send second indication information to the control device of the self-moving device when the processing unit has determined the first key frame, the second indication information being used to instruct the control device to control the self-moving device to start in advance.

[0037] Thirdly, a device for initializing a VIO system is provided, comprising a processor and a memory, wherein the processor and the memory are connected, wherein the memory is used to store program code, and the processor is used to call the program code to execute any possible implementation of the method design in the first aspect above.

[0038] Fourthly, a chip system is provided, which is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected by lines; the interface circuits are used to receive echo signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes any possible implementation of the method design of the first aspect above.

[0039] Fifthly, a computer-readable storage medium is provided storing a computer program or instructions for implementing the method in any possible implementation of the method design of the first aspect.

[0040] In a sixth aspect, a computer program product is provided, wherein when the computer program code or instructions are executed on a computer, the computer performs the method in any possible implementation of the method design of the first aspect described above.

[0041] In a seventh aspect, a self-moving device is provided, comprising: a processor, a controller, and a VIO system, wherein the VIO system includes an image acquisition device and an IMU;

[0042] The processor is configured to: acquire first state information of the VIO system; send first indication information to the controller based on the first state information, the first indication information being used to instruct the controller to control the mobile device to remain stationary for a first time period and then start; acquire image frames at a first frame rate; acquire IMU data at a second frame rate, wherein the second frame rate is greater than the first frame rate; and determine the motion state of the mobile device based on the image frames acquired by the image acquisition device. If the mobile device is determined to be stationary, the processor performs a static initialization operation on the VIO system based on the data acquired by the image acquisition device and the IMU to determine the initial value of the VIO system. The first state information includes at least one of the following: the VIO system initiates a first signal, the first signal indicating that the VIO system is about to start initially; or, a data anomaly occurs during data processing by the VIO system; or, a preset time period has elapsed since the last completion of the static initialization operation.

[0043] Eighthly, a self-moving device is provided, which includes means as in any possible implementation of the second or third aspect. Attached Figure Description

[0044] Figure 1 This is a schematic block diagram of a VIO system 100 provided in an embodiment of this application;

[0045] Figure 2 This is a flowchart illustrating a method 200 for initializing a VIO system provided in an embodiment of this application;

[0046] Figure 3 This is a flowchart illustrating a static initialization method 300 provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram illustrating the determination of a first keyframe and a second keyframe according to an embodiment of this application;

[0048] Figure 5 This is a flowchart illustrating another static initialization method 500 provided in an embodiment of this application;

[0049] Figure 6 This is a flowchart illustrating another static initialization method 600 provided in an embodiment of this application;

[0050] Figure 7 This is a schematic block diagram of a VIO system initialization device 700 provided in an embodiment of this application;

[0051] Figure 8 This is a schematic block diagram of a computer device 800 provided in an embodiment of this application;

[0052] Figure 9 This is a schematic block diagram of a computer-readable storage medium 900 provided in an embodiment of this application. Detailed Implementation

[0053] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In this application, "at least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0054] The use of prefixes such as "first" and "second" in this application embodiment is solely for distinguishing different descriptive objects and does not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes to distinguish descriptive objects in this application embodiment does not constitute a limitation on the described objects. The description of the described objects is found in the claims or the context of the embodiments, and the use of such prefixes should not constitute unnecessary restrictions.

[0055] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0056] VIO (Virtual I / O) is typically mounted on mobile devices to provide them with real-time map and location information. It fuses data from both the image acquisition device and the IMU (Integrated Measurement Unit). While calculating the camera pose from two keyframes acquired by the image acquisition device, it also uses IMU measurement data as model constraints to achieve instant localization and map building. Furthermore, because the IMU can output various navigation parameters, it can also provide the camera with detailed motion information for each frame of the image.

[0057] In some possible embodiments, the self-moving device proposed in this application can be any mechanical device capable of highly autonomous spatial movement, such as a mobile robot, a portable air purifier, a drone, or a driverless vehicle. Among these, mobile robots can include robotic vacuum cleaners, lawnmowers, companion robots, or guide robots.

[0058] VIO systems integrated into self-moving devices can be categorized into two types: loosely coupled and tightly coupled. Loosely coupled VIO systems perform motion estimation based on image frames acquired by an image acquisition device and motion calculation based on motion data from the self-moving device acquired by an IMU (Installation Detector). The results of these two motion estimations and calculations are then fused. While loosely coupled VIO systems have simpler algorithms, the processes of image acquisition and IMU data processing are independent, making it difficult to fully utilize both sets of detection data and ensuring good detection performance. Tightly coupled VIO systems, on the other hand, combine the detection data from the image acquisition device and the IMU to construct motion and observation equations, then perform unified motion estimation for the self-moving device. Compared to loosely coupled VIO systems, tightly coupled VIO systems, although more complex in algorithm, fully utilize the detection data from both the image acquisition device and the IMU, achieving better detection results and being applicable to more complex application scenarios. This makes them a key research focus in the field of visual positioning and navigation technology.

[0059] However, tightly coupled VIO systems are highly nonlinear systems, especially VIO systems with monocular cameras as image acquisition devices. In vision-based real-time localization and mapping applications, the detection scale of the monocular camera and its rotation relative to the direction of gravity are immeasurable. Therefore, tightly coupled VIO systems require a good initial value when fusing visual data and IMU data, and this initial value can be determined by initializing the VIO system.

[0060] Typically, VIO systems are initialized using either dynamic or static initialization. Dynamic initialization refers to the method of initializing the VIO system while the mobile device is in motion. Although the mobile device is in motion for most of its operation, it is often difficult for it to fully meet the motion conditions of dynamic initialization during this period. Furthermore, dynamic initialization is computationally inefficient and its results are difficult to converge. In contrast, static initialization has lower computational complexity, is easier to converge, and is more efficient. Similar to dynamic initialization, static initialization also requires the mobile device to meet certain motion conditions, i.e., the mobile device needs to remain stationary for a period of time. However, compared to dynamic initialization, the motion conditions for static initialization are more explicit and easier to achieve. In summary, static initialization has lower requirements, requires less time, has a higher success rate, and is more robust.

[0061] However, based on existing VIO system initialization methods, static initialization is a passive process. It only performs static initialization when the mobile device is confirmed to be stationary. If the mobile device is in motion for an extended period, VIO initialization will not occur, leading to a gradual increase in accumulated errors and reduced accuracy. Alternatively, another existing VIO system initialization method also involves a passive process. Upon detecting the need for initialization, it first determines the motion state of the mobile device. If stationary, static initialization is performed; otherwise, dynamic initialization is executed. However, this dynamic initialization method also suffers from the problems mentioned above, resulting in poor overall initialization efficiency and difficulty in guaranteeing robustness.

[0062] In view of this, this application proposes a method for initializing a VIO system. By obtaining the state information of the VIO system, the method determines whether the VIO system meets the initialization conditions based on the state information. If the VIO system meets the initialization conditions, the method instructs the control device of the self-moving device to stop the self-moving device, thereby providing conditions for performing static initialization operations on the VIO system.

[0063] Figure 1 This is a schematic block diagram of a VIO system 100 provided in an embodiment of this application.

[0064] In some possible embodiments, the VIO system may include an image acquisition device, an IMU, and a processing device.

[0065] In some possible embodiments, the image acquisition device can be a monocular camera or a binocular camera. Specifically, it can be an optical three-primary-color (red-green-blue, RGB) camera, an infrared (IR) camera, or a depth camera; this application embodiment does not limit this. The image acquisition device in this application embodiment is used to acquire image data from the surroundings of the mobile device.

[0066] In some possible embodiments, the IMU is used to acquire motion data from the mobile device. The IMU includes an accelerometer and a gyroscope, used to acquire acceleration data and angular velocity data from the mobile device, respectively.

[0067] In some possible embodiments, the processing device is used to acquire the state information of the VIO system, determine whether the VIO system meets the initialization conditions, and instruct the control device of the self-moving device to stop the self-moving device, so as to provide conditions for performing static initialization operations on the VIO system.

[0068] In some possible embodiments, the VIO system described above can be mounted on a self-moving device. In this system architecture, the control device is a self-moving device, and the processing device can be a specific functional module in the central processing unit (CPU) of the self-moving device.

[0069] In some possible embodiments, the processing device described above may also be a processor independent of the CPU of the self-moving device, which is capable of establishing a connection with the CPU of the self-moving device and thus instructing the control device to stop the self-moving device.

[0070] In some possible embodiments, the above-mentioned processing device can also be mounted in a server. When the self-moving device establishes a communication connection with the server, when the processing device determines that the VIO system needs to perform static initialization, it can send corresponding instruction information to the control device of the self-moving device through the server, instructing the control device to stop the self-moving device.

[0071] Based on the above-mentioned VIO system, this application proposes a method for initializing the VIO system, so as to enable the self-moving device to actively stop when the VIO system needs to perform static initialization operations, thereby providing conditions for performing static initialization operations on the VIO system.

[0072] Figure 2 This application provides a schematic flowchart of a VIO system initialization method 200. This method 200 is applied to self-moving devices equipped with a VIO system.

[0073] S210: Obtain the first state information of the VIO system.

[0074] The first state information mentioned above is used to indicate the operating state of the VIO system and may include at least one of the following:

[0075] The VIO system initiates a first signal, which indicates that the VIO system is about to start up, or...

[0076] During the data processing process of the VIO system, a data anomaly occurs, or...

[0077] The preset time has elapsed since the last static initialization operation was completed.

[0078] S220: The control device that sends the first instruction information to the self-moving device based on the first status information.

[0079] S230: When it is determined that the self-moving device is stationary, a static initialization operation is performed on the above-mentioned VIO system to determine the initial value of the above-mentioned VIO system.

[0080] It should be understood that if the VIO system is not initialized before each startup, the cumulative error of the VIO system will gradually increase with the accumulation of operating time, affecting the accuracy of VIO system detection. Furthermore, the VIO system and the self-moving device usually start synchronously, meaning that the self-moving device's state will inevitably change from stationary to moving. This directly satisfies the condition for static initialization of the VIO system. Therefore, performing initialization once before each startup is appropriate and necessary.

[0081] In some possible embodiments, the first signal may be a signal sent globally or to the processing device of the VIO system 100 before the VIO system is started.

[0082] In some possible embodiments, if any of the following occurs during the initialization of the VIO system, a data anomaly is considered to have occurred:

[0083] 1. In the process of fusing visual information and IMU data, there is a situation where the estimated state diverges.

[0084] A VIO system includes an image acquisition device and an IMU. During the operation of the VIO system, it is necessary to fuse the sensor data acquired by the image acquisition device and the sensor data acquired by the IMU. For example, state estimation is performed using the Falman filter algorithm. If the covariance of the estimated state diverges, it indicates that the VIO system has experienced a data anomaly, also known as "runaway".

[0085] 2. The VIO system failed to track map points for an extended period of time.

[0086] Tracking map points is the main function of the VIO system. If the VIO system cannot continuously and normally track map points, it means that the VIO system is not running normally. This situation also falls under the "running out of control" situation mentioned above. This abnormal situation needs to be resolved by initializing the VIO system.

[0087] The preset time period can be a reasonable time interval, such as 2 seconds. Furthermore, the preset time period can be adjusted at any time based on the application scenario of the self-mobile device. In necessary cases, such as when the self-mobile device is used in a more complex scenario, the preset time period can be equal to the unit cycle duration of the image acquisition device, that is, map point detection is performed for each frame of the image. If a map point is not tracked in a certain frame of the image, it indicates that the VIO system has experienced a data anomaly.

[0088] 3. Based on auxiliary sensors, determine that the visual positioning error of the VIO system is greater than or equal to the error threshold.

[0089] Self-moving devices are usually equipped with auxiliary sensors, such as GPS. While the VIO system performs visual positioning, the GPS also collects the location information of the self-moving device. Therefore, after the VIO system outputs the visual positioning result, it can be verified with the GPS positioning result to determine the visual positioning error of the VIO system. When the visual positioning error is greater than or equal to the error threshold, it indicates that the cumulative error of the VIO system is too large, which also indicates that the VIO system has data anomalies.

[0090] In some possible embodiments, the aforementioned preset duration can be a reasonable preset duration, such as half an hour. After the initialization operation for the VIO system is completed, it can be considered that the accumulated error of the VIO system has been eliminated. However, during the subsequent operation of the VIO system, errors will accumulate again. Therefore, a timer triggering mechanism can be used to instruct the VIO system to perform an initialization operation at regular intervals. The specific process is as follows: Before starting the timer triggering mechanism, the preset duration is determined. Then, after completing this static initialization operation, the current time is recorded, and the timer is started. After the preset duration has elapsed, the timer will trigger a second signal. This second signal indicates that the preset duration has elapsed since the last completion of the static initialization operation. The information carried by this second signal belongs to the aforementioned first state information.

[0091] Based on the existing VIO system initialization methods mentioned above, if any of the three situations mentioned in S220 occurs, the VIO system can only passively determine the current motion state of the self-moving device. Suppose that a data processing anomaly occurs during the self-moving device's movement, the self-moving device's motion state will obviously not meet the conditions for static initialization. The VIO system will not execute static initialization in a timely manner, so this abnormal state will continue until the self-moving device is detected to be stationary, at which point static initialization can be executed. Even if the VIO system can perform dynamic initialization, based on the known shortcomings of dynamic initialization, its execution efficiency is relatively low.

[0092] In some possible embodiments, when the control device receives the first instruction information, the movement state of the self-moving device can be stationary or in motion, and this application embodiment does not limit this. Even if the self-moving device is in a state of about to start when the control device receives the first instruction information, the control device will control the self-moving device to remain stationary for a first time period before starting it, rather than controlling the self-moving device to start immediately.

[0093] In some possible embodiments, the aforementioned first time period is a preset reasonable time period. The length of this time period needs to ensure that the static initialization operations for the VIO system can be completed normally, typically less than 2 seconds. It should not be too long or too short. These operations may involve the subsequent acquisition of two frames by the VIO system's monocular camera to confirm that the self-moving device is in a stationary state. If the first time period is too long, it will forcibly reduce the efficiency of static initialization; if the first time period is too short, it will cause the VIO system to fail to collect detection data that meets the static initialization requirements in a timely manner, resulting in static initialization failure.

[0094] In some possible embodiments, the first time period described above may also be determined based on the image acquisition frame rate of the VIO system. For example, the higher the image acquisition frame rate, the shorter the first time period.

[0095] In some possible embodiments, the static initialization described above includes the initialization of IMU data and the initialization of visual data. The initialization objects for IMU data include gyroscope zero bias, acceleration zero bias, and gravity direction, while the initialization objects for visual data include feature point depth. Therefore, the initial values ​​of the VIO system described above include gyroscope zero bias, acceleration zero bias, gravity direction, and feature point depth. The gyroscope zero bias, acceleration zero bias, and gravity direction can be determined based on IMU data acquired by the self-moving device in a stationary state, while the feature point depth can be determined based on two frames of images acquired by the self-moving device in a moving state.

[0096] Based on the above technical solution, it is possible to determine whether the current VIO system needs to be initialized by obtaining the status information of the VIO system. When it is determined that the VIO system meets the initialization conditions, the self-moving device is actively controlled to remain still for a first time period before starting up, thereby creating conditions for performing static initialization. This helps to improve the efficiency, success rate and robustness of initializing the VIO system.

[0097] In some possible embodiments, when the image acquisition device in the VIO system is a monocular camera, the above-mentioned S240 can be implemented in the following manner.

[0098] Figure 3 This is a flowchart illustrating a static initialization method 300 provided in an embodiment of this application.

[0099] S310: Sequentially determine the first keyframe and the second keyframe, wherein the first keyframe is acquired by a monocular camera when the self-moving device is determined to be stationary, and the disparity between the first keyframe and the second keyframe is greater than or equal to the second disparity threshold.

[0100] In some possible embodiments, the disparity between image frames can be determined by extracting and matching feature points across the frames. It should be understood that feature points typically refer to corner points or representative blocks and edges in an image, possessing the characteristic of remaining unchanged after slight changes in camera viewpoint. Typically, corner points in the image are chosen as feature points.

[0101] In some possible implementations, corner extraction of image frames can be achieved using algorithms such as the Moravec corner detection algorithm, the Harris corner detection algorithm, or the Shi-Tomasi algorithm.

[0102] In some possible embodiments, feature matching between image frames includes feature point method and direct method. The former is based on descriptor matching and optimizes the solution by minimizing reprojection error, while the latter is based on optical flow tracing, which minimizes the photometric error of image pixels and then directly uses the grayscale information of image pixels for matching.

[0103] Taking the Harris corner detection algorithm as an example, the Harris corner measurement formula can be used to select the feature point with the largest corner response value as the final result. Then, a fast (FAST) feature point detection algorithm is used to detect the first gray-level difference between the specified pixel and its surrounding pixels. If there are enough pixels corresponding to the first gray-level difference that exceed a preset gray-level threshold, it indicates that the specified pixel may be a corner. Then, the Harris corner response value is measured to determine whether it is a corner. After identifying multiple sets of feature points between two frames, the inter-frame relative pose of the two frames can be calculated by minimizing the projection error between 3D and 2D. Common methods include EPnP, UPnP, or P3P methods.

[0104] S320: Determine the second time period based on the first moment of acquiring the first keyframe and the second moment of acquiring the second keyframe.

[0105] Figure 4 This is a schematic diagram illustrating a method for determining a first keyframe and a second keyframe according to an embodiment of this application.

[0106] refer to Figure 4It can be seen that after determining that the self-moving device is stationary, the first frame or the nth frame image acquired when stationary can be used as the first keyframe. However, in the process of initializing the feature point depth for a monocular camera, it is not possible to achieve this using only two image frames with a disparity of 0 or close to 0. Therefore, it is necessary to use two image frames with a disparity greater than or equal to the aforementioned second disparity threshold to achieve feature point depth initialization. These two image frames are the aforementioned first and second keyframes. The second time period between these two keyframes is the time period during which the self-moving device undergoes a motion transition. This motion transition refers to the transition process of the self-moving device from a stationary state to a moving state.

[0107] As explained above, determining the second keyframe is not simply a matter of finding a disparity between an image frame and the first keyframe. The disparity between the two images used for feature point depth initialization must be sufficiently large to meet the conditions for feature point depth initialization. Therefore, the second keyframe may not be the first image frame acquired after the mobile device moves; it could be a subsequently acquired image frame.

[0108] Since the second time interval between the first and second keyframes is the period when the self-moving device undergoes a motion transition, this second time interval necessarily includes a period during which the self-moving device remains stationary. Therefore, the IMU also needs to acquire IMU data within this second time interval for initialization of acceleration bias, gyroscope bias, and gravity direction. The specific process of retrieving the IMU data for initialization from the IMU data within the second time interval is as follows:

[0109] S330: Divide the second time period into a first sub-time period and a second sub-time period, wherein the first sub-time period precedes the second sub-time period in chronological order.

[0110] S340: Based on the IMU data acquired by the VIO system during the first sub-period, determine the average acceleration and the average gyroscope value. The average acceleration is a vector.

[0111] It should be understood that during the second time period, the self-moving device undergoes a motion transition. It can be assumed that the self-moving device remained essentially stationary during the first sub-time period, while it must have moved during the second sub-time period. Therefore, the acceleration values ​​acquired by the VIO system during the first sub-time period can be used to determine the initial values ​​of the VIO system.

[0112] In some possible embodiments, the IMU data described above may include acceleration data and gyroscope data.

[0113] S350: Determine the initial values ​​of the VIO system based on the above-mentioned average acceleration value, the above-mentioned average gyroscope value, the above-mentioned first key frame, and the above-mentioned second key frame.

[0114] In some possible embodiments, the aforementioned acceleration mean is used to determine the acceleration zero bias and gravity direction in the initial values ​​of the VIO system, the aforementioned gyroscope mean is used to determine the gyroscope zero bias in the initial values ​​of the VIO system, and the aforementioned second image frame and the aforementioned third image frame are used to determine the feature point depth in the initial values ​​of the VIO system.

[0115] In some possible embodiments, the initial values ​​of zero acceleration bias, zero gyroscope bias, and direction of gravity can be determined in the following ways.

[0116] Regarding the direction of gravity: the relative direction of gravity can be determined based on the acceleration values ​​acquired by the VIO system during the first sub-period. For example, if the acceleration values ​​acquired during the first sub-period are known... Then assume the reference gravity vector Perpendicular to the horizontal plane of the self-moving device; finally, based on the formula: Determine the above and The rotation amount R between them can be used to determine the direction of gravity in the initial value.

[0117] For gyroscope zero bias: the gyroscope mean value corresponding to the first sub-time period above should be equal to 0, so gyroscope zero bias is equal to gyroscope mean value.

[0118] For zero acceleration bias: the mean acceleration corresponding to the first sub-period above should be equal to the gravitational acceleration, so zero acceleration bias is equal to the difference between the mean acceleration corresponding to the first sub-period above and the gravitational acceleration.

[0119] Regarding the feature point depth: the feature point depth can be initialized based on the second and third image frames mentioned above using a triangulation method.

[0120] Based on the above technical solution, detection data during the period when the mobile device undergoes motion transitions can be reasonably obtained. Static initialization of the VIO system based on a monocular camera can be completed based on this detection data. This initialization method is efficient, has a high success rate, and is robust.

[0121] In some possible embodiments, the first keyframe and the second keyframe in the above method 300 can be obtained in the following manner.

[0122] The first image frame and the second image frame are determined sequentially. When the disparity between the first image frame and the second image frame is less than or equal to the first disparity threshold, the first image frame or the second image frame is determined as the first key frame, wherein the first disparity threshold is less than the second disparity threshold.

[0123] The third image frame is determined as the second keyframe when the disparity between the third image frame and the first keyframe is greater than or equal to the second disparity threshold.

[0124] It should be understood that the first, second, and third image frames mentioned above are acquired by an image acquisition device, and the acquisition sequence of these three image frames is not necessarily consecutive. However, the first image frame is acquired first, followed by the second image frame, and the third image frame is acquired last.

[0125] In some possible embodiments, the first image frame described above may be referred to as the initial frame, which is the first image frame obtained after sending the first indication information described above.

[0126] In some possible embodiments, the aforementioned first disparity threshold is a preset threshold, for example, it can be equal to 0. In this case, the disparity between the first image frame and the second image frame should be equal to the first disparity threshold. Considering that the sensor may have detection errors, the aforementioned first disparity threshold can be preset to a value close to 0. In this case, the disparity between the first image frame and the second image frame should be less than the first disparity threshold. Correspondingly, the second disparity threshold can also be a preset threshold, specifically a reasonable threshold preset based on the application scenario and motion performance of the mobile device.

[0127] It should be understood that when the disparity between the first image frame and the second image frame is less than or equal to the first disparity threshold, it can be determined that the self-moving device is in a stationary state during the time period between acquiring the first image frame and acquiring the second image frame.

[0128] In some possible embodiments, the aforementioned second disparity threshold can also be a preset reasonable threshold, such as 30. It should be understood that the greater the disparity between two frames of images obtained from the perspective of the self-mobile device, the faster the self-mobile device moves. Therefore, taking the second image frame as the first keyframe as an example, when the disparity between the second image frame and the third image frame is greater than or equal to the second disparity threshold, it can be determined that the self-mobile device is in motion during the time period between acquiring the second image frame and acquiring the third image frame, so the third image frame is the second keyframe.

[0129] Based on the above technical solution, it is possible to determine whether the self-moving device is in a stationary state, so as to ensure that the static initialization of the VIO system can be performed under the condition that the self-moving device is already stationary.

[0130] In some possible embodiments, the second keyframe and the corresponding second time period can be determined in the following manner:

[0131] S3201: Determine the first image frame as the first keyframe, which is acquired by a monocular camera when the mobile device is determined to be stationary.

[0132] S3202: Determine the first parallax between each subsequently acquired image frame and the first keyframe.

[0133] S3203: When it is determined that the first disparity between the i-th image frame and the first key frame is greater than the above-mentioned second disparity threshold, the i-th image frame is designated as the second key frame.

[0134] In some possible embodiments, the second keyframe may be the third image frame, and the second image frame may be an image frame between the first keyframe and the second keyframe.

[0135] S3204: Determine the time period from the first keyframe to the acquisition of the second keyframe as the first time window, and slide the first sliding window within the first time window.

[0136] S3205: During the period when the first sliding window slides in the first time window, determine the corresponding average acceleration value based on the acceleration value obtained by the VIO system in the corresponding time period of the first sliding window at each moment.

[0137] S3206: At two adjacent moments, if the average acceleration value corresponding to the first sliding window changes from less than or equal to the first threshold to greater than the second threshold, then the time period corresponding to the first sliding window at the next moment is the second time period mentioned above.

[0138] It should be understood that the aforementioned first threshold can be a value close to 0, for example, 0.1 m / s 2 .

[0139] In some possible embodiments, the second keyframe and the corresponding second time period can be determined in the following manner:

[0140] S3211: Determine the first image frame as the first keyframe, which is acquired by a monocular camera when the self-moving device is determined to be stationary.

[0141] S3212: Determine the first parallax between each subsequently acquired image frame and the first keyframe.

[0142] S3213: If, within a preset window period, the first disparity between each acquired image frame and the first keyframe is less than the first disparity threshold, then the last acquired image frame within the preset window period is determined to be the new first keyframe.

[0143] S3214: If, within a preset window period, it is determined that the first disparity between the i-th image frame and the first keyframe is greater than the aforementioned second disparity threshold, then the i-th image frame is determined to be the second keyframe.

[0144] In some possible embodiments, the second keyframe may be the third image frame, and the second image frame may be the first keyframe determined within a certain preset window period.

[0145] S3215: The time period between the time of acquiring the first keyframe and the time of acquiring the second keyframe is defined as the second time period.

[0146] Based on the above technical solution, the second key frame and the second time period can be accurately determined to ensure that subsequent static initialization operations can be performed normally.

[0147] In some possible embodiments, when the image acquisition device in the VIO system is a monocular camera, the above-mentioned S240 can also be implemented in the following manner.

[0148] Figure 5 This is a flowchart illustrating another static initialization method 500 provided in the embodiments of this application.

[0149] S510: Based on the acceleration values ​​acquired by the VIO system in each third time period, determine the average acceleration value in each third time period. This average acceleration value is a vector.

[0150] S520: Sequentially determine the first average acceleration value and the second average acceleration value, wherein the first average acceleration value is less than or equal to the first acceleration threshold, and the second average acceleration value is greater than or equal to the second acceleration threshold, wherein the first acceleration threshold is less than the second acceleration threshold, and the third time period corresponding to the first average acceleration value and the third time period corresponding to the second average acceleration value are adjacent time periods.

[0151] In some possible embodiments, the first acceleration threshold is a preset threshold. Ideally, the first acceleration threshold should be equal to 0, but in practical applications, sensors may have detection errors, so the first acceleration threshold can be a value close to 0, such as 0.1 m / s². 2 0.2m / s 2 Etc. Correspondingly, the aforementioned second acceleration threshold is also a preset threshold, which can be a reasonable threshold determined based on the application scenario and motion performance of the mobile device, for example, 1 m / s². 2 2m / s 2 wait.

[0152] It should be understood that if the detected acceleration changes from 0 to greater than or equal to the second acceleration threshold within two adjacent time periods, it means that the self-moving device may have experienced a motion jump during those two adjacent time periods. However, since the average first acceleration will also be equal to 0 when the self-moving device is moving at a constant speed, the following steps are required to determine whether the self-moving device is stationary during the third time period corresponding to the average first acceleration:

[0153] S530: When the disparity between the fourth and fifth image frames acquired in the third time period corresponding to the first acceleration mean is less than or equal to the first disparity threshold, and the disparity between the sixth and seventh image frames acquired in the third time period corresponding to the second acceleration mean is greater than or equal to the second disparity threshold, the fourth or fifth image frame is determined as the first key frame and the seventh image frame is determined as the second key frame.

[0154] The first disparity threshold is less than the second disparity threshold.

[0155] S540: Determine the first gyroscope mean value based on the IMU data acquired by the VIO system during the third time period corresponding to the first acceleration mean value.

[0156] S550: Determine the initial values ​​of the VIO system based on the above-mentioned first acceleration average value, the above-mentioned first gyroscope average value, the above-mentioned first key frame, and the above-mentioned second key frame.

[0157] In some possible embodiments, if the disparity between the fourth image frame and the fifth image frame is not less than or equal to the first disparity threshold, it indicates that the mobile device is moving at a constant speed during the third time period. In this scenario, the execution steps can be reversed to S510, and the above process can be re-executed.

[0158] It should be understood that the specific process of determining the initial value of the VIO system is similar to the process of determining the initial value of the VIO system in method 300 above, and will not be described in detail here.

[0159] Based on the above technical solution, it is also possible to determine whether the self-moving device is stationary and to reasonably obtain the detection data during the period when the self-moving device undergoes a motion change. The static initialization of the VIO system based on the monocular camera is completed based on the detection data. This initialization method is efficient, has a high success rate, and is robust.

[0160] In some possible embodiments, during the first time period, once the first keyframe has been determined, a second indication message can be sent to the control device of the self-moving device. This second indication message instructs the control device to start the self-moving device ahead of time. This scheme can be applied to the following scenario: after the VIO system completes the relevant data collection based on the stationary self-moving device during static initialization, if the stationary duration of the self-moving device has not yet reached the first time period, the control device can start the self-moving device ahead of time based on the second indication message, allowing the static initialization to proceed to subsequent steps.

[0161] Based on the above technical solution, considering that the first time period is reserved for the VIO system to collect relevant data of the self-moving device in a static state, the first time period may have the problem of excessively long reserved time. When the above collection process is completed in advance, the self-moving device can be started in advance through the second indication information, so that static initialization can enter the subsequent process, thereby increasing the execution efficiency of static initialization.

[0162] In some possible embodiments, the process of acquiring the first keyframe and the second keyframe, and initializing the feature point depth based on the first keyframe and the second keyframe, can also be completed before the mobile device comes to a standstill. During this process, if the disparity between the two image frames is greater than or equal to the second disparity threshold, the two image frames can be determined to be the first keyframe and the second keyframe, respectively.

[0163] In some possible embodiments, when the image acquisition device in the VIO system is a binocular camera, the above-mentioned S240 can also be implemented in the following manner.

[0164] Figure 6 This is a flowchart illustrating another static initialization method 600 provided in the embodiments of this application.

[0165] S610: Based on the acceleration values ​​acquired by the VIO system in each fourth time period, determine the average acceleration value in each fourth time period. This average acceleration value is a vector.

[0166] S620: Sequentially determine the third acceleration mean and the fourth acceleration mean, wherein the third acceleration mean is less than or equal to the first acceleration threshold, and the fourth acceleration mean is greater than or equal to the second acceleration threshold, wherein the first acceleration threshold is less than the second acceleration threshold, and the fourth time period corresponding to the third acceleration mean and the fourth time period corresponding to the fourth acceleration mean are adjacent time periods.

[0167] S630: Determine the second gyroscope mean value based on the IMU data acquired by the VIO system during the fourth time period corresponding to the third acceleration mean value.

[0168] S640: Determine the initial values ​​of the VIO system based on the average value of the third acceleration and the average value of the second gyroscope.

[0169] It should be understood that when the image acquisition device is a stereo camera, due to the special structure of the stereo camera, it is not necessary to acquire the image frame corresponding to the transition process from stationary to moving of the self-moving device to initialize the feature point depth. Therefore, in this embodiment, the feature point depth can be initialized based on the aforementioned first image frame and the aforementioned second image frame.

[0170] Based on the above technical solution, it is possible to determine whether the self-moving device is in a stationary state and to reasonably obtain the corresponding detection data of the self-moving device within a specified time period. The static initialization of the VIO system based on the binocular camera is completed based on the detection data. This initialization method is highly efficient, has a high success rate, and is robust.

[0171] In some possible embodiments, if it is still not determined whether the self-moving device has stopped after the fifth time period following the transmission of the first instruction information, the first instruction information is retransmitted to the control device.

[0172] In some possible embodiments, the aforementioned fifth time period is a preset time period, for example, 10 seconds.

[0173] In some possible embodiments, the reasons why it is still not determined whether the self-moving device has stopped after the fifth time period include: the first indication information fails to be sent, so that the control device does not receive the first indication information after the fifth time period; or, the control device fails to parse the first indication information and does not perform the corresponding control operation within the fifth time period.

[0174] Based on the above technical solution, a retransmission mechanism for the first indication information is introduced, which can effectively avoid the problem of failure to receive the first indication information, thereby ensuring that the initialization of the VIO system can proceed normally. This increases the reliability of static initialization of the VIO system.

[0175] In some possible embodiments, when it is determined that a static initialization operation is required for the VIO system, it is not necessary to immediately instruct the control device to control the self-moving device to stop first and then start. This is because the self-moving device is usually equipped with other auxiliary sensors to maintain the positioning of the self-moving device for a period of time. Therefore, the timing of the control device to control the self-moving device to stop can be combined with the working trajectory of the self-moving device. For example, if the self-moving device is performing an I-shaped working trajectory, it can wait until the self-moving device turns before instructing the control device to control the self-moving device to stop first and then start.

[0176] Based on the above technical solution, the impact of the aforementioned static initialization operation on the operation of the self-moving device can be reduced.

[0177] Furthermore, embodiments of this application also provide an apparatus for implementing any of the above methods. For example, an apparatus for VIO system initialization is provided, which includes a unit (or means) for implementing any of the above methods for VIO system initialization.

[0178] Figure 7 This is a schematic block diagram of a VIO system initialization device 700 provided in an embodiment of this application. This device is applied to self-moving devices equipped with a VIO system, such as... Figure 7 As shown, the device 700 includes:

[0179] The acquisition unit 710 is used to acquire the first state information of the VIO system.

[0180] The sending unit 720 is used to send first indication information to the control device of the self-moving device according to the first status information. The first indication information is used to instruct the control device to control the self-moving device to remain stationary for a first time period and then start.

[0181] Processing unit 730: When it is determined that the self-moving device is stationary, it performs a static initialization operation on the VIO system to determine the initial value of the VIO system.

[0182] The aforementioned first status information includes at least one of the following:

[0183] The VIO system initiates a first signal, which indicates that the VIO system is about to start up, or...

[0184] During the data processing process of the VIO system, a data anomaly occurs, or...

[0185] The preset time has elapsed since the last static initialization operation was completed.

[0186] In some possible embodiments, when the image acquisition device in the above-described VIO system is a monocular camera, the processing unit 730 is specifically used for:

[0187] The first keyframe and the second keyframe are determined sequentially. The first keyframe is acquired by a monocular camera when the mobile device is stationary. The disparity between the first keyframe and the second keyframe is greater than or equal to the second disparity threshold.

[0188] The second time period is determined based on the first moment of acquiring the first keyframe and the second moment of acquiring the second keyframe;

[0189] The second time period is divided into a first sub-time period and a second sub-time period, with the first sub-time period preceding the second sub-time period in chronological order;

[0190] Based on the inertial measurement unit (IMU) data acquired by the VIO system during the first sub-period, the mean acceleration and the mean gyroscope value are determined, with the mean acceleration being a vector value.

[0191] The initial values ​​of the VIO system are determined based on the average acceleration, the average gyroscope readings, the first keyframe, and the second keyframe.

[0192] In some possible embodiments, the processing unit 730 described above is specifically used for:

[0193] The first image frame and the second image frame are determined sequentially. When the disparity between the first image frame and the second image frame is less than or equal to the first disparity threshold, the first image frame or the second image frame is determined as the first key frame. The first disparity threshold is less than the second disparity threshold.

[0194] The third image frame is determined as the second keyframe when the disparity between the third image frame and the first keyframe is greater than or equal to the second disparity threshold.

[0195] In some possible embodiments, when the image acquisition device in the above-described VIO system is a monocular camera, the processing unit 730 is specifically used for:

[0196] Based on the IMU data acquired by the VIO system in each third time period, the average acceleration value in each third time period is determined, and this average acceleration value is a vector.

[0197] The first acceleration mean and the second acceleration mean are determined sequentially, wherein the first acceleration mean is less than or equal to the first acceleration threshold, the second acceleration mean is greater than or equal to the second acceleration threshold, the first acceleration threshold is less than the second acceleration threshold, and the third time period corresponding to the first acceleration mean and the third time period corresponding to the second acceleration mean are adjacent time periods;

[0198] When the disparity between the fourth and fifth image frames acquired in the third time period corresponding to the first acceleration mean is less than or equal to the first disparity threshold, and the disparity between the sixth and seventh image frames acquired in the third time period corresponding to the second acceleration mean is greater than or equal to the second disparity threshold, the fourth or fifth image frame is determined as the first key frame, and the seventh image frame is determined as the second key frame. Based on the IMU data acquired by the VIO system in the third time period corresponding to the first acceleration mean, the first gyroscope mean is determined, wherein the first disparity threshold is less than the second disparity threshold.

[0199] The initial values ​​of the VIO system are determined based on the first average acceleration value, the first average gyroscope value, the first keyframe, and the second keyframe.

[0200] In some possible embodiments, after the sending unit 720 sends the first indication information, the sending unit 720 is further configured to:

[0201] If it is still not determined whether the aforementioned self-moving device has come to a stop after the fifth time period, the aforementioned first instruction information is resent to the aforementioned control device.

[0202] In some possible embodiments, during the first time period, the sending unit 720 is further configured to: send second indication information to the control device of the self-moving device after the processing unit 730 has determined the first key frame, the second indication information being used to instruct the control device to control the self-moving device to start in advance.

[0203] In some possible embodiments, this application proposes a self-moving device, including: a processor, a controller, and a VIO system, wherein the VIO system includes an image acquisition device and an IMU;

[0204] The processor described above is used to acquire first state information of the VIO system; and send first indication information to the controller based on the first state information. The first indication information is used to instruct the controller to control the mobile device to remain stationary for a first time period and then start.

[0205] The aforementioned image acquisition device is used to acquire image frames according to a first frame rate;

[0206] The aforementioned IMU is used to acquire IMU data at a second frame rate, wherein the second frame rate is greater than the first frame rate;

[0207] The processor is also used to determine the motion state of the self-moving device based on the image frames acquired by the image acquisition device, and when it is determined that the self-moving device is stationary, to perform static initialization operation on the VIO system based on the data acquired by the image acquisition device and the IMU, so as to determine the initial value of the VIO system.

[0208] The aforementioned first status information includes at least one of the following:

[0209] The VIO system initiates a first signal, which indicates that the VIO system is about to start up, or...

[0210] During the data processing process of the VIO system, a data anomaly occurs, or...

[0211] The preset time has elapsed since the last static initialization operation was completed.

[0212] In some possible embodiments, this application also proposes a self-moving device that includes the device 700 described in any of the above embodiments.

[0213] Figure 8 This is a schematic block diagram of a computer device 800 provided in an embodiment of this application. Figure 8 The computer device 800 shown includes a memory 810, a processor 820, and a bus 840. Optionally, the computer device 800 also includes a communication interface 830. The memory 810, processor 820, and communication interface 830 are interconnected via the bus 840.

[0214] The memory 810 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 810 can store programs, and when the program stored in the memory 810 is executed by the processor 820, the processor 820 performs the various steps of the VIO system initialization method of this embodiment. For example, the processor 820 can execute the steps described above. Figures 2 to 6 The method shown.

[0215] The processor 820 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the VIO system initialization method proposed in the method embodiments of this application.

[0216] The processor 820 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the VIO system initialization method of this application can be completed by the hardware integrated logic circuit in the processor 820 or by software instructions.

[0217] The processor 820 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods involved in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 810, and the processor 820 reads the information in memory 810 and combines it with its hardware to complete... Figure 7 The apparatus shown includes units that are required to perform functions, or to perform the methods described in this application. Figures 2 to 6 The method shown.

[0218] The communication interface 830 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 800 and other devices or communication networks.

[0219] Bus 840 may include a pathway for transmitting information between various components of device 800 (e.g., memory 10, processor 820, communication interface 830).

[0220] It should be understood that although the above-described device 800 only shows a memory, processor, and communication interface, those skilled in the art should understand that in specific implementations, device 800 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that device 800 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that device 800 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 8 All the devices shown.

[0221] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0222] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0223] Figure 9 This is a schematic block diagram of a computer-readable storage medium 900 provided for an embodiment of this application. Figure 9The computer-readable storage medium 900 shown stores computer instructions 910. When executed by a processor, the computer instructions 910 can implement the methods corresponding to the above embodiments.

[0224] In some possible embodiments, the computer-readable storage medium 900 can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more sets of available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. Semiconductor media can be solid-state drives (SSDs).

[0225] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0226] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0227] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0228] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0230] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0231] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for initializing a visual inertial odometry (VIO) system, characterized in that, Applied to a self-moving device equipped with a VIO system, wherein the image acquisition device in the VIO system is a monocular camera, the method includes: Obtain the first state information of the VIO system; The first indication information is sent to the control device of the self-moving device according to the first status information. The first indication information is used to instruct the control device to keep the self-moving device stationary for a first time period and then start it up. The first status information includes at least one of the following: the VIO system initiates a first signal, which is used to indicate that the VIO system is about to start up, or, a data anomaly occurs during the VIO system's data processing, or, a preset time has elapsed since the last time the static initialization operation was completed. When it is determined that the self-moving device is stationary, a static initialization operation is performed on the VIO system to determine the initial values ​​of the VIO system, including: The first keyframe and the second keyframe are determined sequentially, wherein the first keyframe is acquired by the monocular camera when the self-moving device is determined to be stationary, and the disparity between the first keyframe and the second keyframe is greater than or equal to the second disparity threshold. The second time period is determined based on the first moment when the first keyframe is acquired and the second moment when the second keyframe is acquired; The second time period is divided into a first sub-time period and a second sub-time period, with the first sub-time period preceding the second sub-time period in chronological order; Based on the inertial measurement unit (IMU) data acquired by the VIO system during the first sub-period, the mean acceleration and the mean gyroscope value are determined, wherein the mean acceleration is a vector. The initial values ​​of the VIO system are determined based on the average acceleration, the average gyroscope reading, the first keyframe, and the second keyframe. or, Based on the IMU data acquired by the VIO system in each third time period, the average acceleration value in each third time period is determined, and the average acceleration value is a vector; a first average acceleration value and a second average acceleration value are determined sequentially, wherein the first average acceleration value is less than or equal to a first acceleration threshold, the second average acceleration value is greater than or equal to a second acceleration threshold, the first acceleration threshold is less than the second acceleration threshold, and the third time period corresponding to the first average acceleration value and the third time period corresponding to the second average acceleration value are adjacent time periods; When the disparity between the fourth and fifth image frames acquired in the third time period corresponding to the first acceleration mean is less than or equal to the first disparity threshold, and the disparity between the sixth and seventh image frames acquired in the third time period corresponding to the second acceleration mean is greater than or equal to the second disparity threshold, the fourth image frame or the fifth image frame is determined as the first key frame, and the seventh image frame is determined as the second key frame. Based on the IMU data acquired by the VIO system in the third time period corresponding to the first acceleration mean, the first gyroscope mean is determined, wherein the first disparity threshold is less than the second disparity threshold. The initial values ​​of the VIO system are determined based on the first average acceleration value, the first average gyroscope value, the first keyframe, and the second keyframe.

2. The method according to claim 1, characterized in that, The step of sequentially determining the first keyframe and the second keyframe includes: The first image frame and the second image frame are determined sequentially. When the disparity between the first image frame and the second image frame is less than or equal to the first disparity threshold, the first image frame or the second image frame is determined as the first key frame, and the first disparity threshold is less than the second disparity threshold. A third image frame is determined. When the disparity between the third image frame and the first key frame is greater than or equal to the second disparity threshold, the third image frame is determined to be the second key frame.

3. The method according to claim 1 or 2, characterized in that, After sending the first indication information, the method further includes: If it is still not determined whether the self-moving device has stopped after the fifth time period, the first indication information is resent to the control device.

4. The method according to any one of claims 1 to 3, characterized in that, During the first time period, the method further includes: Once the first keyframe has been determined, a second indication message is sent to the control device of the self-moving device. The second indication message is used to instruct the control device to control the self-moving device to start in advance.

5. A device for initializing a VIO system, characterized in that, The method includes a processor and a memory, wherein the processor and the memory are connected together, wherein the memory is used to store program code, and the processor is used to call the program code to perform the method as described in any one of claims 1 to 4.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1 to 4.

7. A computer program product, characterized in that, When the computer program code or instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1 to 4.

8. A self-moving device, characterized in that, include: The processor, controller, and VIO system, wherein the VIO system includes an image acquisition device and an IMU, and the image acquisition device is a monocular camera; The processor is configured to: acquire first state information of the VIO system; and send first indication information to the controller based on the first state information, wherein the first indication information is configured to instruct the controller to control the self-moving device to remain stationary for a first time period and then start; wherein the first state information includes at least one of the following: the VIO system initiates a first signal, wherein the first signal is configured to indicate that the VIO system is about to start initially, or, a data anomaly occurs during the VIO system's data processing, or, a preset time period has elapsed since the last completion of the static initialization operation; The image acquisition device is used to acquire image frames according to a first frame rate; The IMU is used to acquire IMU data at a second frame rate, where the second frame rate is greater than the first frame rate. The processor is further configured to determine the motion state of the self-moving device based on image frames acquired by the image acquisition device, and, when it is determined that the self-moving device is stationary, perform a static initialization operation on the VIO system based on data acquired by the image acquisition device and the IMU to determine the initial values ​​of the VIO system, including: The first keyframe and the second keyframe are determined sequentially, wherein the first keyframe is acquired by the monocular camera when the self-moving device is determined to be stationary, and the disparity between the first keyframe and the second keyframe is greater than or equal to the second disparity threshold. The second time period is determined based on the first moment when the first keyframe is acquired and the second moment when the second keyframe is acquired; The second time period is divided into a first sub-time period and a second sub-time period, with the first sub-time period preceding the second sub-time period in chronological order; Based on the inertial measurement unit (IMU) data acquired by the VIO system during the first sub-period, the mean acceleration and the mean gyroscope value are determined, wherein the mean acceleration is a vector. The initial values ​​of the VIO system are determined based on the average acceleration, the average gyroscope reading, the first keyframe, and the second keyframe. or, Based on the IMU data acquired by the VIO system in each third time period, the average acceleration value in each third time period is determined, and the average acceleration value is a vector; a first average acceleration value and a second average acceleration value are determined sequentially, wherein the first average acceleration value is less than or equal to a first acceleration threshold, the second average acceleration value is greater than or equal to a second acceleration threshold, the first acceleration threshold is less than the second acceleration threshold, and the third time period corresponding to the first average acceleration value and the third time period corresponding to the second average acceleration value are adjacent time periods; When the disparity between the fourth and fifth image frames acquired in the third time period corresponding to the first acceleration mean is less than or equal to the first disparity threshold, and the disparity between the sixth and seventh image frames acquired in the third time period corresponding to the second acceleration mean is greater than or equal to the second disparity threshold, the fourth image frame or the fifth image frame is determined as the first key frame, and the seventh image frame is determined as the second key frame. Based on the IMU data acquired by the VIO system in the third time period corresponding to the first acceleration mean, the first gyroscope mean is determined, wherein the first disparity threshold is less than the second disparity threshold. The initial values ​​of the VIO system are determined based on the first average acceleration value, the first average gyroscope value, the first keyframe, and the second keyframe.

Images