Yts-based visual-inertial odometry model training method and device

Abstract

The application provides a YTS-based visual inertial odometer model training method and device, relates to the technical field of model training, and improves the calculation precision of the visual inertial odometer model. The method comprises the following steps: identifying the contact area between a tire and the ground based on a tire image collected by an image collection device through YTS; detecting the corresponding multiple different linear acceleration and angular velocity data and multiple different surrounding environment change images when the intelligent mobile tool moves under multiple different weight distribution data and multiple different non-fixed object weight conditions; determining the linear acceleration and angular velocity data, the surrounding environment change images and the actual total weight corresponding to the fixed self weight, the multiple different weight distribution data, the multiple different non-fixed object weight, each weight distribution data and non-fixed object weight condition as a training sample set, and training the initial visual inertial odometer model by using the training sample set.

Description

Technical Field

[0001] This application relates to the field of model training technology, and in particular to a method and apparatus for training a visual inertial odometry model based on YTS. Background Technology

[0002] Currently, Visual-Inertial Odometry (VIO) is a sensor fusion technology that combines visual and inertial measurement unit (IMU) data to achieve accurate motion estimation. By simultaneously utilizing data from cameras and IMUs, VIO can effectively address the challenges of motion estimation using a single sensor in high-speed, complex environments, and is widely used in fields such as robot navigation, autonomous driving, and augmented reality. However, the data calculation accuracy of VIO models trained in existing technologies is relatively low. Summary of the Invention

[0003] The purpose of this invention is to provide a method and apparatus for training a visual inertial odometry model based on YTS, so as to solve the technical problem of low data calculation accuracy of visual inertial odometry models trained in the prior art.

[0004] In a first aspect, the present invention provides a visual inertial odometry model training method based on YTS, wherein each seat in a smart mobile tool is equipped with a weight detection device for detecting the weight of an object sitting on the seat, and each tire in the smart mobile tool is correspondingly equipped with an image acquisition device, the image acquisition direction of the image acquisition device being towards the tire; the method includes: The fixed weight of the intelligent mobile tool is obtained, and the actual total weight is obtained based on the fixed weight and the weight of the non-fixed object collected by the weight detection device. Based on the tire images acquired by the image acquisition device, the contact area between the tire and the ground is identified by YTS, and the weight distribution data in the intelligent mobile tool is analyzed based on the multiple contact areas corresponding to multiple tires and the weight of the non-fixed object; the intelligent mobile tool is also equipped with a vision sensor, accelerometer and gyroscope; The system detects various linear acceleration and angular velocity data, as well as various ambient environment change images, when the intelligent mobile tool moves under various weight distribution conditions and various non-fixed object weights. The ambient environment change images are acquired by the vision sensor using the YTS, and the linear acceleration and angular velocity data are acquired by the accelerometer and gyroscope. The fixed self-weight, various weight distribution data, various non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution and non-fixed object weight, the surrounding environment change image, and the actual total weight are determined as the training sample set. The initial visual inertial odometry model is trained using the training sample set to obtain the final visual inertial odometry model.

[0005] In an optional implementation, the final visual inertial odometry model is used to represent the inertial data and motion estimation data of the intelligent mobile tool under various weight and weight distribution conditions.

[0006] In an optional implementation, the intelligent mobile tool is equipped with an anti-lock braking system (ABS). After training the initial visual inertial odometry model using the training sample set to obtain the final visual inertial odometry model, the method further includes: In response to the intelligent mobile tool triggering the ABS during movement, the final visual inertial odometry model is used to analyze the inertial data and motion estimation data corresponding to the weight and weight distribution of the intelligent mobile tool when the ABS is triggered; Based on the inertial data and motion estimation data corresponding to each tire on the intelligent mobile tool, a target tire whose inertial data and / or motion estimation data exceed a specified data range is identified from among the multiple tires, and the braking force of the ABS for the brake corresponding to the target tire is increased.

[0007] In an optional implementation, the step of determining the fixed self-weight, various different weight distribution data, various different non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution data and the non-fixed object weight, the surrounding environment change image, and the actual total weight as a training sample set, and using the training sample set to train the initial visual inertial odometry model to obtain the final visual inertial odometry model, includes: The system acquires the user's driving speed and direction control commands for the intelligent mobile tool when each of the linear acceleration and angular velocity data is detected, as well as the corresponding weight distribution data and the weight of the non-fixed object. The fixed self-weight, various different weight distribution data, various different non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution data and non-fixed object weight, the corresponding surrounding environment change image, the actual total weight corresponding to each non-fixed object weight, and the driving speed and direction control commands corresponding to each linear acceleration and angular velocity data detected, along with the corresponding weight distribution data and corresponding non-fixed object weight, are determined as a comprehensive training sample set. The initial visual inertial odometry model is trained using the comprehensive training sample set to obtain the final visual inertial odometry model.

[0008] In an optional implementation, a tire pressure detector is provided at each of the tires in the smart mobility tool; The process involves defining a comprehensive training sample set, which includes the fixed self-weight, various weight distribution data, various non-fixed object weights, linear acceleration and angular velocity data corresponding to each weight distribution data and non-fixed object weight, the corresponding surrounding environment change image, the actual total weight corresponding to each non-fixed object weight, and the driving speed and direction control commands corresponding to each linear acceleration and angular velocity data detected, along with the corresponding weight distribution data and non-fixed object weight. This comprehensive training sample set is then used to train the initial visual inertial odometry model to obtain the final visual inertial odometry model, including: During the movement of the intelligent mobile tool, the tire pressure detector detects the tire pressure change data corresponding to each driving speed and direction control command executed during the movement of the intelligent mobile tool. The image acquisition device acquires a second tire image corresponding to each driving speed and direction control command executed during the movement of the intelligent mobile tool. Based on the second tire image, the YTS identifies the change data of the tire contact area with the ground corresponding to each driving speed and direction control command executed. The following data are used to determine the final comprehensive training sample set: the fixed self-weight, various different weight distribution data, various different non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution data and non-fixed object weight, the corresponding surrounding environment change image, the actual total weight corresponding to each non-fixed object weight, the driving speed and direction control command corresponding to each linear acceleration and angular velocity data detected, the corresponding weight distribution data and corresponding non-fixed object weight, the tire pressure change data corresponding to each driving speed and direction control command, and the corresponding contact area change data. The initial visual inertial odometry model is then trained using the final comprehensive training sample set to obtain the final visual inertial odometry model.

[0009] In an optional implementation, after training the initial visual-inertial odometry model using the training sample set to obtain the final visual-inertial odometry model, the method further includes: During the movement of the intelligent mobile tool, the current driving speed and direction control commands, current weight distribution data, current weight of non-fixed objects, and current fixed weight of the intelligent mobile tool are acquired. Based on the current driving speed and direction control commands, the current weight distribution data, the current weight of the non-fixed object, and the current fixed self-weight, the motion estimation data of the intelligent mobile tool is predicted using the final visual inertial odometry model. The motion estimation data includes linear acceleration prediction results and angular velocity data and prediction results. Based on the linear acceleration prediction result and the angular velocity data and prediction result, the driving trajectory of the intelligent mobile tool is predicted to obtain the driving trajectory prediction result of the intelligent mobile tool under the current driving speed and direction control command.

[0010] In an optional implementation, the method further includes: Based on the fixed weight of the intelligent mobility tool, the weight of the non-fixed object corresponding to each seat in the intelligent mobility tool, and the weight distribution data in the intelligent mobility tool, the linear acceleration and angular acceleration data of the intelligent mobility tool are determined by the following formula: - ; in, This represents the linear acceleration data of the intelligent mobile tool; This refers to the fixed weight of the intelligent mobile tool. Indicates the weight of the non-fixed object in the intelligent mobile tool; indicates; This represents the i-th external force acting on the intelligent mobile tool; Represents gravitational acceleration; This indicates the amount of external force acting on the intelligent mobile tool; This represents the mass of the non-fixed object at the j-th seat on the intelligent mobile tool; This indicates the number of seats in the smart mobility device; ; in, This represents the angular acceleration data of the intelligent mobile tool; This represents the inertial tensor of the intelligent mobile tool about its center of mass; Let represent the vector from the i-th seat on the intelligent mobile tool to the centroid; This represents the i-th external force acting on the intelligent mobile tool; This indicates the amount of external force acting on the intelligent mobile tool; This represents the current angular velocity of the intelligent mobility tool; Let represent the vector from the j-th seat on the intelligent mobile tool to the centroid; This indicates the number of seats in the smart mobility device; This represents the mass of the non-fixed object at the j-th seat on the intelligent mobile tool; It represents the acceleration due to gravity.

[0011] Secondly, the present invention provides a visual inertial odometry model training device based on YTS. Each seat in the intelligent mobility tool is equipped with a weight detection device for detecting the weight of an object on the seat. Each tire in the intelligent mobility tool is correspondingly equipped with an image acquisition device, the image acquisition direction of which is towards the tire. The device includes: The acquisition module is used to acquire the fixed weight of the intelligent mobile tool and to obtain the actual total weight based on the fixed weight and the weight of the non-fixed object collected by the weight detection device. The recognition module is used to identify the contact area between the tire and the ground based on the tire image acquired by the image acquisition device through YTS, and to analyze the weight distribution data in the intelligent mobile tool based on the multiple contact areas corresponding to multiple tires and the weight of the non-fixed object; the intelligent mobile tool is also equipped with a vision sensor, an accelerometer and a gyroscope; The detection module is used to detect various linear acceleration and angular velocity data and various surrounding environment change images corresponding to the movement of the intelligent mobile tool under various different weight distribution data and various different non-fixed object weights; the surrounding environment change images are acquired by the vision sensor using the YTS, and the linear acceleration and angular velocity data are acquired by the accelerometer and the gyroscope; The training module is used to determine the fixed self-weight, various different weight distribution data, various different non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution data and the non-fixed object weight, the surrounding environment change image, and the actual total weight as a training sample set, and to use the training sample set to train the initial visual inertial odometry model to obtain the final visual inertial odometry model.

[0012] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the steps of the method described in any of the foregoing embodiments.

[0013] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to perform the method described in any of the foregoing embodiments.

[0014] This application brings the following beneficial effects: This application provides a method and apparatus for training a visual inertial odometry model based on YTS. Each seat in a smart mobile tool is equipped with a weight detection device for detecting the weight of an object on that seat. Each tire in the smart mobile tool is equipped with an image acquisition device, the image acquisition direction of which is towards the tire. The method can obtain the fixed weight of the smart mobile tool and, based on the fixed weight and the weight of non-fixed objects acquired by the weight detection device, obtain the actual total weight. Based on the tire images acquired by the image acquisition device, YTS is used to identify the contact area between the tire and the ground, and the weight distribution data in the smart mobile tool is analyzed based on the multiple contact areas corresponding to multiple tires and the weight of the non-fixed objects. The smart mobile tool is also equipped with a visual sensor and an accelerometer. The system includes an accelerometer and a gyroscope; it detects various linear acceleration and angular velocity data and various ambient environment change images corresponding to the movement of the intelligent mobile tool under various weight distribution conditions and various non-fixed object weights; the ambient environment change images are acquired by the visual sensor using the YTS, and the linear acceleration and angular velocity data are acquired by the accelerometer and the gyroscope; the system determines the fixed self-weight, various weight distribution conditions, various non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution condition and non-fixed object weight, the ambient environment change images, and the actual total weight as a training sample set, and uses the training sample set to train the initial visual inertial odometry model to obtain the final visual inertial odometry model;In this solution, by installing weight detection and image acquisition devices on the intelligent mobility tool, the system can monitor the vehicle's actual total weight and tire-to-ground contact area in real time. Using visual sensors, accelerometers, and gyroscopes, the system can capture key data such as changes in the surrounding environment, linear acceleration, and angular velocity during movement. This data provides information about the mobility tool's performance under different load conditions. Specifically, by collecting images of environmental changes, linear acceleration, and angular velocity data using visual sensors, accelerometers, and gyroscopes, various dynamic response data of the mobility tool under different weight distributions can be obtained. Then, this rich and diverse data is integrated into a training sample set and used to train the initial visual-inertial odometry model. In other words, the aforementioned data are input as the training sample set into the initial visual-inertial odometry model. The model is trained within a visual inertial odometry (VIO) model, enabling it to learn accurate behavioral patterns under various conditions (such as different weight distributions and the weight of non-fixed objects). Through continuous optimization and adjustment of model parameters, it adapts to a wider range of operating conditions. As the model continuously learns and optimizes by accepting different types of data, it gradually learns how to more accurately predict the behavior of moving tools under various operating conditions. Ultimately, a high-precision VIOMA model is obtained, capable of providing accurate position and orientation estimates in various complex environments. Therefore, the final VIOMA model has higher data calculation accuracy, improving the data calculation accuracy of the trained VIOMA model and solving the technical problem of low data calculation accuracy in existing VIOMA models.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A schematic flowchart illustrating the YTS-based visual inertial odometry model training method provided in this application embodiment; Figure 2 Another flowchart illustrating the YTS-based visual inertial odometry model training method provided in this application embodiment; Figure 3 A schematic diagram of the structure of a YTS-based visual inertial odometry model training device provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this application, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0020] Currently, the data calculation accuracy of visual inertial odometry models trained in existing technologies is relatively low. Therefore, this application provides a method and apparatus for training a visual inertial odometry model based on YTS, which can solve the technical problem of low data calculation accuracy in existing visual inertial odometry models.

[0021] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating a YTS-based visual inertial odometry model training method provided in an embodiment of this application. Each seat in the intelligent mobility tool is equipped with a weight detection device for detecting the weight of an object seated thereon. Each tire in the intelligent mobility tool is equipped with a corresponding image acquisition device, with the image acquisition direction of the image acquisition device facing the tire. Figure 1 As shown, the method may include the following steps: S110: Obtain the fixed weight of the intelligent mobile tool and obtain the actual total weight based on the fixed weight and the weight of the non-fixed object collected by the weight detection device.

[0023] In one possible implementation, the system first needs to acquire the fixed weight of the intelligent mobility vehicle. This data is typically pre-set and stored in the system's database or configuration file. Weight detection devices installed at each seat begin operating, monitoring and recording the weight of objects (such as passengers or cargo) located on the seats in real time. This weight data is transmitted to a central processing unit for further analysis. Image acquisition devices corresponding to each tire are activated, their image acquisition direction facing the part of the tire in contact with the ground. The image acquisition devices periodically take images of the tire-ground contact area and send these images to the processing unit for subsequent analysis. After receiving the data from the weight detection devices, the central processing unit, combined with the known fixed weight, calculates the actual total weight of the intelligent mobility vehicle. This process may involve summarizing weight data from multiple seats and taking into account any possible offsets or error corrections. Using YTS or other algorithms, the tire images received from the image acquisition devices are analyzed to identify the actual contact area between the tires and the ground. By combining the contact area information of different tires and the weight of non-fixed objects, the weight distribution within the intelligent mobility vehicle can be further analyzed.

[0024] The entire process begins with an initialization phase, using the fixed weight of the intelligent mobile tool as a baseline. Next, a data collection phase occurs, employing weight detection and image acquisition devices to collect images of the weight of non-fixed objects and the tire-to-ground contact area. Finally, in the data processing phase, the central processing unit calculates the actual total weight based on the collected information and can further analyze other parameters such as weight distribution. This process ensures that the intelligent mobile tool accurately understands its own load status, providing the necessary data support for its subsequent operations.

[0025] S120 uses the tire image acquired by the image acquisition device to identify the contact area between the tire and the ground through YTS, and analyzes the weight distribution data of the intelligent mobile tool based on the multiple contact areas corresponding to multiple tires and the weight of non-fixed objects.

[0026] Smart mobile tools are also equipped with vision sensors, accelerometers, and gyroscopes.

[0027] In practical applications, YTS (Unity TV Service) refers to the Unity visualization rendering service system. Unity is a real-time 3D interactive content creation and operation platform, enabling creators in fields such as game development, art, architecture, automotive design, and film to turn their ideas into reality. The platform provides a complete software solution for creating, operating, and monetizing any real-time interactive 2D and 3D content, supporting platforms including mobile phones, tablets, PCs, game consoles, augmented reality, and virtual reality devices. The YTS engine is an intelligent engine system that deeply integrates AI algorithms, physical simulation, and 3D digital rendering technology, specifically designed for next-generation intelligent vehicles. Its core objective is to drive comprehensive upgrades in areas such as autonomous driving, vehicle-road collaboration, and intelligent interaction through high-precision simulation, real-time decision optimization, and cross-domain collaboration capabilities, building an integrated intelligent transportation ecosystem encompassing "people-vehicle-road-cloud."

[0028] For example, an image acquisition device (camera) installed on a smart mobility tool (such as a car) is used to capture real-time images of the contact area between each tire and the ground. The images are ensured to be clear and accurately reflect the tire-ground contact. Weight information of non-fixed objects (such as passengers or cargo) is collected via sensors under the seat or other types of weight detection devices. This data is used for subsequent weight distribution analysis. Necessary preprocessing operations, such as cropping, scaling, and noise removal, are performed on the received raw tire images to improve the accuracy of subsequent recognition algorithms. The preprocessed tire images are analyzed using YTS (assuming this refers to a specific image segmentation and recognition technique) or other applicable techniques to accurately identify the actual contact area between the tire and the ground. Based on the contact area data obtained from each tire and the weight information of the non-fixed objects, a specially designed algorithm is used to calculate the weight distribution throughout the smart mobility tool. This step may involve complex mechanical calculations and data analysis to determine the load status of each part. Based on the analyzed weight distribution data, reports can be generated or immediate feedback can be provided to the user or control system. This feedback can be used to adjust vehicle load, optimize driving stability, or provide the driver with suggestions on how to distribute weight more evenly. In some applications, the system may automatically make adjustments based on analysis results, such as dynamically adjusting suspension stiffness to adapt to different load conditions, thereby improving driving safety and comfort. This system execution flow demonstrates the use of modern image processing technology and weight detection methods, combined with advanced data analysis methods, to achieve precise monitoring and optimized management of weight distribution within intelligent mobility tools. This solution not only helps improve vehicle operational safety but also enhances the passenger experience.

[0029] S130 detects various linear acceleration and angular velocity data, as well as various images of changes in the surrounding environment, when the intelligent mobile tool moves under various weight distribution conditions and various non-fixed object weights.

[0030] The images of changes in the surrounding environment are acquired using YTS via a visual sensor, while linear acceleration and angular velocity data are acquired using an accelerometer and a gyroscope.

[0031] The intelligent mobility tool captures real-time images of its surrounding environment using visual sensors (such as cameras). These images include not only tire contact with the ground but also the dynamic environment around the vehicle during movement. Accelerometers and gyroscopes are used to collect linear acceleration and angular velocity data of the intelligent mobility tool in motion. This step provides detailed information about the vehicle's acceleration, deceleration, and turning behavior. Similarly, under-seat or other types of weight sensors are used to detect the weight of non-fixed objects, providing necessary input for subsequent weight distribution analysis. The images of the surrounding environment acquired from the visual sensors are preprocessed (e.g., noise reduction, contrast enhancement) and YTS technology is used to extract useful information, such as the location, type, and relative position changes of obstacles. Based on the raw data from the accelerometers and gyroscopes, the precise linear acceleration and angular velocity of the intelligent mobility tool over different time periods are calculated. Furthermore, other important motion parameters such as vehicle speed and displacement can be deduced from this data. Combining the tire contact area identification results, non-fixed object weight data, and current motion parameters, a comprehensive analysis is performed to evaluate the stability and safety of the intelligent mobility tool under various weight distribution conditions. Based on the above analysis, the system can provide the driver or automatic control system with a series of optimization suggestions or take direct measures. For example, when an unstable state is detected, the system may suggest reassigning cargo or passenger positions; or in an emergency, it may automatically adjust the suspension stiffness to improve vehicle stability. Utilizing environmental information obtained from visual sensors helps the vehicle better understand and adapt to different driving conditions. For example, when obstacles or complex road conditions are detected ahead, the system can provide early warnings or plan a safer route. The system should possess self-learning and continuous optimization capabilities, constantly monitoring various vehicle performance indicators and adjusting algorithms and strategies based on actual operating performance to further improve its performance and reliability.

[0032] This system's execution flow demonstrates how to integrate multiple sensing technologies and combine them with advanced data analysis methods to achieve comprehensive, multi-layered status monitoring and management of intelligent mobile tools. This approach not only significantly improves vehicle safety and ease of operation but also lays a solid foundation for the development of its intelligent capabilities.

[0033] S140, the training sample set is determined by the fixed self-weight, data of various different weight distributions, weights of various different non-fixed objects, linear acceleration and angular velocity data corresponding to each weight distribution and non-fixed object weight, images of changes in the surrounding environment, and the actual total weight. The initial visual inertial odometry model is trained using the training sample set to obtain the final visual inertial odometry model.

[0034] In one possible implementation, the final visual inertial odometry model is used to represent the inertial data and motion estimation data of the intelligent mobile tool under various weight and weight distribution conditions. As one possible implementation, step S140 may specifically include the following steps: Acquire the user's driving speed and direction control commands for the intelligent mobile tool when each linear acceleration and angular velocity data is detected, as well as the corresponding weight distribution data and the weight of non-fixed objects; The comprehensive training sample set is determined by including data on fixed self-weight, various weight distributions, various non-fixed object weights, linear acceleration and angular velocity data corresponding to each weight distribution and non-fixed object weight, corresponding images of surrounding environmental changes, the actual total weight corresponding to each non-fixed object weight, and the driving speed and direction control commands corresponding to each linear acceleration and angular velocity data detected, along with the corresponding weight distribution data and corresponding non-fixed object weights. The initial visual inertial odometry model is then trained using this comprehensive training sample set to obtain the final visual inertial odometry model.

[0035] As an optional implementation, each tire in the intelligent mobility tool is equipped with a tire pressure detector. The aforementioned comprehensive training sample set is determined by including data on the fixed self-weight, various weight distributions, various non-fixed object weights, linear acceleration and angular velocity data corresponding to each weight distribution and non-fixed object weight, corresponding images of surrounding environmental changes, the actual total weight corresponding to each non-fixed object weight, and the driving speed and direction control commands corresponding to each linear acceleration and angular velocity data detected. The initial visual inertial odometry model is then trained using this comprehensive training sample set to obtain the final visual inertial odometry model. Specifically, this may include the following steps: During the movement of the intelligent mobile tool, the tire pressure detector detects the tire pressure change data corresponding to each driving speed and direction control command executed during the movement of the intelligent mobile tool. The image acquisition device acquires the second tire image corresponding to each driving speed and direction control command executed during the movement of the intelligent mobile tool. Based on the second tire image, the YTS identifies the change data of the tire contact area with the ground corresponding to each driving speed and direction control command executed. The final comprehensive training sample set is determined by including data on fixed self-weight, various weight distributions, various non-fixed object weights, linear acceleration and angular velocity data corresponding to each weight distribution and non-fixed object weight, corresponding images of surrounding environmental changes, the actual total weight corresponding to each non-fixed object weight, the driving speed and direction control commands corresponding to each linear acceleration and angular velocity data detected, the corresponding weight distribution data and corresponding non-fixed object weights, tire pressure changes corresponding to each driving speed and direction control command, and corresponding contact area changes. The initial visual inertial odometry model is then trained using the final comprehensive training sample set to obtain the final visual inertial odometry model.

[0036] In this embodiment, by installing a weight detection device and an image acquisition device on the intelligent mobility tool, the system can monitor the vehicle's actual total weight and its tire-to-ground contact area in real time. Using visual sensors, accelerometers, and gyroscopes, the system can capture key data such as changes in the surrounding environment, linear acceleration, and angular velocity during movement. This data provides information about the mobility tool's performance under different load conditions. Specifically, by collecting images of environmental changes, linear acceleration, and angular velocity data using visual sensors, accelerometers, and gyroscopes, various dynamic response data of the mobility tool under different weight distributions can be obtained. Then, this rich and diverse data is integrated into a training sample set and used to train an initial visual-inertial odometry model. Various data sets are used as training samples and input into the initial visual inertial odometry model for training. This enables the model to learn accurate behavior patterns under different conditions (such as different weight distributions, non-fixed object weights, etc.). By continuously optimizing and adjusting the model parameters, it adapts to a wider range of operating conditions. As the model continuously receives different types of data for learning and optimization, it gradually learns how to more accurately predict the behavior of mobile tools under various operating conditions. Ultimately, a high-precision visual inertial odometry model is obtained, which can provide accurate position and orientation estimates in a variety of complex environments. Therefore, the final visual inertial odometry model has higher data calculation accuracy, which improves the data calculation accuracy of the trained visual inertial odometry model.

[0037] In some embodiments, the intelligent mobile tool is equipped with an anti-lock braking system (ABS); after training the initial visual inertial odometry model using the training sample set to obtain the final visual inertial odometry model, the method may further include the following steps: In response to the intelligent mobile tool triggering ABS during movement, the final visual inertial odometry model is used to analyze the inertial data and motion estimation data corresponding to the weight and weight distribution of the intelligent mobile tool when ABS is triggered. Based on the inertial data and motion estimation data corresponding to each tire on the intelligent mobility tool, the target tire whose inertial data and / or motion estimation data exceed the specified data range is identified from multiple tires, and the ABS is adjusted to increase the braking force of the brake corresponding to the target tire.

[0038] By utilizing data on varying weights and their distributions within the training sample set, the VIO model can adapt to and accurately estimate motion states under diverse load conditions. This means that even when the load changes, the model maintains high positioning accuracy, enhancing the system's robustness. Considering the impact of weight and weight distribution on acceleration and angular velocity, the final model can provide more precise linear acceleration and angular velocity data, thereby achieving more accurate position and attitude estimation. This is particularly important for applications requiring high-precision navigation, such as autonomous vehicles and drones. Joint analysis of images showing changes in the surrounding environment with IMU data helps improve the system's understanding of the environment and its ability to cope with complex environments, such as maintaining good navigation performance in scenarios with drastic changes in lighting or unavailable GPS signals. The optimized model can meet accuracy requirements while minimizing computational resource consumption, thus supporting real-time application needs. This is crucial for applications such as mobile robots that require rapid responses to external changes.

[0039] In conclusion, the final visual inertial odometry model can significantly improve the ability of intelligent mobile tools to perform self-localization and motion estimation in complex and changing working environments, providing a solid technical guarantee for their stable operation.

[0040] In some embodiments, after training the initial visual inertial odometry model using the training sample set to obtain the final visual inertial odometry model, as follows: Figure 2 As shown, the method may further include the following steps: S210, during the movement of the intelligent mobile tool, acquires the current driving speed and direction control commands of the intelligent mobile tool, the current weight distribution data, the current weight of non-fixed objects, and the current fixed weight of the intelligent mobile tool itself; S220 uses the final visual inertial odometry model to predict motion estimation data of intelligent mobile tools based on the current driving speed and direction control commands, current weight distribution data, current weight of non-fixed objects, and current fixed self-weight. S230 predicts the driving trajectory of the intelligent mobile tool based on the linear acceleration prediction results and angular velocity data and prediction results, and obtains the driving trajectory prediction results of the intelligent mobile tool under the current driving speed and direction control command.

[0041] The motion estimation data includes linear acceleration and angular velocity predictions. By collecting and utilizing comprehensive information such as various weight distributions, the weight of non-fixed objects and their corresponding linear acceleration and angular velocity data, as well as driving speed and direction control commands, as a training sample set, the final VIO model can maintain high-accuracy position estimation under various physical conditions (including but not limited to different load states and dynamic load changes). Training by combining actual total weight, images of changes in the surrounding environment, and IMU data allows the model to more accurately understand and predict the behavior patterns of intelligent mobility tools, thereby achieving more refined position and attitude estimation, especially in complex or rapidly changing environments. Because the training data covers a wide range of operating conditions and load states, this enhances the model's stability in the face of various uncertainties and changes that may occur in the real world, such as sudden increases or decreases in weight, and irregularly shaped loads.

[0042] By meticulously adjusting and optimizing the initial model, and based on a comprehensive and diverse training sample set, the final VIO model not only improves accuracy but also optimizes computational efficiency, ensuring real-time response capabilities in practical applications. Joint analysis of ambient environment change images and IMU data helps enhance the system's understanding of the environment and its ability to cope with complex situations, such as providing reliable navigation support even in conditions of drastic light changes or when GPS signals are unavailable.

[0043] In summary, the final visual-inertial odometry model obtained through this detailed data collection and complex model training method greatly enhances the autonomous navigation capability of intelligent mobile tools in changing working environments, providing strong technical support for their efficient and safe operation.

[0044] In some embodiments, the method may further include the following steps: Based on the fixed weight of the intelligent mobility tool, the weight of the non-fixed objects corresponding to each seat in the intelligent mobility tool, and the weight distribution data in the intelligent mobility tool, the linear acceleration and angular acceleration data of the intelligent mobility tool are determined by the following formula: - ; in, Represents linear acceleration data for intelligent mobile tools; This indicates the fixed weight of the intelligent mobile tool itself; Indicates the weight of non-fixed objects in the smart mobility tool; indicates; This represents the i-th external force acting on the intelligent mobile tool; Represents gravitational acceleration; Indicates the amount of external force acting on a smart mobile tool; Denotes the mass of the non-fixed object at the j-th seat on the smart mobile device; Indicates the number of seats in a smart mobility device; ; in, This represents the angular acceleration data of intelligent mobile tools; The inertial tensor of an intelligent mobile tool about its center of mass; Let represent the vector from the i-th seat on the smart mobile device to the centroid; This represents the i-th external force acting on the intelligent mobile tool; Indicates the amount of external force acting on a smart mobile tool; Represents the current angular velocity of the intelligent mobility tool; Let represent the vector from the j-th seat on the smart mobile device to the centroid; Indicates the number of seats in a smart mobility device; Denotes the mass of the non-fixed object at the j-th seat on the smart mobile device; It represents the acceleration due to gravity.

[0045] In this embodiment of the application, the calculation method of the above calculation formula can make the calculation results of linear acceleration data and angular acceleration data of intelligent mobile tools more accurate.

[0046] Figure 3 A schematic diagram of a YTS-based visual inertial odometry model training device is provided. Each seat in the intelligent mobility tool is equipped with a weight detection device for detecting the weight of an object sitting on that seat. Each tire in the intelligent mobility tool is equipped with a corresponding image acquisition device, with the image acquisition direction of the image acquisition device facing that tire. Figure 3 As shown, the YTS-based visual inertial odometry model training device 300 includes: The acquisition module 301 is used to acquire the fixed weight of the intelligent mobile tool and to obtain the actual total weight based on the fixed weight and the weight of the non-fixed object collected by the weight detection device. The identification module 302 is used to identify the contact area between the tire and the ground through YTS based on the tire image acquired by the image acquisition device, and to analyze the weight distribution data in the intelligent mobile tool based on the multiple contact areas corresponding to multiple tires and the weight of the non-fixed object; the intelligent mobile tool is also equipped with a vision sensor, an accelerometer and a gyroscope; The detection module 303 is used to detect various linear acceleration and angular velocity data and various surrounding environment change images corresponding to the movement of the intelligent mobile tool under various different weight distribution data and various different non-fixed object weights; the surrounding environment change images are acquired by the vision sensor using the YTS, and the linear acceleration and angular velocity data are acquired by the accelerometer and the gyroscope; The training module 304 is used to determine the fixed self-weight, various different weight distribution data, various different non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution data and the non-fixed object weight, the surrounding environment change image, and the actual total weight as a training sample set, and to use the training sample set to train the initial visual inertial odometry model to obtain the final visual inertial odometry model.

[0047] The YTS-based visual inertial odometry model training device provided in this application embodiment has the same technical features as the YTS-based visual inertial odometry model training method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0048] An electronic device provided in this application embodiment, such as Figure 4 As shown, the electronic device 400 includes a processor 402 and a memory 401. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method provided in the above embodiments.

[0049] See Figure 4 The electronic device also includes a bus 403 and a communication interface 404. The processor 402, the communication interface 404 and the memory 401 are connected via the bus 403. The processor 402 is used to execute executable modules, such as computer programs, stored in the memory 401.

[0050] The memory 401 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 404 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0051] Bus 403 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0052] The memory 401 is used to store programs. After receiving an execution instruction, the processor 402 executes the program. The method executed by the apparatus defined by the process disclosed in any of the preceding embodiments of this application can be applied to the processor 402 or implemented by the processor 402.

[0053] Processor 402 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 402 or by instructions in software form. The processor 402 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 401, and processor 402 reads the information from memory 401 and, in conjunction with its hardware, completes the steps of the above method.

[0054] Corresponding to the above-described YTS-based visual inertial odometry model training method, this application embodiment also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and run by a processor, the computer-executable instructions cause the processor to perform the steps of the above-described YTS-based visual inertial odometry model training method.

[0055] The YTS-based visual inertial odometry model training device provided in this application embodiment can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0056] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0057] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0058] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0059] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0060] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the YTS-based visual inertial odometry model training method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0061] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0062] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for training a visual inertial odometry model based on YTS, characterized in that, Each seat in the intelligent mobility tool is equipped with a weight detection device for detecting the weight of an object sitting on that seat. Each tire in the intelligent mobility tool is equipped with an image acquisition device, the image acquisition direction of which is towards the tire. The method includes: The fixed weight of the intelligent mobile tool is obtained, and the actual total weight is obtained based on the fixed weight and the weight of the non-fixed object collected by the weight detection device. Based on the tire images acquired by the image acquisition device, the contact area between the tire and the ground is identified by YTS, and the weight distribution data in the intelligent mobile tool is analyzed based on the multiple contact areas corresponding to multiple tires and the weight of the non-fixed object; the intelligent mobile tool is also equipped with a vision sensor, accelerometer and gyroscope; The system detects various linear acceleration and angular velocity data, as well as various ambient environment change images, when the intelligent mobile tool moves under various weight distribution conditions and various non-fixed object weights. The ambient environment change images are acquired by the vision sensor using the YTS, and the linear acceleration and angular velocity data are acquired by the accelerometer and gyroscope. The fixed self-weight, various weight distribution data, various non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution and non-fixed object weight, the surrounding environment change image, and the actual total weight are determined as the training sample set. The initial visual inertial odometry model is trained using the training sample set to obtain the final visual inertial odometry model.

2. The method according to claim 1, characterized in that, The final visual inertial odometry model is used to represent the inertial data and motion estimation data of the intelligent mobile tool under various weight and weight distribution conditions.

3. The method according to claim 2, characterized in that, The intelligent mobile tool is equipped with an anti-lock braking system (ABS). After training the initial visual inertial odometry model using the training sample set to obtain the final visual inertial odometry model, the method further includes: In response to the intelligent mobile tool triggering the ABS during movement, the final visual inertial odometry model is used to analyze the inertial data and motion estimation data corresponding to the weight and weight distribution of the intelligent mobile tool when the ABS is triggered; Based on the inertial data and motion estimation data corresponding to each tire on the intelligent mobile tool, a target tire whose inertial data and / or motion estimation data exceed a specified data range is identified from among the multiple tires, and the braking force of the ABS for the brake corresponding to the target tire is increased.

4. The method according to claim 1, characterized in that, The step of determining the fixed self-weight, various different weight distribution data, various different non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution data and the non-fixed object weight, the surrounding environment change image, and the actual total weight as a training sample set, and using the training sample set to train the initial visual inertial odometry model to obtain the final visual inertial odometry model, includes: The system acquires the user's driving speed and direction control commands for the intelligent mobile tool when each of the linear acceleration and angular velocity data is detected, as well as the corresponding weight distribution data and the weight of the non-fixed object. The fixed self-weight, various different weight distribution data, various different non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution data and non-fixed object weight, the corresponding surrounding environment change image, the actual total weight corresponding to each non-fixed object weight, and the driving speed and direction control commands corresponding to each linear acceleration and angular velocity data detected, along with the corresponding weight distribution data and corresponding non-fixed object weight, are determined as a comprehensive training sample set. The initial visual inertial odometry model is trained using the comprehensive training sample set to obtain the final visual inertial odometry model.

5. The method according to claim 4, characterized in that, Each tire in the intelligent mobile tool is equipped with a tire pressure detector; The process involves defining a comprehensive training sample set, which includes the fixed self-weight, various weight distribution data, various non-fixed object weights, linear acceleration and angular velocity data corresponding to each weight distribution data and non-fixed object weight, the corresponding surrounding environment change image, the actual total weight corresponding to each non-fixed object weight, and the driving speed and direction control commands corresponding to each linear acceleration and angular velocity data detected, along with the corresponding weight distribution data and non-fixed object weight. This comprehensive training sample set is then used to train the initial visual inertial odometry model to obtain the final visual inertial odometry model, including: During the movement of the intelligent mobile tool, the tire pressure detector detects the tire pressure change data corresponding to each driving speed and direction control command executed during the movement of the intelligent mobile tool. The image acquisition device acquires a second tire image corresponding to each driving speed and direction control command executed during the movement of the intelligent mobile tool. Based on the second tire image, the YTS identifies the change data of the tire contact area with the ground corresponding to each driving speed and direction control command executed. The following data are used to determine the final comprehensive training sample set: the fixed self-weight, various different weight distribution data, various different non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution data and non-fixed object weight, the corresponding surrounding environment change image, the actual total weight corresponding to each non-fixed object weight, the driving speed and direction control command corresponding to each linear acceleration and angular velocity data detected, the corresponding weight distribution data and corresponding non-fixed object weight, the tire pressure change data corresponding to each driving speed and direction control command, and the corresponding contact area change data. The initial visual inertial odometry model is then trained using the final comprehensive training sample set to obtain the final visual inertial odometry model.

6. The method according to claim 4, characterized in that, After training the initial visual inertial odometry model using the training sample set to obtain the final visual inertial odometry model, the method further includes: During the movement of the intelligent mobile tool, the current driving speed and direction control commands, current weight distribution data, current weight of non-fixed objects, and current fixed weight of the intelligent mobile tool are acquired. Based on the current driving speed and direction control commands, the current weight distribution data, the current weight of the non-fixed object, and the current fixed self-weight, the motion estimation data of the intelligent mobile tool is predicted using the final visual inertial odometry model. The motion estimation data includes linear acceleration prediction results and angular velocity data and prediction results. Based on the linear acceleration prediction result and the angular velocity data and prediction result, the driving trajectory of the intelligent mobile tool is predicted to obtain the driving trajectory prediction result of the intelligent mobile tool under the current driving speed and direction control command.

7. The method according to claim 1, characterized in that, The method further includes: Based on the fixed weight of the intelligent mobility tool, the weight of the non-fixed object corresponding to each seat in the intelligent mobility tool, and the weight distribution data in the intelligent mobility tool, the linear acceleration and angular acceleration data of the intelligent mobility tool are determined by the following formula: - ； in, This represents the linear acceleration data of the intelligent mobile tool; This refers to the fixed weight of the intelligent mobile tool. Indicates the weight of the non-fixed object in the intelligent mobile tool; indicates; This represents the i-th external force acting on the intelligent mobile tool; Represents gravitational acceleration; This indicates the amount of external force acting on the intelligent mobile tool; This represents the mass of the non-fixed object at the j-th seat on the intelligent mobile tool; This indicates the number of seats in the smart mobility device; ； in, This represents the angular acceleration data of the intelligent mobile tool; This represents the inertial tensor of the intelligent mobile tool about its center of mass; Let represent the vector from the i-th seat on the intelligent mobile tool to the centroid; This represents the i-th external force acting on the intelligent mobile tool; This indicates the amount of external force acting on the intelligent mobile tool; This represents the current angular velocity of the intelligent mobility tool; Let represent the vector from the j-th seat on the intelligent mobile tool to the centroid; This indicates the number of seats in the smart mobility device; This represents the mass of the non-fixed object at the j-th seat on the intelligent mobile tool; It represents the acceleration due to gravity.

8. A visual-inertial odometry model training device based on YTS, characterized in that, Each seat in the intelligent mobility tool is equipped with a weight detection device for detecting the weight of an object sitting on that seat. Each tire in the intelligent mobility tool is equipped with an image acquisition device, the image acquisition direction of which is towards the tire. The device includes: The acquisition module is used to acquire the fixed weight of the intelligent mobile tool and to obtain the actual total weight based on the fixed weight and the weight of the non-fixed object collected by the weight detection device. The recognition module is used to identify the contact area between the tire and the ground based on the tire image acquired by the image acquisition device through YTS, and to analyze the weight distribution data in the intelligent mobile tool based on the multiple contact areas corresponding to multiple tires and the weight of the non-fixed object; the intelligent mobile tool is also equipped with a vision sensor, an accelerometer and a gyroscope; The detection module is used to detect various linear acceleration and angular velocity data and various surrounding environment change images corresponding to the movement of the intelligent mobile tool under various different weight distribution data and various different non-fixed object weights; the surrounding environment change images are acquired by the vision sensor using the YTS, and the linear acceleration and angular velocity data are acquired by the accelerometer and the gyroscope; The training module is used to determine the fixed self-weight, various different weight distribution data, various different non-fixed object weights, the linear acceleration and angular velocity data corresponding to each weight distribution data and the non-fixed object weight, the surrounding environment change image, and the actual total weight as a training sample set, and to use the training sample set to train the initial visual inertial odometry model to obtain the final visual inertial odometry model.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images