A mobile robot control method and system based on multimodal model

Abstract

The present invention relates to the field of robot control technology, and specifically discloses a mobile robot control method and system based on a multimodal model. The system combines the operating status data of the multimodal mobile robot through a data calculation module, and includes: a data acquisition module for acquiring the operating status data of the multimodal mobile robot during operation, calculating the data processing influence coefficient of the multimodal mobile robot at different time points, and analyzing the data information processing speed of the multimodal mobile robot at the current time point based on the data, and formulating different control strategies according to the speed of the processing speed, so as to avoid the situation where the multimodal mobile robot drives into a monitored road section before obtaining an analysis result due to poor road conditions and slow data analysis speed of the multimodal mobile robot, thereby preventing the multimodal mobile robot from being unable to move normally or even colliding with an obstacle, thereby improving the working efficiency of the multimodal mobile robot.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a mobile robot control method and system based on a multimodal model. Background Art

[0002] A multimodal action robot refers to a robot that can integrate and process information from multiple modalities, such as text, images, sound, video, etc. By utilizing multimodal artificial intelligence technology, this type of robot can achieve a more comprehensive and in-depth understanding of the external world, thereby providing more intelligent and personalized services.

[0003] Traditional multimodal mobile robot control systems use multiple sensors such as cameras, radars, and lidars to obtain information about the surrounding environment, perform path planning, obstacle avoidance, and other operations to improve driving safety and comfort. The most common method is to use cameras to collect road condition images on the road, and then extract the main feature points based on the collected road condition images. After that, data analysis is performed based on the extracted data information, and the analysis results are used to determine whether the current road conditions are good, thereby realizing functions such as path planning and obstacle avoidance.

[0004] In the existing technology, the control system of a multimodal mobile robot generally analyzes the current road conditions based on photographed road condition images. Since the existing technology lacks monitoring of the data processing speed of the multimodal mobile robot and the multimodal mobile robot continues to move during the analysis process, if the road conditions are poor and the data analysis speed of the multimodal mobile robot is slow, it may reach the monitored road section before obtaining the analysis results, which will cause the robot to be unable to move normally or even collide with obstacles, thereby affecting work efficiency. Summary of the Invention

[0005] The purpose of the present invention is to provide a mobile robot control method and system based on a multimodal model to solve the following technical problems:

[0006] How to improve the working efficiency of multimodal action robots.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A method and system for controlling a mobile robot based on a multimodal model, the system comprising:

[0009] A data acquisition module is used to collect the operating status data of the multimodal mobile robot during its working process;

[0010] A data calculation module is used to calculate the data processing influence coefficient of the multimodal mobile robot at different time points by combining the operating status data of the multimodal mobile robot;

[0011] A data analysis module is used to analyze the data information processing speed of the multimodal mobile robot at the current time point by combining the data processing influence coefficients of the multimodal mobile robot at different time points;

[0012] A decision-making module, configured to stop controlling the multimodal motion robot and perform optimization when it is determined that the data information processing speed of the multimodal motion robot at the current time point is slow;

[0013] The image modality acquisition module is used to collect road image data of the current driving route through the multimodal mobile robot and extract necessary feature points when it is determined that the data information processing speed of the multimodal mobile robot at the current time point is fast;

[0014] The recognition and analysis module is used to analyze the driving risk of the multimodal mobile robot at the current time point by combining the feature point data extracted by the image modality acquisition module.

[0015] Furthermore, the data collected by the data acquisition module includes multimodal mobile robot processor operation data, equipment data and shooting data in historical operations, wherein the processor operation data includes the processor's read and write speed, response speed, memory size and cache size at different time points, and the processor equipment data includes the processor's temperature and voltage intensity.

[0016] Furthermore, the calculation process of the data calculation module includes:

[0017] Through the data information collected in real time by the data acquisition module, the memory change curve of the multi-modal mobile robot processor is established

[0018] By formula Calculate the processor speed influence coefficient δ of the multimodal mobile robot at the i-th time point i ;

[0019] Among them, i is the time point of any data collection after the multimodal mobile robot starts working, dx i is the read and write speed of the multimodal mobile robot processor at the i-th time point, dx y is the preset read and write speed, xy i is the response speed of the multimodal mobile robot processor at the i-th time point, xy y is the preset response speed, dx 01 with xy 01 dx i with xy i The standard value of f c (x) is the first defined function, if f c (x)≥1, let fc (x) = x, otherwise, let f c (x) = 1, t i is the time point of the i-th data collection, t1 is the time point when the multimodal mobile robot starts working, hc i is the cache size of the multimodal mobile robot processor at the i-th time point, hc y The default cache size.

[0020] Furthermore, the calculation process of the data calculation module also includes:

[0021] By formula Calculate the data processing influence coefficient z of the multimodal mobile robot at the i-th time point i ;

[0022] Among them, cl i is the processor temperature of the multimodal mobile robot at the i-th time point, cl y is the preset processor temperature, sw i is the room temperature at time point i, sw y is the preset room temperature, ρ is the error influence coefficient, dy i is the voltage intensity of the multimodal mobile robot at the i-th time point, dy y is the preset voltage intensity, dy 01 For dy i The standard value of tx, a is any image taken by the mobile robot in the past work, b is the total number of images taken by the mobile robot in the past work, a The data size of the a-th image captured by the mobile robot in the past work, For all tx a The average value of .

[0023] Furthermore, the analysis process of the data analysis module includes:

[0024] By transforming the data processing influence coefficient z of the multimodal action robot at the i-th time point into i and the preset influence coefficient threshold z 01 Make a comparison;

[0025] If z i ≥z 01 ,The system determines that the data processing speed of the multimodal action robot at this time point is affected, which reduces the speed of its data processing and analysis, and it is necessary to stop the control of the multimodal action robot and optimize its processor;

[0026] If z i <z 01,The system determines that the data processing speed of the multimodal action robot at ,this point in time is not affected, its data processing and analysis speed is good, ,and its processor does not need to be optimized and can continue to ,be used.

[0027] Furthermore, the analysis process of the identification and analysis module includes:

[0028] By formula Calculate the driving risk index r of the multimodal mobile robot at the i-th time point i ;

[0029] Among them, ps i is the road damage area in the image captured by the multimodal mobile robot at the i-th time point, lf i is the road crack area in the image captured by the multimodal mobile robot at the i-th time point, zmj i The total area of ​​the road in the image captured by the multimodal mobile robot at the i-th time point, sr i is the road damage depth in the image captured by the multimodal mobile robot at the i-th time point, sr i is the preset depth, sd i is the moving speed of the multimodal mobile robot at the i-th time point, sd 01 sd i The standard value of za i is the number of obstacles in the image captured by the multimodal mobile robot at the i-th time point, za i is the preset number of roadblocks, b for za i The standard value of f v For the second defined function, if f c (x)≥1, let f c (x) = x, otherwise, let f c (x)=0, x1 and x2 are weight coefficients.

[0030] Furthermore, the analysis process of the identification and analysis module further includes:

[0031] By transforming the driving risk index r of the multimodal mobile robot at the i-th time point into i Compared with the preset risk index threshold r 01 Make a comparison;

[0032] If r i ≥r 01 ,The system determines that the driving risk of the multimodal mobile robot is high at this time point, and it needs to pause and adjust the route;

[0033] If r i <r 01,The system determines that the driving risk of the multimodal mobile robot at this time point is low, which means that the current route is in good condition and there is no need to adjust the route.

[0034] A method for controlling a mobile robot based on a multimodal model, the method comprising:

[0035] S1: Collecting the operating status data of the multimodal mobile robot during the working process through the data acquisition module;

[0036] S2: Calculate the data processing influence coefficient of the multimodal mobile robot at different time points by combining the operating status data of the multimodal mobile robot with the data calculation module;

[0037] S3: Analyze the data processing speed of the multimodal mobile robot at the current time point by combining the data processing influence coefficient of the multimodal mobile robot at different time points through the data analysis module. If the processing speed is slow, proceed to step S4; otherwise, proceed to step S5.

[0038] S4: When the decision module determines that the data information processing speed of the multimodal mobile robot at the current time point is slow, the control of the multimodal mobile robot is stopped and optimized;

[0039] S5: When it is determined through the image modality acquisition module that the multimodal mobile robot has a high data information processing speed at the current time point, the multimodal mobile robot collects road image data of the current driving route and extracts necessary feature points;

[0040] S6: Analyze the driving risk of the multimodal mobile robot at the current time point by combining the feature point data extracted by the recognition and analysis module with the image modality acquisition module.

[0041] (1) The present invention calculates the data processing influence coefficient of the multimodal motion robot at different time points in combination with the operating status data of the multimodal motion robot through a data calculation module, analyzes the data information processing speed of the multimodal motion robot at the current time point based on the data, and formulates different control strategies according to the speed of its processing speed, thereby avoiding the situation where the multimodal motion robot drives to the monitoring road section before obtaining the analysis result due to poor road conditions and slow data analysis speed of the multimodal motion robot, and then cannot move normally or even collides with obstacles, thereby improving the working efficiency of the multimodal motion robot.

[0042] (2) The present invention combines the data prediction size of the image collected by the multimodal mobile robot, the processor device data and the processor operation speed influence coefficient δ of the multimodal mobile robot at the i-th time point iFinally, diversified data support can improve the accuracy of the calculation results, thereby providing accurate data for subsequent judgment of the data processing speed of the multimodal action robot, thereby ensuring that the system can make the correct control strategy.

[0043] (3) The present invention processes the data processing influence coefficient z of the multimodal action robot at the i-th time point i and the preset influence coefficient threshold z 01 By comparing, it is possible to analyze whether the data processing speed of the multimodal motion robot is affected at that point in time, and to make an accurate judgment on the speed of its data processing and analysis based on the analysis results. By using this judgment method, when it is judged that the data processing speed of the multimodal motion robot is low, the control can be stopped in time and its processor can be optimized to avoid the situation where the multimodal motion robot cannot move normally or even collides with obstacles due to the slow analysis speed.

[0044] (4) The present invention calculates the driving risk index r of the multimodal mobile robot at the i-th time point i Compared with the preset risk index threshold r 01 Through this comparison method, an accurate judgment can be made on the driving risk of the multimodal mobile robot at that point in time, and based on the judgment results, a decision can be made whether the route needs to be adjusted, thereby helping the robot to make more accurate action plans and ensure that the robot completes the task safely and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below with reference to the accompanying drawings.

[0046] Figure 1 This is a schematic block diagram of a mobile robot control system based on a multimodal model in the present invention;

[0047] Figure 2 This is a flow chart of a mobile robot control method based on a multimodal model in the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] See also Figure 1 As shown, in one embodiment, the present application provides a method and system for controlling a mobile robot based on a multimodal model, the system comprising:

[0050] A data acquisition module is used to collect the operating status data of the multimodal mobile robot during its working process;

[0051] A data calculation module is used to calculate the data processing influence coefficient of the multimodal mobile robot at different time points by combining the operating status data of the multimodal mobile robot;

[0052] A data analysis module is used to analyze the data information processing speed of the multimodal mobile robot at the current time point by combining the data processing influence coefficients of the multimodal mobile robot at different time points;

[0053] A decision-making module, configured to stop controlling the multimodal motion robot and perform optimization when it is determined that the data information processing speed of the multimodal motion robot at the current time point is slow;

[0054] The image modality acquisition module is used to collect road image data of the current driving route through the multimodal mobile robot and extract necessary feature points when it is determined that the data information processing speed of the multimodal mobile robot at the current time point is fast;

[0055] The recognition and analysis module is used to analyze the driving risk of the multimodal mobile robot at the current time point by combining the feature point data extracted by the image modality acquisition module;

[0056] Through the above technical solution, this embodiment provides a data acquisition module to collect the operating status data of the multimodal motion robot during the working process. When the multimodal motion robot is working, the data calculation module combines the operating status data of the multimodal motion robot to calculate the data processing influence coefficient of the multimodal motion robot at different time points, and the data analysis module uses this data to analyze the data information processing speed of the multimodal motion robot at the current time point. When it is judged that the data information processing speed of the multimodal motion robot at the current time point is slow, the decision module stops the control of the multimodal motion robot and optimizes it. When it is judged that the data information processing speed of the multimodal motion robot at the current time point is fast, the image modality acquisition module collects road image data of the current driving route of the multimodal motion robot and extracts necessary feature points. Finally, the recognition and analysis module combines the feature point data extracted by the image modality acquisition module to analyze the driving risk of the multimodal motion robot at the current time point.

[0057] Through such a setting, when the multimodal motion robot is working, the data processing influence coefficient of the multimodal motion robot at different time points is calculated through the data calculation module in combination with the operating status data of the multimodal motion robot, and the data information processing speed of the multimodal motion robot at the current time point is analyzed based on the data, and different control strategies are formulated according to the speed of its processing speed, so as to avoid the situation where the multimodal motion robot drives to the monitored road section before obtaining the analysis result due to poor road conditions and slow data analysis speed of the multimodal motion robot, and then cannot move normally or even collides with obstacles, thereby improving the working efficiency of the multimodal motion robot.

[0058] The data collected by the data collection module includes the multimodal mobile robot's processor operation data, device data, and historical operation shooting data. The processor operation data includes the processor's read and write speed, response speed, memory size, and cache size at different time points. The processor device data includes the processor's temperature and voltage intensity.

[0059] Through the above technical solution, this example provides data collected by the data acquisition module, which can reflect the processor operating status of the multimodal mobile robot at different time points during work, thereby providing accurate data for the system to judge the data analysis speed of the multimodal mobile robot.

[0060] The calculation process of the data calculation module includes:

[0061] Through the data information collected in real time by the data acquisition module, the memory change curve of the multi-modal mobile robot processor is established

[0062] By formula Calculate the processor speed influence coefficient δ of the multimodal mobile robot at the i-th time point i ;

[0063] Among them, i is the time point of any data collection after the multimodal mobile robot starts working, dx i is the read and write speed of the multimodal mobile robot processor at the i-th time point, dx y is the preset read and write speed, xy i is the response speed of the multimodal mobile robot processor at the i-th time point, xy y is the preset response speed, dx 01 with xy 01 dx i with xy i The standard value of the above standard value can be selected and set according to the allowable error in the empirical data, f c (x) is the first defined function, if fc (x)≥1, let f c (x) = x, otherwise, let f c (x) = 1, t i is the time point of the i-th data collection, t1 is the time point when the multimodal mobile robot starts working, hc i is the cache size of the multimodal mobile robot processor at the i-th time point, hc y is the preset cache size;

[0064] Through the above technical solution, this example provides the processor operation speed influence coefficient δ of the multimodal mobile robot at the i-th time point i , can be obtained by formula Calculated, where the formula The change in the processor memory of the multimodal mobile robot from the time it starts working to the time of the i-th data collection can be calculated. Therefore, it is obvious that when the read and write speed and response speed of the multimodal mobile robot processor at the i-th time point are slower, and the change in the processor memory and the processor cache are larger, the processor running speed influence coefficient δ of the multimodal mobile robot at the i-th time point is i The larger the value, the slower the data analysis speed of the multimodal mobile robot at the current time point. On the contrary, when the read and write speed and response speed of the multimodal mobile robot processor at the i-th time point are faster, and the change in the processor memory and the processor cache are smaller, the processor running speed influence coefficient δ of the multimodal mobile robot at the i-th time point is i The smaller it is, the faster the data analysis speed of the multimodal action robot is at the current time point;

[0065] By setting it in this way, accurate data can be provided for the subsequent analysis of the processing speed of the multimodal mobile robot, thereby ensuring the accuracy of the analysis results.

[0066] The calculation process of the data calculation module also includes:

[0067] By formula Calculate the data processing influence coefficient z of the multimodal mobile robot at the i-th time point i ;

[0068] Among them, cl i is the processor temperature of the multimodal mobile robot at the i-th time point, cl y is the preset processor temperature, sw i is the room temperature at time point i, sw y is the preset room temperature, ρ is the error influence coefficient, dy i is the voltage intensity of the multimodal mobile robot at the i-th time point, dyy is the preset voltage intensity, dy 01 For dy i The standard value can be set according to the allowable error in the empirical data. a is any image taken by the mobile robot in the past work, b is the total number of images taken by the mobile robot in the past work, tx a The data size of the a-th image captured by the mobile robot in the past work, For all tx a The average value of

[0069] Through the above technical solution, this embodiment provides the data processing influence coefficient z of the multimodal mobile robot at the i-th time point i , can be obtained by formula Calculated, where the formula The data size of the image collected by the multimodal mobile robot in the future can be predicted. Therefore, it is obvious that when the processor temperature of the multimodal mobile robot at the i-th time point is higher than the room temperature, the data size of the collected image is larger, and the voltage intensity of the multimodal mobile robot at the i-th time point is more different from the preset value, the data processing influence coefficient z of the multimodal mobile robot at the i-th time point will be i The larger the value, the greater the impact of the data processing speed of the current multimodal mobile robot will be, which will reduce the speed of data analysis and make it impossible to quickly analyze and execute the acquired instructions or information. On the contrary, when the processor temperature of the multimodal mobile robot at the i-th time point is lower than the room temperature, the data size of the collected image is smaller, and the voltage intensity of the multimodal mobile robot at the i-th time point is smaller than the preset value, the data processing influence coefficient z of the multimodal mobile robot at the i-th time point will be greater. i The smaller it is, the less likely it is that the data processing speed of the current multimodal action robot will be significantly affected, and the speed of data analysis will not be reduced, so the obtained instructions or information can be analyzed and executed quickly;

[0070] Through this calculation method, combined with the data prediction size of the image collected by the multimodal mobile robot, the processor device data and the processor operation speed influence coefficient δ of the multimodal mobile robot at the i-th time point, i Finally, diversified data support can improve the accuracy of calculation results, thereby providing accurate data for subsequent judgment of the data processing speed of the multimodal action robot, and thus ensuring that the system can make correct control strategies.

[0071] The analysis process of the data analysis module includes:

[0072] By transforming the data processing influence coefficient z of the multimodal action robot at the i-th time point into iand the preset influence coefficient threshold z 01 Make a comparison;

[0073] If z i ≥z 01 ,The system determines that the data processing speed of the multimodal action robot at this time point is affected, which reduces the speed of its data processing and analysis, and it is necessary to stop the control of the multimodal action robot and optimize its processor;

[0074] If z i <z 01 ,The system determines that the data processing speed of the multimodal action robot at ,this time point is not affected, its data processing and analysis speed is good, ,and its processor does not need to be optimized and can continue to ,be used;

[0075] Through the above technical solution, this example transforms the data processing influence coefficient z of the multimodal mobile robot at the i-th time point into i and the preset influence coefficient threshold z 01 By comparing, it is possible to analyze whether the data processing speed of the multimodal motion robot is affected at that point in time, and to make an accurate judgment on the speed of its data processing and analysis based on the analysis results. By using this judgment method, when it is judged that the data processing speed of the multimodal motion robot is low, the control can be stopped in time and its processor can be optimized to avoid the situation where the multimodal motion robot cannot move normally or even collides with obstacles due to the slow analysis speed.

[0076] The analysis process of the identification and analysis module includes:

[0077] By formula Calculate the driving risk index r of the multimodal mobile robot at the i-th time point i ;

[0078] Among them, ps i is the road damage area in the image captured by the multimodal mobile robot at the i-th time point, lf i is the road crack area in the image captured by the multimodal mobile robot at the i-th time point, zmj i The total area of ​​the road in the image captured by the multimodal mobile robot at the i-th time point, sr i is the road damage depth in the image captured by the multimodal mobile robot at the i-th time point, sr y is the preset depth, sd i is the moving speed of the multimodal mobile robot at the i-th time point, sd 01 sd i The standard value can be selected and set according to the allowable error in the empirical data. iis the number of obstacles in the image captured by the multimodal mobile robot at the i-th time point, za i is the preset number of roadblocks, b for za i The standard value of the above standard value can be selected and set according to the allowable error in the empirical data, f v For the second defined function, if f c (x)≥1, let f c (x) = x, otherwise, let f c (x) = 0, x1 and x2 are weight coefficients, set according to empirical fitting;

[0079] Through the above technical solution, this embodiment provides the driving risk index r of the multimodal mobile robot at the i-th time point i , can be obtained by formula It is obvious that when the crack area, damaged area and damaged depth of the road in the image taken at the i-th time point are larger, and the speed of the multimodal mobile robot is faster and the number of roadblocks is greater, the driving risk index r of the multimodal mobile robot at the i-th time point is i The higher the value, the higher the risk of continuing to drive on the current route. On the contrary, when the road crack area, damage area and damage depth in the image taken at the i-th time point are smaller, and the speed of the multimodal mobile robot is slower and the number of roadblocks is 0, then the driving risk index r of the multimodal mobile robot at the i-th time point is i The lower the risk, the lower the risk of continuing to travel on the current route. This calculation method reflects the risk factor of the multimodal mobile robot's current route, so that the system decides whether to change the route and provides accurate data to avoid the multimodal mobile robot from being unable to move normally or colliding with obstacles on a high-risk route.

[0080] The analysis process of the identification and analysis module further includes:

[0081] By transforming the driving risk index r of the multimodal mobile robot at the i-th time point into i Compared with the preset risk index threshold r 01 Make a comparison;

[0082] If r i ≥r 01 ,The system determines that the driving risk of the multimodal mobile robot is high at this time point, and it needs to pause and adjust the route;

[0083] If r i <r 01 ,The system determines that the driving risk of the multimodal mobile robot at this time point is low, which means that the current route is in good condition and there is no need to adjust the route;

[0084] Through the above technical solution, this example calculates the driving risk index r of the multimodal mobile robot at the i-th time point i Compared with the preset risk index threshold r 01 Through this comparison method, an accurate judgment can be made on the driving risk of the multimodal mobile robot at that point in time, and based on the judgment results, a decision can be made whether the route needs to be adjusted, thereby helping the robot to make more accurate action plans and ensure that the robot completes the task safely and efficiently.

[0085] See also Figure 2 As shown, a mobile robot control method based on a multimodal model comprises:

[0086] S1: Collecting the operating status data of the multimodal mobile robot during the working process through the data acquisition module;

[0087] S2: Calculate the data processing influence coefficient of the multimodal mobile robot at different time points by combining the operating status data of the multimodal mobile robot with the data calculation module;

[0088] S3: Analyze the data processing speed of the multimodal mobile robot at the current time point by combining the data processing influence coefficient of the multimodal mobile robot at different time points through the data analysis module. If the processing speed is slow, proceed to step S4; otherwise, proceed to step S5.

[0089] S4: When the decision module determines that the data information processing speed of the multimodal mobile robot at the current time point is slow, the control of the multimodal mobile robot is stopped and optimized;

[0090] S5: When it is determined through the image modality acquisition module that the multimodal mobile robot has a high data information processing speed at the current time point, the multimodal mobile robot collects road image data of the current driving route and extracts necessary feature points;

[0091] S6: Analyze the driving risk of the multimodal mobile robot at the current time point through the recognition and analysis module combined with the feature point data extracted by the image modality acquisition module;

[0092] Through the above technical solution, this embodiment provides a mobile robot control method based on a multimodal model, firstly, the operating status data of the multimodal mobile robot during the working process is collected by the data acquisition module, and then the data processing influence coefficient of the multimodal mobile robot at different time points is calculated by the data calculation module in combination with the operating status data of the multimodal mobile robot, and the data analysis module is combined with the data processing influence coefficient of the multimodal mobile robot at different time points to analyze the speed of data information processing of the multimodal mobile robot at the current time point. When it is judged that the data information processing speed of the multimodal mobile robot at the current time point is slow, the control of the multimodal mobile robot is stopped and optimized. When the data information processing speed at the current time point is fast, the multimodal mobile robot collects road image data of the current driving route and extracts necessary feature points. Finally, the recognition and analysis module is combined with the feature point data extracted by the image modality acquisition module to analyze the driving risk of the multimodal mobile robot at the current time point.

[0093] Through such a setting, by combining the operating status data of the multimodal motion robot during work, an accurate judgment can be made on the data processing impact coefficient of the multimodal motion robot at different time points, and different follow-up plans can be formulated based on the judgment, thereby avoiding the situation where the multimodal motion robot drives to the monitored road section before obtaining the analysis result due to poor road conditions and slow data analysis speed of the multimodal motion robot, and then cannot move normally or even collides with obstacles, thereby improving the work efficiency of the multimodal motion robot.

[0094] The above is a detailed description of an embodiment of the present invention. However, the content described is only a preferred embodiment of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A mobile robot control system based on a multimodal model, characterized in that: The system comprises: A data acquisition module is used to collect the operating status data of the multimodal mobile robot during its working process; A data calculation module is used to calculate the data processing influence coefficient of the multimodal mobile robot at different time points by combining the operating status data of the multimodal mobile robot; A data analysis module is used to analyze the data information processing speed of the multimodal mobile robot at the current time point by combining the data processing influence coefficients of the multimodal mobile robot at different time points; A decision-making module, configured to stop controlling the multimodal motion robot and perform optimization when it is determined that the data information processing speed of the multimodal motion robot at the current time point is slow; The image modality acquisition module is used to collect road image data of the current driving route through the multimodal mobile robot and extract necessary feature points when it is determined that the data information processing speed of the multimodal mobile robot at the current time point is fast; The recognition and analysis module is used to analyze the driving risk of the multimodal mobile robot at the current time point by combining the feature point data extracted by the image modality acquisition module; The data collected by the data collection module includes the multimodal mobile robot's processor operation data, device data, and historical operation shooting data. The processor operation data includes the processor's read and write speed, response speed, memory size, and cache size at different time points. The processor device data includes the processor's temperature and voltage intensity. The calculation process of the data calculation module includes: The data information collected in real time by the data acquisition module is used to establish the memory change curve θ(t) of the multimodal mobile robot processor. By formula Calculate the processor speed influence coefficient δ of the multimodal mobile robot at the i-th time point i ; Among them, i is the time point of any data collection after the multimodal mobile robot starts working, dx i is the read and write speed of the multimodal mobile robot processor at the i-th time point, dx y is the preset read and write speed, xy i is the response speed of the multimodal mobile robot processor at the i-th time point, xy y is the preset response speed, dx 01 with xy 01 dx i with xy i The standard value of f c (x) is the first defined function, if f c (x)≥1, let f c (x) = x, otherwise, let f c (x) = 1, t i is the time point of the i-th data collection, t1 is the time point when the multimodal mobile robot starts working, hc i is the cache size of the multimodal mobile robot processor at the i-th time point, hc y is the preset cache size; The calculation process of the data calculation module also includes: By formula Calculate the data processing influence coefficient z of the multimodal mobile robot at the i-th time point i ; Among them, cl i is the processor temperature of the multimodal mobile robot at the i-th time point, cl y is the preset processor temperature, sw i is the room temperature at time point i, sw y is the preset room temperature, ρ is the error influence coefficient, dy i is the voltage intensity of the multimodal mobile robot at the i-th time point, dy y is the preset voltage intensity, dy 01 For dy i The standard value of tx, a is any image taken by the mobile robot in the past work, b is the total number of images taken by the mobile robot in the past work, a The data size of the a-th image captured by the mobile robot in the past work, For all tx a The average value of The analysis process of the data analysis module includes: By transforming the data processing influence coefficient z of the multimodal action robot at the i-th time point into i and the preset influence coefficient threshold z 01 Make a comparison; If z i ≥z 01 ,The system determines that the data processing speed of the multimodal action robot at this time point is affected, which reduces the speed of its data processing and analysis, and it is necessary to stop the control of the multimodal action robot and optimize its processor; If z i <z 01 ,The system determines that the data processing speed of the multimodal action robot at ,this point in time is not affected, its data processing and analysis speed is good, ,and its processor does not need to be optimized and can continue to ,be used.

2. The mobile robot control system based on a multimodal model according to claim 1, characterized in that: The analysis process of the identification and analysis module includes: By formula Calculate the driving risk index r of the multimodal mobile robot at the i-th time point i ; Among them, ps i is the road damage area in the image captured by the multimodal mobile robot at the i-th time point, lf i is the road crack area in the image captured by the multimodal mobile robot at the i-th time point, zmj i The total area of ​​the road in the image captured by the multimodal mobile robot at the i-th time point, sr i is the road damage depth in the image captured by the multimodal mobile robot at the i-th time point, sr y is the preset depth, sd i is the moving speed of the multimodal mobile robot at the i-th time point, sd 01 sd i The standard value of za i is the number of obstacles in the image captured by the multimodal mobile robot at the i-th time point, za i is the preset number of roadblocks, b for za i The standard value of f v For the second defined function, if f c (x)≥1, let f c (x) = x, otherwise, let f c (x)=0, x1 and x2 are weight coefficients.

3. The mobile robot control system based on a multimodal model according to claim 2, characterized in that: The analysis process of the identification and analysis module further includes: By transforming the driving risk index r of the multimodal mobile robot at the i-th time point into i and the preset risk index threshold r 01 Make a comparison; If r i ≥r 01 ,The system determines that the driving risk of the multimodal mobile robot is high at this time point, and it needs to pause and adjust the route; If r i <r 01 ,The system determines that the driving risk of the multimodal mobile robot at this time point is low, which means that the current route is in good condition and there is no need to adjust the route.

4. A method for controlling a mobile robot based on a multimodal model, wherein the method adopts a mobile robot control system based on a multimodal model according to claims 1-3, characterized in that: The method comprises: S1: Collecting the operating status data of the multimodal mobile robot during the working process through the data acquisition module; S2: Calculate the data processing influence coefficient of the multimodal mobile robot at different time points by combining the operating status data of the multimodal mobile robot with the data calculation module; S3: Analyze the data processing speed of the multimodal mobile robot at the current time point by combining the data processing influence coefficient of the multimodal mobile robot at different time points through the data analysis module. If the processing speed is slow, proceed to step S4; otherwise, proceed to step S5. S4: When the decision module determines that the data information processing speed of the multimodal mobile robot at the current time point is slow, the control of the multimodal mobile robot is stopped and optimized; S5: When it is determined through the image modality acquisition module that the multimodal mobile robot has a high data information processing speed at the current time point, the multimodal mobile robot collects road image data of the current driving route and extracts necessary feature points; S6: Analyze the driving risk of the multimodal mobile robot at the current time point through the recognition and analysis module combined with the feature point data extracted by the image modality acquisition module.

Images