Mobile robot control method and system based on multi-modal model

Abstract

The invention relates to the technical field of robot control, and particularly discloses an action robot control method and system based on a multi-modal model, and the system comprises a data acquisition module, a data processing module, a data processing module and a control module, the data processing module is used for collecting operation state data of the multi-mode action robot in the working process, calculating data processing influence coefficients of the multi-mode action robot at different time points, and analyzing the data information processing speed of the multi-mode action robot at the current time point according to the data. And different control strategies are formulated according to the processing speed, so that the situation that the multi-mode action robot runs to a monitored road section before obtaining an analysis result due to relatively poor road conditions and relatively low data analysis speed of the multi-mode action robot, so that the situation that the multi-mode action robot cannot normally move and even collides with an obstacle occurs is avoided; therefore, the working efficiency of the multi-mode action robot is improved.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a mobile robot control method and system based on a multimodal model. Background Art

[0002] A multimodal action robot refers to a robot that can integrate and process information from multiple modalities, such as text, images, sound, video, etc. By utilizing multimodal artificial intelligence technology, this type of robot can achieve a more comprehensive and in-depth understanding of the external world, thereby providing more intelligent and personalized services.

[0003] The traditional multimodal mobile robot control system uses multiple sensors such as cameras, radars, and lidars to obtain information about the surrounding environment, perform path planning, obstacle avoidance and other operations, and improve driving safety and comfort. The most common method is to collect road condition pictures on the road through cameras, and then extract the main feature points based on the collected road condition pictures. After that, data analysis is performed based on the extracted data information, and the analysis results are used to determine whether the current road condition is good, thereby realizing functions such as path planning and obstacle avoidance.

[0004] In the prior art, the control system of a multimodal action robot generally analyzes the current road condition based on the taken road condition pictures. Since the prior art lacks monitoring of the data processing speed of the multimodal action robot and the multimodal action robot will continue to move during the analysis process, if the road condition is poor and the data analysis speed of the multimodal action robot is slow, it may drive to the monitored road section before the analysis result is obtained, which will cause the robot to be unable to move normally or even collide with obstacles, thereby affecting work efficiency. Summary of the invention

[0005] The purpose of the present invention is to provide a mobile robot control method and system based on a multimodal model to solve the following technical problems:

[0006] How to improve the working efficiency of multimodal action robots.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A method and system for controlling a mobile robot based on a multimodal model, the system comprising:

[0009] A data acquisition module is used to collect the operating status data of the multimodal action robot during its working process;

[0010] A data calculation module, used to calculate the data processing influence coefficient of the multimodal action robot at different time points in combination with the operation status data of the multimodal action robot;

[0011] A data analysis module, used to analyze the data information processing speed of the multimodal action robot at the current time point by combining the data processing influence coefficient of the multimodal action robot at different time points;

[0012] A decision-making module, used to stop controlling the multimodal action robot and perform optimization when it is determined that the data information processing speed of the multimodal action robot at the current time point is slow;

[0013] The image modality acquisition module is used to collect the road image data of the current driving route through the multimodal action robot and extract the necessary feature points when it is determined that the data information processing speed of the multimodal action robot at the current time point is fast;

[0014] The recognition and analysis module is used to analyze the driving risk of the multimodal action robot at the current time point in combination with the feature point data extracted by the image modality acquisition module.

[0015] Furthermore, the data collected by the data acquisition module include multimodal mobile robot processor operation data, equipment data and shooting data in historical operations, wherein the processor operation data includes the processor's read and write speed, response speed, memory size and cache size at different time points, and the processor equipment data includes the processor's temperature and voltage intensity.

[0016] Furthermore, the calculation process of the data calculation module includes:

[0017] The data information collected in real time by the data acquisition module is used to establish the memory change curve of the multi-modal action robot processor.

[0018] By formula Calculate the processor speed influence coefficient δ of the multimodal mobile robot at the i-th time point i ;

[0019] Among them, i is the time point of any data collection after the multimodal mobile robot starts working, dx i is the read and write speed of the multimodal mobile robot processor at the i-th time point, dx y is the preset read and write speed, xy i is the response speed of the multimodal mobile robot processor at the i-th time point, xy y is the preset response speed, dx 01 With xy 01 dx i With xy i The standard value, f c (x) is the first defined function, if f c (x)≥1, let fc (x) = x, otherwise, let f c (x) = 1, t i is the time point of the i-th data collection, t 1 is the time point when the multimodal action robot starts working, hc i is the cache size of the multimodal mobile robot processor at the i-th time point, hc y The default cache size.

[0020] Furthermore, the calculation process of the data calculation module also includes:

[0021] By formula Calculate the data processing influence coefficient z of the multimodal mobile robot at the i-th time point i ;

[0022] Among them, cl i is the processor temperature of the multimodal mobile robot at the i-th time point, cl y is the preset processor temperature, sw i is the room temperature at the i-th time point, sw y is the preset room temperature, ρ is the error influence coefficient, dy i is the voltage intensity of the multimodal mobile robot at the i-th time point, dy y is the preset voltage intensity, dy 01 For i The standard value of tx, a is any image taken by the mobile robot in the past work, b is the total number of images taken by the mobile robot in the past work, a is the data size of the ath image taken by the mobile robot in the past work, For all tx a The average value of .

[0023] Furthermore, the analysis process of the data analysis module includes:

[0024] By transforming the data processing influence coefficient z of the multimodal action robot at the i-th time point i and the preset influence coefficient threshold z 01 Make a comparison;

[0025] If i ≥z 01 , the system determines that the data processing speed of the multimodal action robot at this time point is affected, which reduces the speed of its data processing and analysis, and needs to stop the control of the multimodal action robot and optimize its processor;

[0026] If i <z 01,The system determines that the data processing speed of the multimodal action robot at this time point is not affected, and its data processing and analysis speed is good. There is no need to optimize its processor and it can continue to be used.

[0027] Furthermore, the analysis process of the identification and analysis module includes:

[0028] By formula Calculate the driving risk index r of the multimodal mobile robot at the i-th time point i ;

[0029] Among them, ps i is the road damage area in the image taken by the multimodal mobile robot at the i-th time point, lf i is the road crack area in the image taken by the multimodal mobile robot at the i-th time point, zmj i The total area of ​​the road in the image captured by the multimodal mobile robot at the i-th time point, sr i is the road damage depth in the image taken by the multimodal mobile robot at the i-th time point, sr i is the preset depth, sd i is the moving speed of the multimodal mobile robot at the i-th time point, sd 01 sd i The standard value of i is the number of obstacles in the image taken by the multimodal mobile robot at the i-th time point, za i is the preset number of roadblocks, b for za i The standard value, f v For the second defined function, if f c (x)≥1, let f c (x) = x, otherwise, let f c (x)=0, x1 and x2 are weight coefficients.

[0030] Furthermore, the analysis process of the identification and analysis module also includes:

[0031] By transforming the driving risk index r of the multimodal mobile robot at the i-th time point into i The preset risk index threshold r 01 Make a comparison;

[0032] If r i ≥r 01 ,The system determines that the driving risk of the multimodal action robot is high at this time point, and needs to stop moving and adjust the route;

[0033] If r i <r 01,The system determines that the driving risk of the multimodal mobile robot at this time point is low, which means that the current route is in good condition and there is no need to adjust the route.

[0034] A method for controlling a mobile robot based on a multimodal model, the method comprising:

[0035] S1: Collecting the operation status data of the multimodal action robot during the working process through the data acquisition module;

[0036] S2: Calculate the data processing influence coefficient of the multimodal action robot at different time points by combining the operation status data of the multimodal action robot through the data calculation module;

[0037] S3: Analyze the data processing speed of the multimodal action robot at the current time point by combining the data processing influence coefficient of the multimodal action robot at different time points through the data analysis module. If the processing speed is slow, proceed to step S4; otherwise, proceed to step S5.

[0038] S4: when the decision module determines that the data information processing speed of the multimodal action robot at the current time point is slow, the control of the multimodal action robot is stopped and optimized;

[0039] S5: When it is determined through the image modality acquisition module that the data information processing speed of the multimodal mobile robot at the current time point is fast, the multimodal mobile robot collects road image data of the current driving route and extracts necessary feature points;

[0040] S6: Analyze the driving risk of the multi-modal mobile robot at the current time point by combining the feature point data extracted by the recognition and analysis module with the image modality acquisition module.

[0041] (1) The present invention calculates the data processing influence coefficient of the multimodal motion robot at different time points in combination with the running status data of the multimodal motion robot through a data calculation module, analyzes the data information processing speed of the multimodal motion robot at the current time point based on the data, and formulates different control strategies according to the processing speed, thereby avoiding the situation where the multimodal motion robot drives to the monitoring road section before obtaining the analysis result due to poor road conditions and slow data analysis speed of the multimodal motion robot, and then cannot move normally or even collides with obstacles, thereby improving the working efficiency of the multimodal motion robot.

[0042] (2) The present invention combines the data prediction size of the image collected by the multimodal mobile robot, the processor device data, and the processor operation speed influence coefficient δ of the multimodal mobile robot at the i-th time point iFinally, diversified data support can improve the accuracy of the calculation results, thereby providing accurate data for the subsequent judgment of the data processing speed of the multimodal action robot, thereby ensuring that the system can make the correct control strategy.

[0043] (3) The present invention calculates the data processing influence coefficient z of the multimodal action robot at the i-th time point i and the preset influence coefficient threshold z 01 By comparing, it is possible to analyze whether the data processing speed of the multimodal action robot is affected at that point in time, and to make an accurate judgment on the speed of its data processing analysis based on the analysis results. By using this judgment method, when it is judged that the data processing speed of the multimodal action robot is low, the control can be stopped in time and its processor can be optimized to avoid the situation where the multimodal action robot cannot move normally or even collides with obstacles due to the slow analysis speed.

[0044] (4) The present invention calculates the driving risk index r of the multimodal mobile robot at the i-th time point i The preset risk index threshold r 01 Through this comparison method, an accurate judgment can be made on the driving risk of the multimodal action robot at that point in time, and a decision can be made based on the judgment result whether the route needs to be adjusted, thereby helping the robot to make more accurate action plans and ensure that the robot completes the task safely and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below in conjunction with the accompanying drawings.

[0046] Figure 1 It is a schematic block diagram of a mobile robot control system based on a multimodal model in the present invention;

[0047] Figure 2 It is a flow chart of a method for controlling a mobile robot based on a multimodal model in the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] See also Figure 1 As shown, in one embodiment, the present application provides a method and system for controlling a mobile robot based on a multimodal model, the system comprising:

[0050] A data acquisition module is used to collect the operating status data of the multimodal action robot during its working process;

[0051] A data calculation module, used to calculate the data processing influence coefficient of the multimodal action robot at different time points in combination with the operation status data of the multimodal action robot;

[0052] A data analysis module, used to analyze the data information processing speed of the multimodal action robot at the current time point by combining the data processing influence coefficient of the multimodal action robot at different time points;

[0053] A decision-making module, used to stop controlling the multimodal action robot and perform optimization when it is determined that the data information processing speed of the multimodal action robot at the current time point is slow;

[0054] The image modality acquisition module is used to collect the road image data of the current driving route through the multimodal action robot and extract the necessary feature points when it is determined that the data information processing speed of the multimodal action robot at the current time point is fast;

[0055] The recognition and analysis module is used to analyze the driving risk of the multimodal action robot at the current time point by combining the feature point data extracted by the image modality acquisition module;

[0056] Through the above technical scheme, this embodiment provides a data acquisition module to collect the running status data of the multimodal action robot during the working process. When the multimodal action robot is working, the data processing influence coefficient of the multimodal action robot at different time points is calculated by the data calculation module in combination with the running status data of the multimodal action robot, and the data information processing speed of the multimodal action robot at the current time point is analyzed by the data analysis module. When it is judged that the data information processing speed of the multimodal action robot at the current time point is slow, the control of the multimodal action robot is stopped and optimized through the decision module. When it is judged that the data information processing speed of the multimodal action robot at the current time point is fast, the road image data of the current driving route of the multimodal action robot is collected through the image modality acquisition module, and the necessary feature points are extracted. Finally, the driving risk of the multimodal action robot at the current time point is analyzed by the recognition and analysis module in combination with the feature point data extracted by the image modality acquisition module.

[0057] Through such an arrangement, when the multimodal motion robot is working, the data processing influence coefficient of the multimodal motion robot at different time points is calculated through the data calculation module in combination with the operating status data of the multimodal motion robot, and the data information processing speed of the multimodal motion robot at the current time point is analyzed based on the data, and different control strategies are formulated according to the processing speed, so as to avoid the situation where the multimodal motion robot drives to the monitored road section before obtaining the analysis result due to poor road conditions and the slow data analysis speed of the multimodal motion robot, and then cannot move normally or even collides with obstacles, thereby improving the working efficiency of the multimodal motion robot.

[0058] The data collected by the data collection module include the multimodal mobile robot processor operation data, equipment data and shooting data in historical operations, wherein the processor operation data includes the processor's read and write speed, response speed, memory size and cache size at different time points, and the processor equipment data includes the processor's temperature and voltage intensity;

[0059] Through the above technical solution, this example provides data collected by the data acquisition module, which can reflect the processor operating status of the multimodal action robot at different time points during work, thereby providing accurate data for the system to judge the data analysis speed of the multimodal action robot.

[0060] The calculation process of the data calculation module includes:

[0061] The data information collected in real time by the data acquisition module is used to establish the memory change curve of the multi-modal action robot processor.

[0062] By formula Calculate the processor speed influence coefficient δ of the multimodal mobile robot at the i-th time point i ;

[0063] Among them, i is the time point of any data collection after the multimodal mobile robot starts working, dx i is the read and write speed of the multimodal mobile robot processor at the i-th time point, dx y is the preset read and write speed, xy i is the response speed of the multimodal mobile robot processor at the i-th time point, xy y is the preset response speed, dx 01 With xy 01 dx i With xy i The standard value of f can be selected and set according to the allowable error in the empirical data. c (x) is the first defined function, if fc (x)≥1, let f c (x) = x, otherwise, let f c (x) = 1, t i is the time point of the i-th data collection, t 1 is the time point when the multimodal action robot starts working, hc i is the cache size of the multimodal mobile robot processor at the i-th time point, hc y is the preset cache size;

[0064] Through the above technical solution, this example provides a multimodal mobile robot processor operation speed influence coefficient δ at the i-th time point i , can be obtained by formula Calculated, where the formula The change in the processor memory of the multimodal action robot from the time when it starts working to the time when the i-th data is collected can be calculated. Therefore, it is obvious that when the read and write speed and response speed of the multimodal action robot processor at the i-th time point are slower, and the change in the processor memory and the processor cache are larger, the processor running speed influence coefficient δ of the multimodal action robot at the i-th time point is i The larger the value, the slower the data analysis speed of the multimodal action robot at the current time point. On the contrary, when the read / write speed and response speed of the multimodal action robot processor at the i-th time point are faster, and the change in the processor memory and the processor cache are smaller, the processor running speed influence coefficient δ of the multimodal action robot at the i-th time point is i The smaller it is, the faster the data analysis speed of the multimodal action robot is at the current time point;

[0065] Through such a setting, accurate data can be provided for the subsequent analysis of the processing speed of the multimodal action robot, thereby ensuring the accuracy of the analysis results.

[0066] The calculation process of the data calculation module also includes:

[0067] By formula Calculate the data processing influence coefficient z of the multimodal mobile robot at the i-th time point i ;

[0068] Among them, cl i is the processor temperature of the multimodal mobile robot at the i-th time point, cl y is the preset processor temperature, sw i is the room temperature at the i-th time point, sw y is the preset room temperature, ρ is the error influence coefficient, dy iis the voltage intensity of the multimodal mobile robot at the i-th time point, dy y is the preset voltage intensity, dy 01 For i The standard value can be set according to the allowable error in the empirical data. a is any image taken by the mobile robot in the past work, b is the total number of images taken by the mobile robot in the past work, tx a is the data size of the ath image taken by the mobile robot in the past work, For all tx a The average value of

[0069] Through the above technical solution, this embodiment provides the data processing influence coefficient z of the multimodal mobile robot at the i-th time point i , can be obtained by formula Calculated, where the formula The data size of the image collected by the multimodal action robot in the future can be predicted. Therefore, it is obvious that when the processor temperature of the multimodal action robot at the i-th time point is higher than the room temperature, the data size of the collected image is larger, and the voltage intensity of the multimodal action robot at the i-th time point is more different from the preset value, the data processing influence coefficient z of the multimodal action robot at the i-th time point is i The larger the value, the greater the impact of the current multimodal action robot's data processing speed will be, which will reduce the speed of data analysis and make it impossible to quickly analyze and execute the acquired instructions or information. On the contrary, when the processor temperature of the multimodal action robot at the i-th time point is lower than the room temperature, the data size of the collected image is smaller, and the voltage intensity of the multimodal action robot at the i-th time point is smaller than the preset value, the data processing impact coefficient z of the multimodal action robot at the i-th time point is i The smaller it is, the data processing speed of the current multimodal action robot will not be significantly affected, and the speed of data analysis will not be reduced, and the obtained instructions or information can be quickly analyzed and executed;

[0070] Through this calculation method, combined with the data prediction size of the image collected by the multimodal mobile robot, the processor device data and the processor running speed influence coefficient δ of the multimodal mobile robot at the i-th time point i Finally, diversified data support can improve the accuracy of the calculation results, thereby providing accurate data for the subsequent judgment of the data processing speed of the multimodal action robot, and thus ensuring that the system can make the correct control strategy.

[0071] The analysis process of the data analysis module includes:

[0072] By transforming the data processing influence coefficient z of the multimodal action robot at the i-th time point i and the preset influence coefficient threshold z 01 Make a comparison;

[0073] If i ≥z 01 , the system determines that the data processing speed of the multimodal action robot at this time point is affected, which reduces the speed of its data processing and analysis, and needs to stop the control of the multimodal action robot and optimize its processor;

[0074] If i <z 01 ,The system determines that the data processing speed of the multimodal action robot at this time point is not affected, and its data processing and analysis speed is good, so there is no need to optimize its processor and it can continue to be used;

[0075] Through the above technical solution, this example transforms the data processing influence coefficient z of the multimodal action robot at the i-th time point into i and the preset influence coefficient threshold z 01 By comparing, it is possible to analyze whether the data processing speed of the multimodal action robot is affected at that point in time, and to make an accurate judgment on the speed of its data processing analysis based on the analysis results. By using this judgment method, when it is judged that the data processing speed of the multimodal action robot is low, the control can be stopped in time and its processor can be optimized to avoid the situation where the multimodal action robot cannot move normally or even collides with obstacles due to the slow analysis speed.

[0076] The analysis process of the identification and analysis module includes:

[0077] By formula Calculate the driving risk index r of the multimodal mobile robot at the i-th time point i ;

[0078] Among them, ps i is the road damage area in the image taken by the multimodal mobile robot at the i-th time point, lf i is the road crack area in the image taken by the multimodal mobile robot at the i-th time point, zmj i The total area of ​​the road in the image captured by the multimodal mobile robot at the i-th time point, sr i is the road damage depth in the image taken by the multimodal mobile robot at the i-th time point, sr y is the preset depth, sd i is the moving speed of the multimodal mobile robot at the i-th time point, sd 01 sd iThe standard value can be selected and set according to the allowable error in the empirical data. i is the number of obstacles in the image taken by the multimodal mobile robot at the i-th time point, za i is the preset number of roadblocks, b for za i The standard value of f can be selected and set according to the allowable error in the empirical data. v For the second defined function, if f c (x)≥1, let f c (x) = x, otherwise, let f c (x) = 0, x1 and x2 are weight coefficients, set according to empirical fitting;

[0079] Through the above technical solution, this embodiment provides the driving risk index r of the multimodal mobile robot at the i-th time point: i , can be obtained by formula It is obvious that when the road crack area, damaged area and damaged depth in the image taken at the i-th time point are larger, and the speed of the multimodal mobile robot is faster and the number of roadblocks is greater, the driving risk index r of the multimodal mobile robot at the i-th time point is i The higher the value, the higher the risk of continuing to drive on the current route. On the contrary, when the road crack area, damaged area and damaged depth in the image taken at the i-th time point are smaller, and the speed of the multimodal mobile robot is slower, the number of roadblocks is 0, then the driving risk index r of the multimodal mobile robot at the i-th time point is i The lower it is, the lower the risk of continuing on the current route. This calculation method reflects the risk factor of the multimodal mobile robot's current route, so that the system decides whether to change the route and provide accurate data to avoid the multimodal mobile robot being unable to move normally or colliding with obstacles on a high-risk route.

[0080] The analysis process of the identification and analysis module also includes:

[0081] By transforming the driving risk index r of the multimodal mobile robot at the i-th time point into i The preset risk index threshold r 01 Make a comparison;

[0082] If r i ≥r 01 ,The system determines that the driving risk of the multimodal action robot is high at this time point, and needs to stop moving and adjust the route;

[0083] If r i <r 01, the system determines that the driving risk of the multimodal mobile robot at this time point is low, which means that the current route is in good condition and there is no need to adjust the route;

[0084] Through the above technical solution, this example calculates the driving risk index r of the multimodal mobile robot at the i-th time point i The preset risk index threshold r 01 Through this comparison method, an accurate judgment can be made on the driving risk of the multimodal action robot at that point in time, and a decision can be made based on the judgment result whether the route needs to be adjusted, thereby helping the robot to make more accurate action plans and ensure that the robot completes the task safely and efficiently.

[0085] See also Figure 2 As shown, a method for controlling a mobile robot based on a multimodal model comprises:

[0086] S1: Collecting the operation status data of the multimodal action robot during the working process through the data acquisition module;

[0087] S2: Calculate the data processing influence coefficient of the multimodal action robot at different time points by combining the operation status data of the multimodal action robot through the data calculation module;

[0088] S3: Analyze the data processing speed of the multimodal action robot at the current time point by combining the data processing influence coefficient of the multimodal action robot at different time points through the data analysis module. If the processing speed is slow, proceed to step S4; otherwise, proceed to step S5.

[0089] S4: when the decision module determines that the data information processing speed of the multimodal action robot at the current time point is slow, the control of the multimodal action robot is stopped and optimized;

[0090] S5: When it is determined through the image modality acquisition module that the data information processing speed of the multimodal mobile robot at the current time point is fast, the multimodal mobile robot collects road image data of the current driving route and extracts necessary feature points;

[0091] S6: Analyze the driving risk of the multimodal mobile robot at the current time point by combining the feature point data extracted by the recognition and analysis module with the image modality acquisition module;

[0092] Through the above technical scheme, this embodiment provides a method for controlling a mobile robot based on a multimodal model, firstly, the operation status data of the multimodal mobile robot in the working process is collected through the data acquisition module, and then the data processing influence coefficient of the multimodal mobile robot at different time points is calculated through the data calculation module in combination with the operation status data of the multimodal mobile robot, and the data analysis module is combined with the data processing influence coefficient of the multimodal mobile robot at different time points to analyze the speed of data information processing of the multimodal mobile robot at the current time point, and when it is judged that the data information processing speed of the multimodal mobile robot at the current time point is slow, the control of the multimodal mobile robot is stopped and optimized, and when the data information processing speed at the current time point is fast, the road image data of the current driving route is collected by the multimodal mobile robot, and necessary feature points are extracted, and finally the driving risk of the multimodal mobile robot at the current time point is analyzed by the recognition and analysis module in combination with the feature point data extracted by the image modality acquisition module;

[0093] Through such a setting, by combining the operating status data of the multimodal motion robot during work, it is possible to make an accurate judgment on the data processing influence coefficient of the multimodal motion robot at different time points, and formulate different follow-up plans based on the judgment, thereby avoiding the situation where the multimodal motion robot drives to the monitored road section before obtaining the analysis result due to poor road conditions and slow data analysis speed of the multimodal motion robot, and then cannot move normally or even collides with obstacles, thereby improving the work efficiency of the multimodal motion robot.

[0094] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A mobile robot control system based on a multimodal model, characterized in that: The system comprises: A data acquisition module is used to collect the operating status data of the multimodal action robot during its working process; A data calculation module, used to calculate the data processing influence coefficient of the multimodal action robot at different time points in combination with the operation status data of the multimodal action robot; A data analysis module, used to analyze the data information processing speed of the multimodal action robot at the current time point by combining the data processing influence coefficient of the multimodal action robot at different time points; A decision-making module, used to stop controlling the multimodal action robot and perform optimization when it is determined that the data information processing speed of the multimodal action robot at the current time point is slow; The image modality acquisition module is used to collect the road image data of the current driving route through the multimodal action robot and extract the necessary feature points when it is determined that the data information processing speed of the multimodal action robot at the current time point is fast; The recognition and analysis module is used to analyze the driving risk of the multimodal action robot at the current time point in combination with the feature point data extracted by the image modality acquisition module.

2. The mobile robot control system based on a multimodal model according to claim 1, characterized in that: The data collected by the data acquisition module include the multimodal mobile robot processor operation data, equipment data and shooting data in historical operations, among which the processor operation data includes the processor's read and write speed, response speed, memory size and cache size at different time points, and the processor equipment data includes the processor's temperature and voltage intensity.

3. The mobile robot control system based on a multimodal model according to claim 1, characterized in that: The calculation process of the data calculation module includes: The data information collected in real time by the data acquisition module is used to establish the memory change curve θ(t) of the multi-modal action robot processor. By formula Calculate the processor speed influence coefficient δ of the multimodal mobile robot at the i-th time point i ; Among them, i is the time point of any data collection after the multimodal mobile robot starts working, dx i is the read and write speed of the multimodal mobile robot processor at the i-th time point, dx y is the preset read and write speed, xy i is the response speed of the multimodal mobile robot processor at the i-th time point, xy y is the preset response speed, dx 01 With xy 01 dx i With xy i The standard value, f c (x) is the first defined function, if f c (x)≥1, let f c (x) = x, otherwise, let f c (x) = 1, t i is the time point of the i-th data collection, t1 is the time point when the multimodal action robot starts working, hc i is the cache size of the multimodal mobile robot processor at the i-th time point, hc y The default cache size.

4. The mobile robot control system based on a multimodal model according to claim 3, characterized in that: The calculation process of the data calculation module also includes: By formula Calculate the data processing influence coefficient z of the multimodal mobile robot at the i-th time point i ; Among them, cl i is the processor temperature of the multimodal mobile robot at the i-th time point, cl y is the preset processor temperature, sw i is the room temperature at the i-th time point, sw y is the preset room temperature, ρ is the error influence coefficient, dy i is the voltage intensity of the multimodal mobile robot at the i-th time point, dy y is the preset voltage intensity, dy 01 For i The standard value of tx, a is any image taken by the mobile robot in the past work, b is the total number of images taken by the mobile robot in the past work, a is the data size of the ath image taken by the mobile robot in the past work, For all tx a The average value of .

5. The mobile robot control system based on a multimodal model according to claim 4, characterized in that: The analysis process of the data analysis module includes: By transforming the data processing influence coefficient z of the multimodal action robot at the i-th time point i and the preset influence coefficient threshold z 01 Make a comparison; If i ≥z 01 , the system determines that the data processing speed of the multimodal action robot at this time point is affected, which reduces the speed of its data processing and analysis, and needs to stop the control of the multimodal action robot and optimize its processor; If i <z 01 ,The system determines that the data processing speed of the multimodal action robot at this time point is not affected, and its data processing and analysis speed is good. There is no need to optimize its processor and it can continue to be used.

6. The mobile robot control system based on a multimodal model according to claim 5, characterized in that: The analysis process of the identification and analysis module includes: By formula Calculate the driving risk index r of the multimodal mobile robot at the i-th time point i ; Among them, ps i is the road damage area in the image taken by the multimodal mobile robot at the i-th time point, lf i is the road crack area in the image taken by the multimodal mobile robot at the i-th time point, zmj i The total area of ​​the road in the image captured by the multimodal mobile robot at the i-th time point, sr i is the road damage depth in the image taken by the multimodal mobile robot at the i-th time point, sr y is the preset depth, sd i is the moving speed of the multimodal mobile robot at the i-th time point, sd 01 sd i The standard value of i is the number of obstacles in the image taken by the multimodal mobile robot at the i-th time point, za i is the preset number of roadblocks, b for za i The standard value, f v For the second defined function, if f c (x)≥1, let f c (x) = x, otherwise, let f c (x)=0, x1 and x2 are weight coefficients.

7. The mobile robot control system based on a multimodal model according to claim 6, characterized in that: The analysis process of the identification and analysis module also includes: By transforming the driving risk index r of the multimodal mobile robot at the i-th time point into i The preset risk index threshold r 01 Make a comparison; If r i ≥r 01 ,The system determines that the driving risk of the multimodal action robot is high at this time point, and needs to stop moving and adjust the route; If r i <r 01 ,The system determines that the driving risk of the multimodal mobile robot at this time point is low, which means that the current route is in good condition and there is no need to adjust the route.

8. A method for controlling a mobile robot based on a multimodal model, the method adopting a mobile robot control system based on a multimodal model as claimed in claims 1 to 7, characterized in that: The method comprises: S1: Collecting the operation status data of the multimodal action robot during the working process through the data acquisition module; S2: Calculate the data processing influence coefficient of the multimodal action robot at different time points by combining the operation status data of the multimodal action robot through the data calculation module; S3: Analyze the data processing speed of the multimodal action robot at the current time point by combining the data processing influence coefficient of the multimodal action robot at different time points through the data analysis module. If the processing speed is slow, proceed to step S4; otherwise, proceed to step S5. S4: when the decision module determines that the data information processing speed of the multimodal action robot at the current time point is slow, the control of the multimodal action robot is stopped and optimized; S5: When it is determined through the image modality acquisition module that the data information processing speed of the multimodal mobile robot at the current time point is fast, the multimodal mobile robot collects road image data of the current driving route and extracts necessary feature points; S6: Analyze the driving risk of the multimodal mobile robot at the current time point by combining the recognition and analysis module with the feature point data extracted by the image modality acquisition module.

Images