Inspection robot control method and device, computer device and storage medium

Abstract

The application relates to a method and device for controlling an inspection robot, computer equipment and a storage medium. The method comprises the following steps: determining historical visual dynamic tokens and historical context tokens according to historical video frames corresponding to historical inspection videos through an image tokenizer; generating a future scene token sequence corresponding to a future inspection scene according to the historical visual dynamic tokens and the historical context tokens through a target collaborative prediction model; determining a future action token sequence of the inspection robot according to the future scene token sequence, the historical visual dynamic tokens and an inspection task instruction through the target collaborative prediction model; converting the future action token sequence into a motion control instruction of the inspection robot, and controlling the inspection robot to perform the inspection task based on the motion control instruction. The above scheme improves the autonomous decision-making capability, environmental adaptability and task execution accuracy of the inspection operation.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to methods, devices, computer equipment, and storage media for controlling inspection robots. Background Technology

[0002] Currently, the autonomous navigation of inspection robots mainly adopts a technical solution that combines multiple sensors such as LiDAR and depth cameras with SLAM (Simultaneous Localization and Mapping). The current visual navigation methods for inspection robots generally separate environmental perception and path planning into two independent modules that are executed in series. First, the perception module completes obstacle recognition, and then the planning module calculates the driving path.

[0003] In existing visual navigation technologies for inspection robots, multi-sensor fusion solutions have high hardware costs and poor adaptability to dynamic environments. Decoupling perception and planning can lead to problems such as latency and error accumulation, lack of forward-looking prediction capabilities, and high data annotation costs. Even when adopting the world model paradigm, it is only used as a simulation data engine or a simple unified architecture, without achieving deep collaboration between prediction and planning at the knowledge level, making it difficult to fully improve the safety and robustness of inspection route planning.

[0004] Improving the accuracy and safety of visual navigation for inspection robots in diverse scenarios, while reducing the cost of visual navigation for inspection robots, is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide a control method, device, and computer equipment for inspection robots that can improve the accuracy and safety of visual navigation in diverse scenarios and reduce the cost of visual navigation for inspection robots, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a method for controlling an inspection robot, the method comprising:

[0007] The image tokenizer determines the historical visual dynamic token and historical context token based on the historical video frames corresponding to the historical inspection videos.

[0008] By using a target collaborative prediction model, a future scene token sequence corresponding to the future inspection scenario is generated based on historical visual dynamic tokens and historical context tokens.

[0009] The future action token sequence of the inspection robot is determined by the target collaborative prediction model based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions.

[0010] The future action token sequence is converted into motion control commands for the inspection robot, and the inspection robot is controlled to perform inspection tasks based on the motion control commands.

[0011] In one embodiment, an image tokenizer determines a historical visual dynamic token and a historical context token based on historical video frames corresponding to historical inspection videos, including:

[0012] By using the high-resolution context branch of the image tokenizer, a deep convolutional neural network is used to encode the first full-resolution image in the historical video frames corresponding to the historical inspection video, thereby determining the high-resolution image features and historical context tokens.

[0013] By using the low-resolution dynamic branch of the image tokenizer, feature extraction is performed on the downsampled images of the historical video frames corresponding to the historical inspection videos to determine the low-resolution image features.

[0014] The high-resolution image features are used as keys and values ​​for cross-attention calculation, and the low-resolution image features are used as query vectors for cross-attention calculation. Cross-attention calculation is performed based on the keys, values, and query vectors to determine the enhanced features.

[0015] Based on the enhanced features, historical visual dynamic tokens are determined.

[0016] In one embodiment, a target collaborative prediction model generates a future scene token sequence corresponding to the future inspection scene based on historical visual dynamic tokens and historical context tokens, including:

[0017] Based on the timestamps corresponding to the historical token sequences, the historical visual dynamic tokens are arranged to determine the historical visual token sequences.

[0018] The historical visual token sequence and historical context token are used as precondition information;

[0019] By using a target collaborative prediction model, autoregressive prediction is performed based on precondition information to determine the dynamic token for the future inspection scenario within a future time period.

[0020] In one embodiment, a target collaborative prediction model determines the future action token sequence of the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions, including:

[0021] Obtain the inspection task instruction, and perform vectorization processing on the inspection task instruction to determine the task instruction embedded token;

[0022] Using a target collaborative prediction model, a future scene token sequence is used as context information. An action generation process is triggered based on a pre-set target start token. After the action generation process is triggered, a future action token sequence is generated based on the context information, the task instruction embedded token, and the historical visual dynamic token.

[0023] In one embodiment, the future action token sequence is converted into motion control commands for the inspection robot, and the inspection robot is controlled to perform inspection tasks based on the motion control commands, including:

[0024] A lightweight fully connected neural network is used to determine the target pose data of the inspection robot based on the future action token sequence and the mapping relationship between action symbols and spatial positions.

[0025] The target pose data is converted into linear velocity and angular velocity commands using a robot motion model, and these commands are used as motion control commands for the inspection robot.

[0026] The motion control commands are sent to the bottom-level controller of the inspection robot so that the inspection robot can be controlled to perform inspection tasks through the bottom-level controller.

[0027] In one embodiment, the above-mentioned inspection robot control method further includes:

[0028] The first sample inspection scene video is acquired, and the first sample training scene video is divided into first sample short video segments. The first sample image frame corresponding to the first sample short video segment is converted into a first sample visual token sequence through an image tokenizer.

[0029] Based on the first sample visual token sequence, unsupervised model training is performed on the collaborative prediction model to be trained to determine the pre-trained collaborative prediction model.

[0030] Based on the second sample visual token sequence, sample task instruction, and sample start token corresponding to the second sample inspection scene video, the pre-trained collaborative prediction model is trained to determine the target collaborative prediction model.

[0031] In one embodiment, the pre-trained collaborative prediction model is trained based on the second sample visual token sequence, sample task instructions, and sample start token corresponding to the second sample inspection scene video to determine the target collaborative prediction model, including:

[0032] Obtain the second sample visual token sequence corresponding to the second sample inspection scene video, and determine the sample future scene token based on the second sample visual token sequence using a pre-trained collaborative prediction model.

[0033] Based on the sample future scene token, the sample task instruction corresponding to the second sample inspection scene video, the sample start token, and the sample history token corresponding to the second sample inspection scene video, the predicted running trajectory of the inspection robot is determined.

[0034] Based on the sample future scene token and the actual future scene token corresponding to the second sample inspection scene video, determine the cross-entropy loss function.

[0035] The mean absolute error loss function is determined based on the predicted and actual operating trajectories.

[0036] The model parameters of the pre-trained collaborative prediction model are optimized based on the cross-entropy loss function and the mean absolute error loss function to determine the target collaborative prediction model.

[0037] Secondly, this application also provides a control device for an inspection robot, the device comprising:

[0038] The dynamic token generation module is used to determine the historical visual dynamic token and the historical context token based on the historical video frames corresponding to the historical inspection video through the image tokenizer.

[0039] The future scene token generation module is used to generate a future scene token sequence corresponding to the future inspection scene based on historical visual dynamic tokens and historical context tokens through a target collaborative prediction model.

[0040] The future action token generation module is used to determine the future action token sequence of the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions through a target collaborative prediction model.

[0041] The motion control command determination module is used to convert the future action token sequence into motion control commands for the inspection robot, and control the inspection robot to perform inspection tasks based on the motion control commands.

[0042] Thirdly, this application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0043] The image tokenizer determines the historical visual dynamic token and historical context token based on the historical video frames corresponding to the historical inspection videos.

[0044] By using a target collaborative prediction model, a future scene token sequence corresponding to the future inspection scenario is generated based on historical visual dynamic tokens and historical context tokens.

[0045] The future action token sequence of the inspection robot is determined by the target collaborative prediction model based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions.

[0046] The future action token sequence is converted into motion control commands for the inspection robot, and the inspection robot is controlled to perform inspection tasks based on the motion control commands.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0048] The image tokenizer determines the historical visual dynamic token and historical context token based on the historical video frames corresponding to the historical inspection videos.

[0049] By using a target collaborative prediction model, a future scene token sequence corresponding to the future inspection scenario is generated based on historical visual dynamic tokens and historical context tokens.

[0050] The future action token sequence of the inspection robot is determined by the target collaborative prediction model based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions.

[0051] The future action token sequence is converted into motion control commands for the inspection robot, and the inspection robot is controlled to perform inspection tasks based on the motion control commands.

[0052] The aforementioned inspection robot control method, device, and computer equipment, through an image tokenizer, determine historical visual dynamic tokens and historical context tokens based on historical video frames corresponding to historical inspection videos; through a target collaborative prediction model, generate a future scene token sequence corresponding to the future inspection scene based on the historical visual dynamic tokens and historical context tokens; through the target collaborative prediction model, determine a future action token sequence for the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions; convert the future action token sequence into motion control instructions for the inspection robot, and control the inspection robot to execute inspection tasks based on the motion control instructions. This solves the problems in existing inspection robot visual navigation technologies, such as high hardware costs and poor adaptability to dynamic environments with multi-sensor fusion schemes, latency and error accumulation due to decoupling perception and planning, lack of forward-looking prediction capabilities, and high data annotation costs. The above scheme utilizes an image tokenizer to encode features in historical inspection video frames, determining historical visual dynamic tokens and historical context tokens. This enables lightweight and structured processing of visual information, reducing data processing pressure. A target collaborative prediction model, combined with historical token information, predicts future inspection scene sequences. Simultaneously, it integrates inspection task instructions to accurately predict the future action token sequence of the inspection robot. Based on this future action token sequence, it determines the robot's motion control instructions, controlling the robot to execute the inspection task. This allows the inspection robot to perceive environmental changes in advance and make predictive action decisions tailored to task requirements, reducing the robot's visual navigation costs and improving its autonomous decision-making ability, environmental adaptability, and task execution accuracy during inspection operations. Attached Figure Description

[0053] Figure 1 This is an application environment diagram of the inspection robot control method in one embodiment;

[0054] Figure 2 This is a flowchart illustrating the control method for an inspection robot in one embodiment;

[0055] Figure 3 This is a flowchart illustrating the target collaborative prediction model training method in one embodiment;

[0056] Figure 4 This is a structural block diagram of the inspection robot control device in one embodiment;

[0057] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] The inspection robot control method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104, through an image tokenizer, determines historical visual dynamic tokens and historical context tokens based on historical video frames corresponding to historical inspection videos; through a target collaborative prediction model, it generates a future scene token sequence corresponding to the future inspection scene based on the historical visual dynamic tokens and historical context tokens; through the target collaborative prediction model, it determines a future action token sequence for the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions; it converts the future action token sequence into motion control instructions for the inspection robot, controls the inspection robot to execute the inspection task based on the motion control instructions, and sends feedback information on whether the inspection robot successfully executed the inspection task to terminal 102 via the communication network.

[0060] In one embodiment, such as Figure 2 As shown, a control method for an inspection robot is provided. This embodiment illustrates the method applied to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0061] S210. Using an image tokenizer, determine the historical visual dynamic token and the historical context token based on the historical video frames corresponding to the historical inspection videos.

[0062] The image tokenizer is an image encoding conversion component that converts visual images into a sequence of computationally computable visual tokens. Historical inspection video refers to video data collected by the inspection robot using camera equipment during inspection tasks throughout history. For example, historical inspection video could be inspection video of the ten seconds preceding the current moment.

[0063] It should be noted that the main hardware of the inspection robot uses a wheeled chassis capable of differential or omnidirectional movement and carrying onboard computing equipment. A monocular industrial camera is mounted horizontally in the center of the front end of the inspection robot. The camera should have a global shutter to avoid image distortion during high-speed movement.

[0064] Specifically, video frames are extracted from historical inspection videos to obtain continuous temporal historical video frames. An image token microphone is then used to encode these historical video frames into historical visual dynamic tokens. Simultaneously, the first video frame from the historical video frames is selected for high-resolution feature encoding to generate a historical context token.

[0065] For example, based on the historical video frames corresponding to the historical inspection videos, historical visual dynamic tokens and historical context tokens are determined, including:

[0066] S2101. Through the high-resolution context branch of the image tokenizer, a deep convolutional neural network is used to encode the first full-resolution image in the historical video frame corresponding to the historical inspection video, and to determine the high-resolution image features and historical context token.

[0067] The high-resolution context branch of the image tokenizer is a dedicated coding branch responsible for extracting the overall global view and static environmental information of the entire scene. It processes only the first high-resolution original image to generate historical context tokens. The first full-resolution image refers to the first complete high-resolution original image in the historical video frames without compression or downsampling. The parameters of the high-resolution context branch of the image tokenizer are fixed and do not require training or updates.

[0068] Specifically, a high-resolution context-specific encoding branch is pre-set inside the image tokenizer. The first full-resolution image in the historical video frame corresponding to the historical inspection video is selected as the input of the deep convolutional neural network. The deep convolutional neural network completes the multi-level spatial feature extraction and encoding operation of the first full-resolution image. While fully preserving the global scene details and environmental layout information of the image, the high-resolution image features corresponding to the first full-resolution image are output. The high-resolution image features are then mapped to generate a historical context token that can be accessed for model operation, thereby completing the standardized encoding output of global environmental context information.

[0069] S2102. Through the low-resolution dynamic branch of the image tokenizer, feature extraction is performed on the downsampled image of the historical video frame corresponding to the historical inspection video to determine the low-resolution image features.

[0070] The low-resolution dynamic branch is a dedicated coding path in the image tokenizer that extracts temporal dynamic features after downsampling consecutive video frames. The low-resolution dynamic branch is a trainable component. The downsampled image is obtained by reducing the size and resolution of the original image corresponding to a historical video frame. In essence, downsampling historical video frames to obtain smaller images, and then feeding them into the low-resolution dynamic branch to extract temporal dynamic features, reduces computation and speeds up coding while focusing on inter-frame motion changes.

[0071] Specifically, through the low-resolution dynamic branch built into the image tokenizer, the original image corresponding to the historical video frame is first downsampled to compress the image size and resolution to obtain a lightweight downsampled image. Then, the feature extraction network built into the branch is used to extract features from the downsampled image, capture temporal dynamic information such as scene changes and target motion between frames, and output low-resolution image features that are adapted to the needs of temporal inference.

[0072] S2103. Use high-resolution image features as keys and values ​​for cross-attention calculation, and low-resolution image features as query vectors for cross-attention calculation. Perform cross-attention calculation based on the keys, values, and query vectors to determine the enhanced features.

[0073] Understandably, with the help of attention guidance, the encoder of the low-resolution branch is able to capture the core content of the image with very little information.

[0074] S2104. Based on the enhanced features, determine the historical visual dynamic token.

[0075] Specifically, the enhanced features are encoded into a 4×7 two-dimensional feature network by an encoder. This two-dimensional feature network is then flattened to form a one-dimensional feature sequence. This one-dimensional feature sequence is then vectorized using a codebook to determine the historical visual dynamic token. The codebook is a pre-built vector lookup table containing 8192 sets of standard feature vectors, essentially a standard vocabulary for visual features. Each feature in the two-dimensional feature grid is compared to the codebook, and the set of standard prototype vectors with the highest similarity is found. Using the codebook containing 8192 prototype vectors as a lookup table, similarity matching is performed position-by-position for each flattened feature in the one-dimensional feature sequence. The index number of the most fitting prototype vector is selected to replace the original feature data; this index number is the dynamic token, realizing the quantization encoding of continuous visual features into discrete digital tokens. The encoder is a network module used to compress, refine, and regularize image features into a fixed-format feature grid; it belongs to the feature encoding component of the image tokenizer.

[0076] The above scheme retains the complete global scene layout and fine environmental information of the first frame through a high-resolution context branch, generating accurate historical context tokens to provide a global benchmark for scene perception. At the same time, it compresses image data through a low-resolution dynamic branch, extracts image features from downsampled images, effectively reducing computational power consumption. Then, it uses cross-attention to achieve deep fusion of global environmental information and temporal dynamic features, giving dynamic features global scene prior constraints. The resulting historical visual dynamic tokens take into account the integrity of the overall environment while accurately highlighting local dynamic changes, which can improve the accuracy and scene adaptability of subsequent model environment inference, while balancing visual coding accuracy and overall computational efficiency.

[0077] S220. Using the target collaborative prediction model, generate a future scene token sequence corresponding to the future inspection scene based on historical visual dynamic tokens and historical context tokens.

[0078] Among them, the collaborative prediction model is a sequence prediction model based on the Transformer architecture that combines global scene information and temporal dynamic change information to jointly infer and predict future inspection visual scenes. The future scene token sequence is an ordered discrete token combination derived by the target collaborative prediction model based on historical visual tokens, used to represent the content and dynamic changes of subsequent inspection environment images.

[0079] Specifically, historical visual dynamic tokens and historical context tokens are input into the target collaborative prediction model. Based on the environmental patterns of the inspection scene and the correlation between dynamic changes between frames learned by the collaborative prediction model, and combined with global scene context information and temporal dynamic evolution characteristics, temporal autoregressive inference calculation is performed to predict and generate discrete visual tokens that conform to the scene development trend. Finally, they are sequentially spliced ​​to form a complete future scene token sequence corresponding to the future inspection scene.

[0080] For example, using a target collaborative prediction model, a future scene token sequence corresponding to the future inspection scenario is generated based on historical visual dynamic tokens and historical context tokens, including:

[0081] Based on the timestamps corresponding to the historical token sequences, the historical visual dynamic tokens are arranged to determine the historical visual token sequence; the historical visual token sequence and historical context tokens are used as precondition information; through the target collaborative prediction model, autoregressive prediction is performed based on the precondition information to determine the future scene dynamic tokens corresponding to the future inspection scenarios in the future time period.

[0082] The future time period can be the next ten seconds.

[0083] Specifically, based on the timestamps corresponding to the historical token sequences, the historical visual dynamic tokens are arranged in an orderly manner to form a time-series historical visual token sequence. The historical visual token sequence is then integrated with the historical context tokens to form complete precondition information. The precondition information is input into the target collaborative prediction model. The target collaborative prediction model performs autoregressive step-by-step deduction and prediction according to the time sequence rules based on the input information, and outputs the future scene dynamic token corresponding to the inspection scene in the future time period.

[0084] For example, the target collaborative prediction model first predicts future scene dynamic tokens within a certain time period based on historical visual dynamic tokens. This process is entirely based on learning the laws of the physical world and does not presuppose the robot's own action prediction. The above prediction process can be represented as: given a historical token sequence And historical context token Z1, the token sequence for the next n frames generated by the target collaborative prediction model. : It should be noted that during the autoregressive inference process of the target collaborative prediction model to generate future visual tokens for each frame, only previously generated frames are used as the basis for generation. No variables or embedded information representing the future actions of the inspection robot itself are introduced throughout the process. This ensures the objectivity and independent inference attributes of the scene prediction results. During the engineering implementation phase, when the target collaborative prediction model generates dynamic tokens for the future scene, it uses an attention mechanism to only retrieve historical observation tokens and already generated future visual tokens for information interaction and computation. It does not call up data related to action tokens that have not yet been generated, strictly decoupling scene environment prediction from robot motion planning.

[0085] The above solution can effectively improve the accuracy and timing rationality of dynamic token prediction in future scenarios, providing a reliable environmental prediction basis for subsequent robot action planning.

[0086] S230. Using a target collaborative prediction model, determine the future action token sequence of the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions.

[0087] Among them, the inspection task instruction refers to the predetermined operation execution instructions and task constraint information issued to the inspection robot. The inspection task instruction includes the inspection route, designated inspection points, inspection and detection items, operation priority, area inspection range, start and stop control instructions, special operation requirements, and task execution priority, which are used to define the robot's inspection operation objectives and execution standards.

[0088] Specifically, the future scene token sequence, the acquired historical visual dynamic tokens, and the inspection task instructions are input into the target collaborative prediction model. This allows the target collaborative prediction model to integrate future environment prediction information, past temporal visual information, and actual task requirements. Through the built-in attention interaction logic, it completes the deep fusion calculation of multiple types of information and outputs a future action token sequence for the inspection robot that adapts to scene changes and task requirements.

[0089] For example, using a target collaborative prediction model, based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions, the future action token sequence of the inspection robot is determined, including:

[0090] The inspection task instructions are obtained and vectorized to determine the task instruction embedding token. Using the target collaborative prediction model, the future scene token sequence is used as context information. Based on the pre-set target start token, the action generation process is triggered. After the action generation process is triggered, the future action token sequence is generated according to the context information, the task instruction embedding token, and the historical visual dynamic token.

[0091] The target initiation token is a predefined "planning start" token used to trigger the action generation phase. The model input process is divided into two parts: the first part contains various inputs such as historical visual information and task instructions; the second part, upon the appearance of the initiation token, formally switches the process state. As a trigger node, the initiation token enables the target-coordinated prediction model to perform the generation calculation of inspection action tokens.

[0092] Specifically, the inspection task instructions are acquired and vectorized to obtain standardized task instruction embedding tokens. Using the generated future scene token sequence as environmental context information, the target collaborative prediction model triggers the action generation process based on the preset target start token. After the action generation process is triggered, the target collaborative prediction model integrates environmental context information, task instruction embedding tokens, and historical visual dynamic tokens to complete multi-dimensional information fusion calculations, generating a future action token sequence for the inspection robot that adapts to the inspection scenario and operational requirements.

[0093] The above solution generates action tokens based on inspection task instructions, future scene token sequences, and historical visual dynamic tokens. This enables the inspection robot's inspection actions to adapt to changes in the inspection scene and the requirements of the task, improving the rationality, accuracy, and adaptability of the inspection action planning.

[0094] S240: Convert the future action token sequence into motion control instructions for the inspection robot, and control the inspection robot to perform inspection tasks based on the motion control instructions.

[0095] Specifically, the future action token sequence is decoded and analyzed to determine the standardized motion control commands that the inspection robot can recognize and execute. The motion control commands include specific control parameters such as the inspection robot's movement displacement, turning angle, travel speed, and stopping point. The inspection robot performs inspection tasks according to the preset path and work requirements based on the motion control commands.

[0096] For example, converting a future sequence of action tokens into motion control commands for an inspection robot, and controlling the inspection robot to perform inspection tasks based on the motion control commands, includes:

[0097] A lightweight, fully connected neural network is used to determine the target pose data of the inspection robot based on the future action token sequence and the mapping relationship between action symbols and spatial positions. The target pose data is converted into linear velocity and angular velocity commands through the robot motion model, and these commands are used as motion control commands for the inspection robot. The motion control commands are sent to the bottom controller of the inspection robot to control the robot to perform inspection tasks.

[0098] The mapping relationship between action symbols and spatial positions refers to the pre-constructed mapping data table and mapping rules between action token symbols and the spatial coordinates and posture angles of the inspection robot. The robot motion model is a kinematic state model established based on the robot's own motion posture, driving speed, and pose changes. It is used to characterize the correspondence between the robot's real-time pose, heading angle, driving distance, and motion control quantities under chassis drive control.

[0099] Specifically, a pre-trained lightweight fully connected neural network is used to determine the target pose data that the inspection robot needs to reach, based on a pre-established mapping relationship between action symbols and spatial positions, and a future sequence of action tokens. The robot's motion model is then invoked to convert the target pose data into linear velocity and angular velocity commands that can drive the inspection robot. These linear velocity and angular velocity commands serve as the robot's motion control commands. The motion control commands are then sent to the inspection robot's underlying controller, which drives the robot to perform its respective inspection tasks.

[0100] The above solution uses a lightweight fully connected neural network to determine the target pose data of the inspection robot, and then uses the robot motion model to convert the target pose data into motion control commands and send them to the underlying controller to achieve precise control of the inspection robot.

[0101] In the aforementioned inspection robot control method, an image tokenizer determines historical visual dynamic tokens and historical context tokens based on historical video frames corresponding to historical inspection videos. A target collaborative prediction model generates a future scene token sequence corresponding to the future inspection scene based on the historical visual dynamic tokens and historical context tokens. The target collaborative prediction model then determines a future action token sequence for the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions. This future action token sequence is converted into motion control instructions for the inspection robot, and the robot is controlled to execute inspection tasks based on these instructions. This method solves the problems in existing inspection robot visual navigation technologies, such as high hardware costs and poor adaptability to dynamic environments with multi-sensor fusion schemes, latency and error accumulation due to decoupling perception and planning, lack of forward-looking prediction capabilities, and high data annotation costs. The above scheme utilizes an image tokenizer to encode features in historical inspection video frames, determining historical visual dynamic tokens and historical context tokens. This enables lightweight and structured processing of visual information, reducing data processing pressure. A target collaborative prediction model, combined with historical token information, predicts future inspection scene sequences. Simultaneously, it integrates inspection task instructions to accurately predict the future action token sequence of the inspection robot. Based on this future action token sequence, it determines the robot's motion control instructions, controlling the robot to execute the inspection task. This allows the inspection robot to perceive environmental changes in advance and make predictive action decisions tailored to task requirements, reducing the robot's visual navigation costs and improving its autonomous decision-making ability, environmental adaptability, and task execution accuracy during inspection operations.

[0102] For example, such as Figure 3 As shown, the training methods for the target collaborative prediction model include:

[0103] S310. Obtain the first sample inspection scene video, and divide the first sample training scene video into first sample short video segments. Use an image tokenizer to convert the first sample image frame corresponding to the first sample short video segment into a first sample visual token sequence.

[0104] Among them, the first sample inspection scene video refers to the unlabeled monitoring video or robot inspection video collected from various inspection environments of the inspection robot. The first sample inspection scene video does not contain action and trajectory labeling information.

[0105] Specifically, the first sample inspection scene video is acquired and segmented according to a preset duration standard to obtain several short video clips. The first sample image frames corresponding to each short video clip are extracted and encoded using a pre-deployed image tokenizer, converting them into a visual token sequence, i.e., the first sample visual token sequence. This completes the tokenization preprocessing of the sample data, providing standardized sequence-type training data for subsequent model training.

[0106] S320. Based on the first sample visual token sequence, perform unsupervised model training on the collaborative prediction model to be trained, and determine the pre-trained collaborative prediction model.

[0107] For example, the first half of the visual token sequence is taken as the input data for model training. The collaborative prediction model to be trained generates the predicted second half of the visual token based on the first half of the visual token sequence. The model loss function is determined based on the predicted second half of the visual token and the actual second half of the first half of the visual token sequence. The model parameters of the collaborative prediction model to be trained are then adjusted based on the model loss function to complete the autoregressive training of the collaborative prediction model to be trained and determine the pre-trained collaborative prediction model.

[0108] S330. Based on the second sample visual token sequence, sample task instruction, and sample start token corresponding to the second sample inspection scene video, train the pre-trained collaborative prediction model to determine the target collaborative prediction model.

[0109] For example, the methods for determining the standard collaborative prediction model include:

[0110] The process involves obtaining the second sample visual token sequence corresponding to the second sample inspection scene video, and determining the sample future scene token based on the second sample visual token sequence using a pre-trained collaborative prediction model. Based on the sample future scene token, the sample task instruction corresponding to the second sample inspection scene video, the sample start token, and the sample history token corresponding to the second sample inspection scene video, the predicted running trajectory of the inspection robot is determined. Based on the sample future scene token and the actual future scene token corresponding to the second sample inspection scene video, a cross-entropy loss function is determined. Based on the predicted running trajectory and the actual running trajectory, a mean absolute error loss function is determined. Finally, the model parameters of the pre-trained collaborative prediction model are optimized based on the cross-entropy loss function and the mean absolute error loss function to determine the target collaborative prediction model.

[0111] Specifically, the second sample visual token sequence corresponding to the second sample inspection scene video is obtained. This second sample visual token sequence is then input into a pre-trained collaborative prediction model to obtain the sample future scene token. Finally, the pre-trained collaborative prediction model, based on the sample future scene token, the sample task instruction matching the inspection scene, the sample start token, and the sample history token, outputs the predicted running trajectory corresponding to the inspection robot through model computation.

[0112] Based on the sample future scene tokens predicted by the pre-trained collaborative prediction model and the actual future scene tokens corresponding to the inspection scene, the cross-entropy loss function is determined. Based on the predicted running trajectory output by the pre-trained collaborative prediction model and the actual running trajectory within the scene, the mean absolute error loss function is calculated.

[0113] Using the cross-entropy loss function and the mean absolute error loss function as model optimization constraints, the gradient is backpropagated and the network parameters of the pre-trained co-prediction model are iteratively adjusted. The weight parameters are continuously iteratively optimized until the pre-trained co-prediction model converges, thus obtaining the target co-prediction model.

[0114] Based on the above embodiments, the inspection robot control method includes:

[0115] A high-resolution context-specific encoding branch is pre-set inside the image tokenizer. The first full-resolution image in the historical video frame corresponding to the historical inspection video is selected as the input of the deep convolutional neural network. The deep convolutional neural network completes the multi-level spatial feature extraction and encoding operation of the first full-resolution image. While fully preserving the global scene details and environmental layout information of the image, the high-resolution image features corresponding to the first full-resolution image are output. The high-resolution image features are then mapped to generate a historical context token that can be accessed for model operation, thereby completing the standardized encoding output of global environmental context information.

[0116] The low-resolution dynamic branch built into the image tokenizer first performs downsampling processing on the original image corresponding to the historical video frame, compressing the image size and resolution to obtain a lightweight downsampled image. Then, the feature extraction network built into the branch is used to extract features from the downsampled image, capturing temporal dynamic information such as inter-frame scene changes and target motion, and outputting low-resolution image features that are adapted to the needs of temporal deduction.

[0117] High-resolution image features are used as keys and values ​​for cross-attention computation, and low-resolution image features are used as query vectors for cross-attention computation. Cross-attention computation is performed based on the keys, values, and query vectors to determine the enhanced features.

[0118] The enhanced features are encoded into a 4×7 two-dimensional feature network by an encoder. This two-dimensional feature network is then flattened to form a one-dimensional feature sequence. This one-dimensional feature sequence is then vectorized using a codebook to determine the historical visual dynamic token. The codebook is a pre-built vector lookup table containing 8192 sets of standard feature vectors, essentially a standard vocabulary for visual features. Each feature in the two-dimensional feature grid is compared to the codebook, and the set of standard prototype vectors with the highest similarity is found. Using the codebook containing 8192 prototype vectors as a lookup table, similarity matching is performed position-by-position for each flattened feature in the one-dimensional feature sequence. The index number of the most fitting prototype vector is selected to replace the original feature data; this index number is the dynamic token, realizing the quantization encoding of continuous visual features into discrete digital tokens. The encoder is a network module used to compress, refine, and regularize image features into a fixed-format feature grid; it is a feature encoding component within the image tokenizer.

[0119] The first sample inspection scene video is acquired and segmented according to a preset duration standard to obtain several short video clips. The first sample image frames corresponding to each short video clip are extracted and encoded using a pre-deployed image tokenizer, converting them into a visual token sequence, i.e., the first sample visual token sequence. This completes the tokenization preprocessing of the sample data, providing standardized sequence-type training data for subsequent model training. The first half of the first sample visual token sequence is used as the model training input data. The collaborative prediction model to be trained generates predicted second-half visual tokens based on the first half of the first sample visual token sequence. The model loss function is determined based on the predicted second-half visual tokens and the actual second-half first sample visual token sequence. The model parameters of the collaborative prediction model to be trained are then adjusted based on the model loss function, completing the autoregressive training of the collaborative prediction model to be trained and determining the pre-trained collaborative prediction model.

[0120] The second sample visual token sequence corresponding to the second sample inspection scene video is obtained. This sequence is then input into a pre-trained collaborative prediction model to obtain the sample future scene token. The pre-trained collaborative prediction model then uses the sample future scene token, the sample task instruction matching the inspection scene, the sample start token, and the sample history token to output the predicted running trajectory of the inspection robot. Based on the sample future scene token predicted by the pre-trained collaborative prediction model and the actual future scene token corresponding to the inspection scene, the cross-entropy loss function is determined. The mean absolute error loss function is calculated based on the predicted running trajectory output by the pre-trained collaborative prediction model and the actual running trajectory within the scene. Using the cross-entropy loss function and the mean absolute error loss function as model optimization constraints, gradients are backpropagated and the various network parameters of the pre-trained collaborative prediction model are iteratively adjusted. This continuous iterative optimization of the weight parameters continues until the pre-trained collaborative prediction model converges, yielding the target collaborative prediction model.

[0121] The collaborative prediction model is a sequence prediction model based on the Transformer architecture that combines global scene information with temporal dynamic change information to jointly infer and predict future inspection visual scenes.

[0122] Based on the timestamps corresponding to the historical token sequences, the historical visual dynamic tokens are arranged in an orderly manner to form a time-series historical visual token sequence. Then, the historical visual token sequence and the historical context tokens are integrated into complete precondition information. The precondition information is input into the target collaborative prediction model. The target collaborative prediction model performs autoregressive step-by-step inference and prediction according to the time sequence based on the input information, and outputs the future scene dynamic token corresponding to the inspection scene in the future time period.

[0123] The inspection task instructions are acquired and vectorized to obtain standardized task instruction embedding tokens. Using a target collaborative prediction model, a sequence of future scene tokens is generated as environmental context information. Based on a preset target start token, an action generation process is triggered. Once triggered, the target collaborative prediction model integrates environmental context information, task instruction embedding tokens, and historical visual dynamic tokens to perform multi-dimensional information fusion calculations, generating a future action token sequence for the inspection robot that adapts to the inspection scenario and operational requirements. The inspection robot's hardware consists of a wheeled chassis capable of differential or omnidirectional movement and carrying onboard computing equipment. A monocular industrial camera is mounted horizontally in the center of the front of the robot. The camera should have a global shutter to avoid image distortion during high-speed movement.

[0124] Using a pre-trained lightweight fully connected neural network, the target pose data that the inspection robot needs to reach is determined based on a pre-established mapping relationship between action symbols and spatial positions, as well as future action token sequences. The robot's motion model is then invoked to convert the target pose data into linear velocity and angular velocity commands that can drive the inspection robot. These linear velocity and angular velocity commands serve as the robot's motion control commands. These motion control commands are then sent to the inspection robot's underlying controller, which drives the robot to perform its respective inspection tasks.

[0125] For example, based on the above scheme, the target collaborative prediction model outputs the probability distribution of each token during the generation of future visual tokens. It performs prediction uncertainty detection for key areas such as the path ahead. If the prediction confidence is determined to be low, it is identified as an uncertain scenario, and the robot is then controlled to slow down or stop, and a conservative obstacle avoidance strategy such as rule-based edge detour is switched. On the other hand, before the trajectory command is sent to the controller, a rapid verification module is used to check the trajectory feasibility, verifying whether the trajectory points conflict with the static map and whether they exceed the robot's own physical movement limits. A complete visual navigation system for the inspection robot is built based on dual safety control. By using a deep learning collaborative architecture to integrate forward prediction and real-time planning, the robot can be effectively guaranteed to complete autonomous inspection operations in complex dynamic environments with low cost and high safety.

[0126] In the aforementioned inspection robot control method, an image tokenizer determines historical visual dynamic tokens and historical context tokens based on historical video frames corresponding to historical inspection videos. A target collaborative prediction model generates a future scene token sequence corresponding to the future inspection scene based on the historical visual dynamic tokens and historical context tokens. The target collaborative prediction model then determines a future action token sequence for the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions. This future action token sequence is converted into motion control instructions for the inspection robot, and the robot is controlled to execute inspection tasks based on these instructions. This method solves the problems in existing inspection robot visual navigation technologies, such as high hardware costs and poor adaptability to dynamic environments with multi-sensor fusion schemes, latency and error accumulation due to decoupling perception and planning, lack of forward-looking prediction capabilities, and high data annotation costs. The above scheme utilizes an image tokenizer to encode features in historical inspection video frames, determining historical visual dynamic tokens and historical context tokens. This enables lightweight and structured processing of visual information, reducing data processing pressure. A target collaborative prediction model, combined with historical token information, predicts future inspection scene sequences. Simultaneously, it integrates inspection task instructions to accurately predict the future action token sequence of the inspection robot. Based on this future action token sequence, it determines the robot's motion control instructions, controlling the robot to execute the inspection task. This allows the inspection robot to perceive environmental changes in advance and make predictive action decisions tailored to task requirements, reducing the robot's visual navigation costs and improving its autonomous decision-making ability, environmental adaptability, and task execution accuracy during inspection operations.

[0127] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0128] Based on the same inventive concept, this application also provides an inspection robot control device for implementing the inspection robot control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the inspection robot control device provided below can be found in the limitations of the inspection robot control method described above, and will not be repeated here.

[0129] In one embodiment, such as Figure 4 As shown, a control device for an inspection robot is provided, including: a dynamic token generation module 401, a future scene token generation module 402, a future action token generation module 403, and a motion control command determination module 404, wherein:

[0130] The dynamic token generation module 401 is used to determine the historical visual dynamic token and the historical context token based on the historical video frame corresponding to the historical inspection video through the image tokenizer.

[0131] The future scene token generation module 402 is used to generate a future scene token sequence corresponding to the future inspection scene based on the historical visual dynamic token and the historical context token through the target collaborative prediction model.

[0132] The future action token generation module 403 is used to determine the future action token sequence of the inspection robot based on the future scene token sequence, historical visual dynamic tokens and inspection task instructions through a target collaborative prediction model.

[0133] The motion control command determination module 404 is used to convert the future action token sequence into motion control commands for the inspection robot, and control the inspection robot to perform inspection tasks based on the motion control commands.

[0134] For example, the dynamic token generation module 401 is specifically used for:

[0135] By using the high-resolution context branch of the image tokenizer, a deep convolutional neural network is used to encode the first full-resolution image in the historical video frames corresponding to the historical inspection video, thereby determining the high-resolution image features and historical context tokens.

[0136] By using the low-resolution dynamic branch of the image tokenizer, feature extraction is performed on the downsampled images of the historical video frames corresponding to the historical inspection videos to determine the low-resolution image features.

[0137] The high-resolution image features are used as keys and values ​​for cross-attention calculation, and the low-resolution image features are used as query vectors for cross-attention calculation. Cross-attention calculation is performed based on the keys, values, and query vectors to determine the enhanced features.

[0138] Based on the enhanced features, historical visual dynamic tokens are determined.

[0139] For example, the future scenario token generation module 402 is specifically used for:

[0140] Based on the timestamps corresponding to the historical token sequences, the historical visual dynamic tokens are arranged to determine the historical visual token sequences.

[0141] The historical visual token sequence and historical context token are used as precondition information;

[0142] By using a target collaborative prediction model, autoregressive prediction is performed based on precondition information to determine the dynamic token for the future inspection scenario within a future time period.

[0143] For example, the future action token generation module 403 is specifically used for:

[0144] Obtain the inspection task instruction, and perform vectorization processing on the inspection task instruction to determine the task instruction embedded token;

[0145] Using a target collaborative prediction model, a future scene token sequence is used as context information. An action generation process is triggered based on a pre-set target starter token. After the action generation process is triggered, a future action token sequence is generated based on the context information, the task instruction embedded token, and the historical visual dynamic token.

[0146] For example, the motion control command determination module 404 is specifically used for:

[0147] A lightweight fully connected neural network is used to determine the target pose data of the inspection robot based on the future action token sequence and the mapping relationship between action symbols and spatial positions.

[0148] The target pose data is converted into linear velocity and angular velocity commands using a robot motion model, and these commands are used as motion control commands for the inspection robot.

[0149] The motion control commands are sent to the bottom-level controller of the inspection robot so that the inspection robot can be controlled to perform inspection tasks through the bottom-level controller.

[0150] For example, the above-mentioned inspection robot control device further includes a model training module, specifically used for:

[0151] The first sample inspection scene video is acquired, and the first sample training scene video is divided into first sample short video segments. The first sample image frame corresponding to the first sample short video segment is converted into a first sample visual token sequence through an image tokenizer.

[0152] Based on the first sample visual token sequence, unsupervised model training is performed on the collaborative prediction model to be trained to determine the pre-trained collaborative prediction model.

[0153] Based on the second sample visual token sequence, sample task instruction, and sample start token corresponding to the second sample inspection scene video, the pre-trained collaborative prediction model is trained to determine the target collaborative prediction model.

[0154] For example, the above-mentioned model training module is also specifically used for:

[0155] Obtain the second sample visual token sequence corresponding to the second sample inspection scene video, and determine the sample future scene token based on the second sample visual token sequence using a pre-trained collaborative prediction model.

[0156] Based on the sample future scene token, the sample task instruction corresponding to the second sample inspection scene video, the sample start token, and the sample history token corresponding to the second sample inspection scene video, the predicted running trajectory of the inspection robot is determined.

[0157] Based on the sample future scene token and the actual future scene token corresponding to the second sample inspection scene video, determine the cross-entropy loss function.

[0158] The mean absolute error loss function is determined based on the predicted and actual operating trajectories.

[0159] The model parameters of the pre-trained collaborative prediction model are optimized based on the cross-entropy loss function and the mean absolute error loss function to determine the target collaborative prediction model.

[0160] Each module in the aforementioned inspection robot control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0161] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method for controlling an inspection robot. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or a key vector, trackball, or touchpad set on the computer device casing, or an external key vector disk, touchpad, or mouse, etc.

[0162] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0163] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0164] Step 1: Using the image tokenizer, determine the historical visual dynamic token and the historical context token based on the historical video frames corresponding to the historical inspection videos;

[0165] Step 2: Using the target collaborative prediction model, generate a future scene token sequence corresponding to the future inspection scenario based on historical visual dynamic tokens and historical context tokens;

[0166] Step 3: Using the target collaborative prediction model, determine the future action token sequence of the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions;

[0167] Step 4: Convert the future action token sequence into motion control commands for the inspection robot, and control the inspection robot to perform inspection tasks based on the motion control commands.

[0168] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0169] Step 1: Using the image tokenizer, determine the historical visual dynamic token and the historical context token based on the historical video frames corresponding to the historical inspection videos;

[0170] Step 2: Using the target collaborative prediction model, generate a future scene token sequence corresponding to the future inspection scenario based on historical visual dynamic tokens and historical context tokens;

[0171] Step 3: Using the target collaborative prediction model, determine the future action token sequence of the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions;

[0172] Step 4: Convert the future action token sequence into motion control commands for the inspection robot, and control the inspection robot to perform inspection tasks based on the motion control commands.

[0173] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0174] Step 1: Using the image tokenizer, determine the historical visual dynamic token and the historical context token based on the historical video frames corresponding to the historical inspection videos;

[0175] Step 2: Using the target collaborative prediction model, generate a future scene token sequence corresponding to the future inspection scenario based on historical visual dynamic tokens and historical context tokens;

[0176] Step 3: Using the target collaborative prediction model, determine the future action token sequence of the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions;

[0177] Step 4: Convert the future action token sequence into motion control commands for the inspection robot, and control the inspection robot to perform inspection tasks based on the motion control commands.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0179] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0180] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A control method for an inspection robot, characterized in that, include: The image tokenizer determines the historical visual dynamic token and historical context token based on the historical video frames corresponding to the historical inspection videos. By using a target collaborative prediction model, a future scene token sequence corresponding to the future inspection scenario is generated based on historical visual dynamic tokens and historical context tokens. The future action token sequence of the inspection robot is determined by the target collaborative prediction model based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions. The future action token sequence is converted into motion control commands for the inspection robot, and the inspection robot is controlled to perform inspection tasks based on the motion control commands.

2. The method according to claim 1, characterized in that, The image tokenizer determines historical visual dynamic tokens and historical context tokens based on the historical video frames corresponding to the historical inspection videos, including: By using the high-resolution context branch of the image tokenizer, a deep convolutional neural network is used to encode the first full-resolution image in the historical video frames corresponding to the historical inspection video, thereby determining the high-resolution image features and historical context tokens. By using the low-resolution dynamic branch of the image tokenizer, feature extraction is performed on the downsampled images of the historical video frames corresponding to the historical inspection videos to determine the low-resolution image features. The high-resolution image features are used as keys and values ​​for cross-attention calculation, and the low-resolution image features are used as query vectors for cross-attention calculation. Cross-attention calculation is performed based on the keys, values, and query vectors to determine the enhanced features. Based on the enhanced features, historical visual dynamic tokens are determined.

3. The method according to claim 1, characterized in that, Using a target collaborative prediction model, based on historical visual dynamic tokens and historical context tokens, a future scene token sequence corresponding to the future inspection scenario is generated, including: Based on the timestamps corresponding to the historical token sequences, the historical visual dynamic tokens are arranged to determine the historical visual token sequences. The historical visual token sequence and historical context token are used as precondition information; By using a target collaborative prediction model, autoregressive prediction is performed based on precondition information to determine the dynamic token for the future inspection scenario within a future time period.

4. The method according to claim 1, characterized in that, Using a target collaborative prediction model, the future action token sequence of the inspection robot is determined based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions, including: Obtain the inspection task instruction, and perform vectorization processing on the inspection task instruction to determine the task instruction embedded token; Using a target collaborative prediction model, a future scene token sequence is used as context information. An action generation process is triggered based on a pre-set target starter token. After the action generation process is triggered, a future action token sequence is generated based on the context information, the task instruction embedded token, and the historical visual dynamic token.

5. The method according to claim 1, characterized in that, Converting the future action token sequence into motion control commands for the inspection robot, and controlling the inspection robot to perform inspection tasks based on the motion control commands, includes: A lightweight fully connected neural network is used to determine the target pose data of the inspection robot based on the future action token sequence and the mapping relationship between action symbols and spatial positions. The target pose data is converted into linear velocity and angular velocity commands using a robot motion model, and these commands are used as motion control commands for the inspection robot. The motion control commands are sent to the bottom-level controller of the inspection robot so that the inspection robot can be controlled to perform inspection tasks through the bottom-level controller.

6. The method according to claim 1, characterized in that, Also includes: The first sample inspection scene video is acquired, and the first sample training scene video is divided into first sample short video segments. The first sample image frame corresponding to the first sample short video segment is converted into a first sample visual token sequence through an image tokenizer. Based on the first sample visual token sequence, unsupervised model training is performed on the collaborative prediction model to be trained to determine the pre-trained collaborative prediction model. Based on the second sample visual token sequence, sample task instruction, and sample start token corresponding to the second sample inspection scene video, the pre-trained collaborative prediction model is trained to determine the target collaborative prediction model.

7. The method according to claim 6, characterized in that, Based on the second sample visual token sequence, sample task instructions, and sample start token corresponding to the second sample inspection scene video, the pre-trained collaborative prediction model is trained to determine the target collaborative prediction model, including: Obtain the second sample visual token sequence corresponding to the second sample inspection scene video, and determine the sample future scene token based on the second sample visual token sequence using a pre-trained collaborative prediction model. Based on the sample future scene token, the sample task instruction corresponding to the second sample inspection scene video, the sample start token, and the sample history token corresponding to the second sample inspection scene video, the predicted running trajectory of the inspection robot is determined. Based on the sample future scene token and the actual future scene token corresponding to the second sample inspection scene video, determine the cross-entropy loss function. The mean absolute error loss function is determined based on the predicted and actual operating trajectories. The model parameters of the pre-trained collaborative prediction model are optimized based on the cross-entropy loss function and the mean absolute error loss function to determine the target collaborative prediction model.

8. A control device for an inspection robot, characterized in that, The inspection robot control device includes: The dynamic token generation module is used to determine the historical visual dynamic token and the historical context token based on the historical video frames corresponding to the historical inspection video through the image tokenizer. The future scene token generation module is used to generate a future scene token sequence corresponding to the future inspection scene based on historical visual dynamic tokens and historical context tokens through a target collaborative prediction model. The future action token generation module is used to determine the future action token sequence of the inspection robot based on the future scene token sequence, historical visual dynamic tokens, and inspection task instructions through a target collaborative prediction model. The motion control command determination module is used to convert the future action token sequence into motion control commands for the inspection robot, and control the inspection robot to perform inspection tasks based on the motion control commands.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images