Robot inspection operation method and device and electronic equipment

Abstract

The invention provides a robot inspection operation method and device and electronic equipment, and the method comprises the steps: obtaining an inspection task, dividing the inspection task through a task distribution module based on a visual language large model in combination with a facility arrangement diagram of a current environment, generating a sub-task set, the robot meeting the task requirement is selected; for each sub-task in the sub-task set, generating a navigation instruction of the robot by combining the sub-task and video data acquired by a robot end in real time by a video language large model-based reasoning navigation module; a motion control module based on terrain and obstacle perception controls the robot to move to an operation position according to the navigation instruction; and the operation execution module based on the large model drives the robot to carry out operation according to the task content of the subtask. According to the invention, diversified track traffic facility inspection operation requirements can be met, and the problem of poor universality of robot inspection operation technology in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of robot inspection technology, and in particular to a robot inspection operation method, device and electronic equipment. Background Art

[0002] Rail transit has become an important mode of transportation for urban residents to commute, thanks to its large passenger capacity, convenience, comfort, and low per capita energy consumption. According to the "Urban Rail Transit 2023 Statistics and Analysis Report" released by the Urban Rail Transit Association, the total length of rail transit operating lines has reached 11,224 kilometers, and the total length of lines under construction is 5,671 kilometers. However, the inspection of rail transit facilities is still mainly manual, which has the disadvantages of low efficiency and high labor intensity, and faces the contradiction between the labor shortage caused by the aging population and the inspection demand caused by the rapid growth of operating mileage.

[0003] Robot technology is gradually being applied to the rail transit industry due to its advantages of high inspection efficiency and stable operation quality. Chinese invention patents CN116533252A, CN117885117A, CN118274845A, and CN117523637A respectively disclose a control method for a high-speed railway passenger station inspection robot arm operating system, a remote-controlled overhead line inspection method and inspection robot, a subway station robot inspection system and inspection method, and a suspicious person detection method, system, and medium at a railway station based on robot inspection. The above inspection technology solutions are applied to a single rail transit facility and have the disadvantage of poor versatility. Developing robot technology for different rail transit facilities separately faces the problems of high R&D costs and difficult later maintenance.

[0004] There is currently no effective solution to the problem of poor versatility of robot inspection technology in existing related technologies. Summary of the invention

[0005] The present invention provides a robot inspection operation method, device and electronic equipment, which are used to solve the defect of poor versatility of robot inspection technology in the prior art.

[0006] In a first aspect, the present invention provides a robot inspection method, comprising: S101, obtaining an inspection task, dividing the inspection task based on the task allocation module of the visual language large model in combination with the facility layout diagram of the current environment, generating a subtask set, and selecting a robot that meets the task requirements; S102, for each subtask in the subtask set, the reasoning navigation module based on the video language large model combines the subtask and the video data collected in real time by the robot end to generate a navigation instruction for the robot; S103, a motion control module based on terrain and obstacle perception controls the robot to move to a working position according to the navigation instruction; S104, the operation execution module based on the large model drives the robot to perform the operation according to the task content of the subtask; S105, repeating steps S102 to S104 until all subtasks are completed.

[0007] According to a robot inspection operation method provided by the present invention, the facility layout diagram includes a station signal equipment plan layout diagram, a subway station floor diagram, a subway depot layout diagram, and a fire protection plan layout diagram.

[0008] According to a robot inspection operation method provided by the present invention, the inference navigation module based on the video language large model combines the subtask and the video data collected in real time by the robot end to generate the navigation instructions of the robot, including: Encode the video image frames collected in real time by the robot end into current video image frame tokens, and concatenate the current image frame tokens with historical image frame tokens to obtain an image frame token set; Obtaining a sign semantic token in a video image frame collected in real time by the robot end; A navigation instruction for the robot is generated according to the task instruction of the subtask, the image frame token set and the logo semantic token.

[0009] According to a robot inspection operation method provided by the present invention, the sign semantics include rail transit line signs, subway station guide signs, equipment text signs, and vehicle depot guide signs.

[0010] According to a robot inspection operation method provided by the present invention, a motion control module based on terrain and obstacle perception controls the robot to move to an operation position according to the navigation instruction, including: Planning an initial motion path according to the navigation instruction, loading a default flat ground motion strategy, and controlling the robot to start moving; Identifying the terrain type of the robot's current environment based on the front image data of the robot; Selecting and loading a corresponding motion strategy from a pre-built motion strategy library according to the terrain type of the robot's current environment; The robot is detected based on the RGBD image data in front of the robot to see if there are any obstacles in the movement path in front of the robot, and the movement path of the robot is replanned based on the obstacle information.

[0011] According to a robot inspection operation method provided by the present invention, a large-model-based operation execution module drives the robot to perform the operation according to the task content of the subtask, including: The RGBD image collected by the robot is obtained, and the perception network in the job execution module extracts the RGBD image features; according to the subtask instruction and the RGBD image features, the visual language model in the job execution module generates the job description text corresponding to the subtask; According to the pre-built job skill library, the job description text and the RGBD image features, the visual language action model in the job execution module plans the set of job sub-actions for the robot to complete this job; According to the job sub-action set, the corresponding job skill is loaded from the job skill library, and the sub-action set corresponding to the subtask is executed.

[0012] According to a robot inspection operation method provided by the present invention, if the visual language action large model fails to plan the operation sub-action set, or the visual language action large model determines that the robot fails to execute the sub-action set, then: The robot issues a request for manual intervention, and performs the remaining sub-action sets by manual teleoperation; The operation data during manual remote operation is recorded, and the operation skills of the robot are trained according to the operation data, and the operation skill library is updated.

[0013] According to a robot inspection operation method provided by the present invention, the operation skill of the robot is trained according to the operation data, and the operation skill library is updated, including: According to the sub-action set, a simulation training field including the robot and the operating equipment is constructed; Based on the operation data of the artificial teleoperation, using imitation learning training to obtain an initialization strategy for reinforcement learning training; Based on the reward function of the trained job skills, generate the reward function for this training; The robot is trained according to the simulation training ground, initialization strategy and reward function to obtain initial operation skills; The type and initial range of environmental parameters are selected according to the sub-action, and the robot performs the operation using the initial operation skill. If the operation is confirmed to be successful, the current type and range of environmental parameters are recorded; The robot is trained again according to the type and range of the environmental parameters to obtain final operating skills; The final job skill is added to the job skill library.

[0014] In a second aspect, the present invention further provides a robot inspection operation device, comprising: The task allocation sub-device is used to obtain the inspection task, divide the inspection task in combination with the facility layout diagram of the current environment, generate a sub-task set, and select a robot that meets the task requirements; An inference navigation sub-device, used to generate navigation instructions for the robot by combining the sub-tasks and video data collected in real time by the robot end; A motion control sub-device, used for controlling the robot to move to a working position according to the navigation instruction; The operation execution sub-device is used to drive the robot to perform the operation according to the task content of the sub-task based on the operation execution module of the large model.

[0015] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the robot inspection method as described in the first aspect above is implemented.

[0016] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot inspection method as described in the first aspect above.

[0017] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the robot inspection method as described in the first aspect above.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The robot inspection operation method provided by the present invention, by applying a basic large model, gives the robot inspection system the ability to understand and allocate tasks, understand scenarios, perform reasoning navigation, and continuously learn operating skills, so that it can meet the diverse inspection operation needs of rail transit facilities, and solves the problem of poor versatility of robot inspection technology in existing related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flow chart of the robot inspection operation method provided by the present invention; Figure 2 is a schematic diagram of a facility layout diagram in an embodiment of the present invention; Figure 3 is a schematic diagram of a method for generating robot navigation instructions in an embodiment of the present invention; Figure 4 is a schematic diagram of a method for controlling the motion of a robot in an embodiment of the present invention; Figure 5 is a schematic diagram of a method for driving a robot to perform inspection operations in an embodiment of the present invention; Figure 6 It is a schematic diagram of a flow chart of determining the execution status of sub-actions in a patrol inspection operation performed by a robot in an embodiment of the present invention; Figure 7 It is a structural block diagram of the robot inspection operation device provided by the present invention; Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The present invention provides a robot inspection operation method, which can be completed by the following robot inspection operation system for rail transit facilities, the system comprising: Inspection and centralized control system, robot, inference navigation module, motion control module, camera 1, operation execution module, camera 2, mobile communication device, navigation module, motion control module, camera 1, operation execution module, camera 2 and mobile communication device are installed on the robot, camera 1 is installed at the front end of the robot to provide video data for the inference navigation module, camera 2 is installed at the end of the operation execution module to collect RGBD image data of the operation target, the inference navigation module is used to generate robot navigation instructions, the motion control module is used to control the robot to move to the operation position according to the navigation instructions, the operation execution module is used to carry out inspection operations, and the mobile communication device is used to realize the communication function between the robot and the inspection and centralized control system; The motion control module is also equipped with a motion strategy library, which contains motion strategies for different terrains; The inspection and control system is also equipped with a task allocation module, a facility layout diagram module, a reinforcement learning module 1, an operation skill library, an imitation learning module, and a reinforcement learning module 2. The task allocation module generates a corresponding subtask set according to the inspection task and selects a robot that meets the task requirements. The facility layout diagram module stores the facility layout diagram. The reinforcement learning module 1 trains the motion strategy library of the motion control module for motion strategies for different terrains. The operation skill library contains the robot operation skills trained by the imitation learning module and the reinforcement learning module 2. The imitation learning module and the reinforcement learning module 2 train the corresponding operation skills according to the manual remote operation data and store them in the operation skill library.

[0023] Figure 1 is a flow chart of the robot inspection method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step S101, obtaining inspection tasks, dividing the inspection tasks based on the task allocation module of the visual language large model in combination with the facility layout diagram of the current environment, generating a subtask set, and selecting a robot that meets the task requirements; Step S102, for each subtask in the subtask set, the inference navigation module based on the video language large model combines the subtask and the video data collected in real time by the robot end to generate navigation instructions for the robot; Step S103, the motion control module based on terrain and obstacle perception controls the robot to move to the working position according to the navigation instruction; Step S104, the operation execution module based on the large model drives the robot to perform the operation according to the task content of the subtask; Step S105, repeating steps S102 to S104 until all subtasks are completed.

[0024] In this method, first, the inspection tasks of this inspection are obtained, and the visual language model is combined with the facility layout diagram of this inspection environment to divide the inspection tasks into several subtasks. In the division process, it is necessary not only to understand the task content of the inspection tasks, but also to understand the scene of this inspection environment, so as to reasonably generate these subtasks and construct a subtask set. E , and select the corresponding robot. Then, for each subtask, the inference navigation module based on the video language large model combines the task content of the subtask and the video data collected by the robot to generate the navigation instructions of the robot. The motion control module based on terrain and obstacle perception controls the robot to move to the work position where the subtask needs to be performed according to the navigation instructions. Finally, the work execution module based on the large model drives the robot to perform the work according to the task content of the subtask. Since each task set EThere may be multiple subtasks in the system, so when executing this method, the above steps S102 to S104 can be repeatedly executed in the order of the subtasks until all subtasks are completed. In the above process, by applying the basic large model, the robot inspection system is endowed with the ability of task understanding and allocation, scene understanding, reasoning navigation, and continuous learning of operation skills, so that it can meet the diverse needs of rail transit facility inspection operations, and solve the problem of poor versatility of robot inspection technology in existing related technologies.

[0025] In the above method, the inspection task format is: “Rail transit facility type inspection location operation type” Among them, "Railway Transit Facility Type" is any type of rail transit facility, "Inspection Location" specifies the location where the robot operates in the rail transit facility, and "Inspection Operation" specifies the type of operation performed at the "Inspection Location". For example, "Fire Protection Facility Inspection on the Waiting Floor of a Subway Station" means to conduct inspections on the fire protection facilities on the waiting floor of a subway station. Figure 2 is a schematic diagram of a facility layout diagram in an embodiment of the present invention, combined with Figure 2 The fire protection facility layout diagram shown divides this inspection task into a set of subtasks {Inspection of fire-fighting facilities and fire-fighting 01, inspection of fire-fighting facilities and fire-fighting 02, inspection of fire-fighting facilities and fire-fighting 03, inspection of fire-fighting facilities and fire-fighting 04, inspection of fire-fighting facilities and fire-fighting 05, inspection of fire-fighting facilities and fire-fighting 06}.

[0026] In some embodiments, rail transit facilities include rail transit lines, subway stations, traction substations, subway depots and other rail transit infrastructure. Correspondingly, the facility layout diagram includes a station signal equipment plan layout diagram, a subway station floor diagram, a subway depot layout diagram, and a fire protection plan layout diagram.

[0027] Figure 3 Schematic diagram of a method for generating robot navigation instructions in an embodiment of the present invention. Figure 3 As shown, in some of the embodiments, step S102, the reasoning navigation module based on the video language large model combines the subtasks and the video data collected in real time by the robot end to generate navigation instructions for the robot, including: encoding the video image frames collected in real time by the robot end into video image frame tokens, and splicing the video image frame tokens with the historical image frame tokens of the robot end to obtain an image frame token set; obtaining the landmark semantic tokens in the video image frames collected in real time by the robot end; generating navigation instructions for the robot according to the task instructions of the subtask, the image frame token set and the landmark semantic tokens.

[0028] In this embodiment, the sign semantics include rail transit line signs, subway station guide signs, equipment text signs, and vehicle depot guide signs.

[0029] This embodiment can be completed by calling the inference navigation module based on the video language model. For example, at time , enter the current subtask command into the navigation module and the current video image frame on the robot side , historical video image frame token The encoder module in the navigation module converts the current video image frame Encoded as video image frame tokens The text encoder in the navigation module converts the subtask instructions Encoded as instruction tokens , the scene text recognition module in the navigation module obtains the video image frame The sign meaning in The language model in the navigation module is based on the subtask instruction token , a symbolic token and image frame tokens , generate navigation instructions .

[0030] Figure 4 Schematic diagram of a method for controlling the motion of a robot in an embodiment of the present invention. Figure 4 As shown, in some embodiments, step S103, the motion control module based on terrain and obstacle perception controls the robot to move to the working position according to the navigation instruction, including: planning the initial motion path according to the navigation instruction, loading the default flat ground motion strategy, and controlling the robot to start moving; identifying the terrain category of the robot's current environment according to the robot's front image data; selecting and loading the motion strategy in the pre-built motion strategy library according to the terrain category of the robot's current environment; detecting whether there are obstacles in the robot's motion path according to the robot's front RGBD data, and re-planning the robot's motion path according to the obstacle information. The low-level motion controller outputs the target position of each joint of the robot according to the robot's front RGBD image and the selected motion strategy.

[0031] For example, at the time , according to the navigation instructions , the path planning module generates the initial motion path , load the default flat ground motion strategy , the robot starts to move. During the robot's movement, the terrain estimation module runs at a frequency of 2Hz to identify the terrain of the robot's current environment. Select and load motion strategies in the motion strategy library At the same time, the obstacle perception module runs at a frequency of 2Hz to detect whether there are obstacles on the front movement path. If there are obstacles, the path planning module will re-plan the movement path based on the obstacle information. The low-level motion controller includes Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Multilayer Perceptron (MLP) and ActorNetwork; CNN and LSTM encode the image data in front of the robot to obtain image features, MLP performs feature fusion on image features and motion path information, and finally the action network combines the current motion strategy and fusion features to output the target position of each joint of the robot at a frequency of 50Hz. The proportional-differential controller converts the joint target position into joint torque, thereby driving the robot to move along the planned path.

[0032] Furthermore, the motion strategy library is a robot motion strategy adapted to different terrains obtained through training with a reinforcement learning module. The reinforcement learning module trains the robot motion strategy in simulation training grounds with different terrains, including concrete floors, marble floors, sidewalks, stairs, slopes, ballastless tracks, ballasted tracks, snow, and grass.

[0033] Figure 5 Schematic diagram of a method for driving a robot to perform inspection operations in an embodiment of the present invention. Figure 5 As shown, in some of the embodiments, step S104, the job execution module based on the large model drives the robot to perform the job according to the task content of the subtask, including: obtaining the RGBD image data collected by the robot, and generating the job description text corresponding to the RGBD image data; planning the job sub-action set for the robot to complete this job according to the job description text and the pre-built job skill library; loading the corresponding job skills from the job skill library according to the job sub-action set, and executing the task content of the subtask. It is worth noting that the RGBD image data in this embodiment and the video / RGBD image data in the previous text come from cameras installed at different locations.

[0034] Exemplarily, first, the camera installed at the end of the robot's operating device collects RGBD image data, and the perception network in the job execution module extracts RGBD image features. Then, based on the subtask instructions and RGBD image features, the visual language model in the job execution module generates the job description text corresponding to the subtask. Next, based on the pre-built job skill library, job description text, and RGBD image features, the visual language action model plans the set of job sub-actions to complete this job. The robot's job execution device loads the corresponding job skills from the job skill library according to the job sub-action set, and executes the job sub-actions in sequence.

[0035] Based on the above embodiment, if the planning of the set of sub-actions of the operation fails, or the execution of the sub-actions fails, then: the robot issues a request for manual intervention, and the remaining sub-actions are executed by manual remote control; the operation data during manual remote control is recorded, and the robot's operation skills are trained based on the operation data, and the operation skill library is updated.

[0036] For example, Figure 6 FIG. 1 is a flow chart of determining the execution status of sub-actions during the inspection process of the robot in an embodiment of the present invention. Figure 6 As shown, the robot sends out a request for human intervention. After receiving the request, the person at the remote end uses the real-time image data transmitted back from the robot to remotely operate the robot to complete the task, and simultaneously records the person's remote operation data. The data is then used to train new robot operation skills and update the operation skill library.

[0037] Specifically, the robot's operating skills are trained according to the operating data, and the operating skill library is updated, including: constructing a simulation training field including the robot and operating equipment according to the sub-action set; using imitation learning training based on the operating data of manual teleoperation to obtain an initialization strategy for reinforcement learning training; generating a reward function for this training based on the reward function of the trained operating skills; training the robot according to the simulation training field, the initialization strategy and the reward function to obtain initial operating skills; selecting the type and initial range of environmental parameters according to the sub-actions, the robot using the initial operating skills to perform the operation, and recording the current type and range of environmental parameters when confirming that the operation is successful; training the robot again according to the type and range of environmental parameters to obtain final operating skills; and adding the final operating skills to the operating skill library.

[0038] Exemplarily, first, a simulation training ground including robots and operating equipment is constructed based on a set of sub-actions. Then, the artificial teleoperation data is input, and the imitation learning method learns the initialization strategy for reinforcement learning training; based on the reward function used by the trained operating skills, the visual language large model generates a reward function for training, and the reinforcement learning method combines the initialization strategy and reward function to train the robot in the simulation training ground to obtain the initial operating skills. The visual language large model selects the type and initial range of environmental parameters according to the sub-actions, and the robot uses the initial operating skills to perform the operation. If the operation is successful, the parameter is considered to be a valid parameter value, and the valid environmental parameter type and range are determined accordingly. According to the above-determined environmental parameter type and range, the robot is trained again to obtain the final operating skills to improve the robustness of the robot's operating skills. Finally, the final operating skills that have been trained are added to the operating skill library.

[0039] In summary, this method applies basic models such as the visual language large model in robot inspection operations. By pre-learning Internet-scale data, it has rich pre-learning knowledge and can realize inspection task allocation, scene understanding, reasoning navigation and operation planning, continuous autonomous learning and other functions in various rail transit scenarios, thereby giving robots the ability to carry out inspection operations in various rail transit facilities.

[0040] The present invention also provides a robot inspection operation device. The robot inspection operation device provided by the present invention is described below. The robot inspection operation device described below and the robot inspection operation method described above can be referenced to each other. Figure 7 is a structural block diagram of the robot inspection operation device provided by the present invention, such as Figure 7 As shown, the device comprises: The task allocation sub-device 701 is used to obtain the inspection task, divide the inspection task in combination with the facility layout diagram of the current environment, generate a sub-task set, and select a robot that meets the task requirements; The reasoning navigation sub-device 702 is used to generate navigation instructions for the robot by combining the sub-tasks and the video data collected in real time by the robot end; The motion control sub-device 703 is used to control the robot to move to the working position according to the navigation instruction; The job execution sub-device 704 is used to drive the robot to perform the job according to the task content of the sub-task based on the job execution module of the large model.

[0041] When the device is in use, first, the task allocation sub-device 701 obtains the inspection task of this inspection work, and divides the inspection task into several sub-tasks in combination with the facility layout diagram of the inspection environment. In the division process, it is necessary not only to understand the task content of the inspection task, but also to understand the scene of the inspection environment, so as to reasonably generate these sub-tasks and construct a sub-task set. E , and select a robot that meets the task requirements. Then, for each subtask, the reasoning navigation sub-device 702 combines the task content of the subtask and the video data collected in real time by the robot end to generate the navigation instructions of the robot. The motion control sub-device 703 then controls the robot to move to the working position where the subtask needs to be performed according to the navigation instructions. Finally, the job execution sub-device 704 drives the robot to perform the job according to the task content of the subtask. Since each subtask set contains multiple subtasks, when executing this method, the above steps are repeated in the order of the subtasks until all subtasks are completed. In the above process, by applying the basic large model, the device is endowed with the capabilities of task understanding and allocation, scene understanding, reasoning navigation, and continuous learning of operating skills, so that it can meet the diverse needs of rail transit facility inspection operations, and solve the problem of poor versatility of robot inspection technology in existing related technologies.

[0042] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 801, a communications interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communications interface 802 and the memory 803 communicate with each other via the communication bus 804. The processor 801 may call the logic instructions in the memory 803 to execute the robot inspection operation method, which includes: S101, obtaining inspection tasks, dividing the inspection tasks based on the task allocation module of the visual language large model and the facility layout diagram of the current environment, generating a subtask set, and selecting a robot that meets the task requirements; S102, for each subtask in the subtask set, the reasoning navigation module based on the video language large model combines the subtask and the video data collected in real time by the robot end to generate a navigation instruction for the robot; S103, a motion control module based on terrain and obstacle perception controls the robot to move to a working position according to the navigation instruction; S104, the operation execution module based on the large model drives the robot to perform the operation according to the task content of the subtask; S105, repeating steps S102 to S104 until all subtasks are completed.

[0043] In addition, the logic instructions in the above-mentioned memory 803 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0044] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the robot inspection operation method provided by the above methods, the method includes: S101, obtaining inspection tasks, dividing the inspection tasks based on the task allocation module of the visual language large model and the facility layout diagram of the current environment, generating a subtask set, and selecting a robot that meets the task requirements; S102, for each subtask in the subtask set, the reasoning navigation module based on the video language large model combines the subtask and the video data collected in real time by the robot end to generate a navigation instruction for the robot; S103, a motion control module based on terrain and obstacle perception controls the robot to move to a working position according to the navigation instruction; S104, the operation execution module based on the large model drives the robot to perform the operation according to the task content of the subtask; S105, repeating steps S102 to S104 until all subtasks are completed.

[0045] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the robot inspection operation method provided by the above methods, the method comprising: S101, obtaining inspection tasks, dividing the inspection tasks based on the task allocation module of the visual language large model and the facility layout diagram of the current environment, generating a subtask set, and selecting a robot that meets the task requirements; S102, for each subtask in the subtask set, the reasoning navigation module based on the video language large model combines the subtask and the video data collected in real time by the robot end to generate a navigation instruction for the robot; S103, a motion control module based on terrain and obstacle perception controls the robot to move to a working position according to the navigation instruction; S104, the operation execution module based on the large model drives the robot to perform the operation according to the task content of the subtask; S105, repeating steps S102 to S104 until all subtasks are completed.

[0046] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0047] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot inspection method, characterized in that: include: S101, obtaining an inspection task, dividing the inspection task based on the task allocation module of the visual language large model in combination with the facility layout diagram of the current environment, generating a subtask set, and selecting a robot that meets the task requirements; S102, for each subtask in the subtask set, the reasoning navigation module based on the video language large model combines the subtask and the video data collected in real time by the robot end to generate a navigation instruction for the robot; S103, a motion control module based on terrain and obstacle perception controls the robot to move to a working position according to the navigation instruction; S104, the operation execution module based on the large model drives the robot to perform the operation according to the task content of the subtask; S105, repeating steps S102 to S104 until all subtasks are completed.

2. The robot inspection method according to claim 1, characterized in that: The facility layout diagram includes a station signal equipment plan layout diagram, a subway station floor diagram, a subway depot layout diagram, and a fire protection plan layout diagram.

3. The robot inspection method according to claim 1, characterized in that: The reasoning navigation module based on the video language large model combines the subtasks and the video data collected in real time by the robot to generate navigation instructions for the robot, including: Encode the video image frames collected in real time by the robot end into current video image frame tokens, and concatenate the current image frame tokens with historical image frame tokens to obtain an image frame token set; Obtaining a sign semantic token in a video image frame collected in real time by the robot end; A navigation instruction for the robot is generated according to the task instruction of the subtask, the image frame token set and the logo semantic token.

4. The robot inspection method according to claim 3, characterized in that: The sign semantics include rail transit line signs, subway station guide signs, equipment text signs, and vehicle depot guide signs.

5. The robot inspection method according to claim 1, characterized in that: The motion control module based on terrain and obstacle perception controls the robot to move to the working position according to the navigation instruction, including: Planning an initial motion path according to the navigation instruction, loading a default flat ground motion strategy, and controlling the robot to start moving; Identifying the terrain category of the robot's current environment based on the front image data of the robot; Selecting and loading a corresponding motion strategy from a pre-built motion strategy library according to the terrain type of the robot's current environment; The robot is detected based on the RGBD image data in front of the robot to see if there are any obstacles in the movement path in front of the robot, and the movement path of the robot is replanned based on the obstacle information.

6. The robot inspection method according to claim 1, characterized in that: The operation execution module based on the large model drives the robot to perform the operation according to the task content of the subtask, including: The RGBD image collected by the robot is obtained, and the perception network in the job execution module extracts the RGBD image features; according to the subtask instructions and the RGBD image features, the visual language model in the job execution module generates the job description text corresponding to the subtask; According to the pre-built job skill library, the job description text and the RGBD image features, the visual language action model in the job execution module plans the set of job sub-actions for the robot to complete this job; According to the job sub-action set, the corresponding job skill is loaded from the job skill library, and the sub-action set corresponding to the subtask is executed.

7. The robot inspection method according to claim 6, characterized in that: If the visual language action model fails to plan the set of sub-actions for the operation, or the visual language action model determines that the robot fails to execute the set of sub-actions, then: The robot issues a request for manual intervention, and performs the remaining sub-action sets by manual teleoperation; The operation data during manual remote operation is recorded, and the operation skills of the robot are trained according to the operation data, and the operation skill library is updated.

8. The robot inspection method according to claim 7, characterized in that: Training the robot's operating skills according to the operating data, and updating the operating skill library, including: According to the sub-action set, a simulation training field including the robot and the operating equipment is constructed; Based on the operation data of the artificial teleoperation, using imitation learning training to obtain an initialization strategy for reinforcement learning training; Based on the reward function of the trained job skills, generate the reward function for this training; The robot is trained according to the simulation training ground, initialization strategy and reward function to obtain initial operation skills; The type and initial range of environmental parameters are selected according to the sub-action, and the robot performs the operation using the initial operation skill. If the operation is confirmed to be successful, the current type and range of environmental parameters are recorded; The robot is trained again according to the type and range of the environmental parameters to obtain final operating skills; The final job skill is added to the job skill library.

9. A robot inspection device, characterized in that: include: The task allocation sub-device is used to obtain the inspection task, divide the inspection task in combination with the facility layout diagram of the current environment, generate a sub-task set, and select a robot that meets the task requirements; An inference navigation sub-device, used to generate navigation instructions for the robot by combining the sub-tasks and video data collected in real time by the robot end; A motion control sub-device, used for controlling the robot to move to a working position according to the navigation instruction; The operation execution sub-device is used to drive the robot to perform the operation according to the task content of the sub-task based on the operation execution module of the large model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the robot inspection method according to any one of claims 1 to 8 is implemented.

Images