Robot learning method, device, equipment and storage medium based on dual camera fusion

Abstract

The present application relates to the field of robot learning technology, and provides a robot learning method, device, equipment and storage medium based on dual camera fusion, the method comprising: in the process of the task execution subject performing the target task, the posture information of the task execution subject is obtained in real time, and the first image frame is obtained in real time through the first camera carried by the task execution subject, and it is determined whether the image frame includes preset image information; if not, the current posture information of the task execution subject is obtained based on the preset iterative closest point global optimization algorithm; the initial position information of the task execution subject in the second coordinate system is obtained through the preset second camera, and the initial position information is converted from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, so as to obtain the current position information of the task execution subject. The method can accurately obtain the posture information of the task execution subject.

Description

Technical Field

[0001] The present application relates to the field of robot learning technology, and in particular to a robot learning method, device, equipment and storage medium based on dual-camera fusion. Background Art

[0002] Imitation learning, as a cutting-edge technology in the field of robot learning, usually relies on high-quality expert demonstration data. However, how to collect such demonstration data efficiently and accurately remains a challenge.

[0003] Among traditional data collection methods, teleoperation is one of the commonly used methods. However, teleoperation has some inherent limitations, such as insufficient reaction time and control ability. In addition, due to the diversity of human operation styles and the human errors that may be introduced during teleoperation, the collected data often have uneven trajectories and lack consistency, which increases the difficulty of post-processing and analyzing data, making it difficult for imitation learning algorithms to extract general and stable strategies from them.

[0004] Although data collection in simulation environments is fast, the "sim2real" transfer problem becomes a major obstacle. Skills in simulation environments are difficult to apply directly to the real world because the lighting conditions, surface materials, and physical properties of the real world are often simplified or ignored in simulation environments, resulting in poor performance of models trained in simulation in real environments.

[0005] With the development of SLAM (Simultaneous Localization and Mapping) technology, environmental perception capabilities have been significantly improved. As a leader in visual SLAM algorithms, ORB_SLAM3 performs well in real-time positioning and map construction. However, ORB_SLAM3 is prone to losing tracking during initialization, fast movement, and close-range operations, limiting its application potential in complex dynamic environments. Summary of the invention

[0006] The present application provides a robot learning method, device, equipment and storage medium based on dual-camera fusion to solve the problems raised by the above background technology.

[0007] In a first aspect, the present application provides a robot learning method based on dual-camera fusion, comprising:

[0008] In the process of the task execution subject performing the target task, the posture information of the task execution subject is obtained in real time through a preset posture acquisition algorithm, and a first image frame is obtained in real time through a first camera carried by the task execution subject, and it is determined whether the image frame includes preset image information;

[0009] If not included, obtaining the current posture information of the task execution subject based on a preset iterative closest point global optimization algorithm;

[0010] Acquire the initial position information of the task execution subject in the second coordinate system through a preset second camera, and convert the initial position information from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, so as to obtain the current position information of the task execution subject; wherein the second camera is fixedly arranged, the first camera corresponds to the first coordinate system, and the second camera corresponds to the second coordinate system;

[0011] Determine that the posture information and the position information are current posture information of the task execution subject;

[0012] After the task execution subject finishes executing the target task, each acquired posture information is time-sequentially arranged to obtain posture data information, and each first image frame is time-sequentially arranged to obtain visual data information;

[0013] The control robot learns the process of the task execution subject performing the target task based on the posture data information and the visual data information.

[0014] In a possible implementation, the method for obtaining the preset image information includes:

[0015] Setting a plurality of marks in the execution area of ​​the target task; each mark is different from another;

[0016] When the task execution subject executes the target task from a starting position at a preset execution speed, acquiring a second image frame in real time through the first camera;

[0017] Determining whether there is a common mark in each of the second image frames;

[0018] If so, it is determined that the image information corresponding to the shared mark is the preset image information.

[0019] In a possible implementation, after determining whether there is a common mark in each of the second image frames, the method further includes:

[0020] If it does not exist, respectively counting the number of the second image frames corresponding to each of the marks, and arranging each of the marks in sequence based on the number of the second image frames corresponding to each of the marks, to obtain a mark sequence; wherein the number corresponding to each mark in the mark sequence decreases in sequence;

[0021] Deleting each second image frame in which a first mark in the mark sequence appears from all the second image frames, and determining the first mark as a first target mark;

[0022] Deleting each second image frame in which a second mark in the mark sequence appears from all remaining second image frames, and determining the second mark as a second target mark;

[0023] Iterate the steps of deleting each second image frame in which the first mark in the mark sequence appears from all the second image frames, and determining that the first mark is the first target mark, until all the second image frames are deleted;

[0024] The image information corresponding to each target mark is respectively determined to be the preset image information.

[0025] In a possible implementation, the obtaining the initial position information of the task execution subject in the second coordinate system through a preset second camera includes:

[0026] Mapping the second coordinate system to the execution area of ​​the target task; the coordinate origin of the second coordinate system is a designated point on the second camera;

[0027] The coordinate point of the designated point on the task execution subject in the second coordinate system is determined as the initial position information.

[0028] In a possible implementation, the method for acquiring the conversion relationship between the first coordinate system and the second coordinate system includes:

[0029] Before the task execution subject executes the target task, a first coordinate system is constructed with the position of the designated point of the task execution subject as the coordinate origin;

[0030] Constructing a second rectangular coordinate system with the position of the designated point of the second camera as the coordinate origin;

[0031] Setting a mark in the execution area of ​​the target task; the mark appears in the field of view of the first camera and the second camera;

[0032] Mapping the first coordinate system to the execution area of ​​the target task, and obtaining a first coordinate point of the center of the circumscribed circle of the mark in the first coordinate system;

[0033] Mapping the second coordinate system to the execution area of ​​the target task, and obtaining a second coordinate point of the center of the circumscribed circle of the mark in the second coordinate system;

[0034] The conversion relationship is generated based on the first coordinate point and the second coordinate point.

[0035] In a second aspect, the present application provides a robot learning device based on dual-camera fusion, comprising:

[0036] A first acquisition module is used to acquire the posture information of the task execution subject in real time through a preset posture acquisition algorithm during the process of the task execution subject performing the target task, and to acquire a first image frame in real time through a first camera carried by the task execution subject, and to determine whether the image frame includes preset image information;

[0037] A second acquisition module is used to acquire the current posture information of the task execution subject based on a preset iterative closest point global optimization algorithm if it is not included;

[0038] A conversion module, used for obtaining the initial position information of the task execution subject in the second coordinate system through a preset second camera, and converting the initial position information from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, so as to obtain the current position information of the task execution subject; wherein the second camera is fixedly arranged, the first camera corresponds to the first coordinate system, and the second camera corresponds to the second coordinate system;

[0039] A determination module, used to determine that the posture information and the position information are the current posture information of the task execution subject;

[0040] A time sequence arrangement module, used for, after the task execution subject finishes executing the target task, performing time sequence arrangement on each acquired posture information to obtain posture data information, and performing time sequence arrangement on each first image frame to obtain visual data information;

[0041] A control module is used to control the robot to learn the process of the task execution subject performing the target task based on the posture data information and the visual data information.

[0042] In a third aspect, the present application provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the robot learning method based on dual-camera fusion as described in any one of the above items is implemented.

[0043] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the robot learning method based on dual-camera fusion as described in any one of the above items is implemented.

[0044] The present application provides a robot learning method, device, equipment and storage medium based on dual camera fusion. The method includes: in the process of the task execution subject performing the target task, the posture information of the task execution subject is obtained in real time through a preset posture acquisition algorithm, and the first image frame is obtained in real time through the first camera carried by the task execution subject, and it is determined whether the image frame includes the preset image information; if not, the current posture information of the task execution subject is obtained based on the preset iterative closest point global optimization algorithm; the initial position information of the task execution subject in the second coordinate system is obtained through the preset second camera, and the initial position information is converted from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, so as to obtain the current position information of the task execution subject; wherein, the second camera is fixedly set, the first camera corresponds to the first coordinate system, and the second camera corresponds to the second coordinate system; the posture information and the position information are determined to be the current posture information of the task execution subject; after the task execution subject finishes executing the target task, the posture information obtained is arranged in time sequence to obtain the posture data information, and the first image frames are arranged in time sequence to obtain the visual data information;

[0045] The process of controlling the robot to learn the task execution subject to perform the target task based on the posture data information and the visual data information. The method acquires the first image frame in real time through the first camera, and determines whether the image frame includes the preset image information, and when the image frame does not include the preset image information, acquires the current posture information of the task execution subject based on the preset iterative closest point global optimization algorithm, and at the same time, acquires the initial position information of the task execution subject in the second coordinate system through the preset second camera, and converts the initial position information from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, and obtains the current position information of the task execution subject. When the ORB_SLAM3 algorithm loses tracking, it can still accurately acquire the posture information of the task execution subject, thereby improving the application potential of the ORB_SLAM3 algorithm in complex dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0047] Figure 1A schematic diagram of a flow chart of a robot learning method based on dual-camera fusion provided in an embodiment of the present application;

[0048] Figure 2 A schematic block diagram of the structure of a robot learning device based on dual-camera fusion provided in an embodiment of the present application;

[0049] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0052] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0053] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0054] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0055] See also Figure 1 , Figure 1 A schematic diagram of the process of a robot learning method based on dual camera fusion provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes steps S1 to S6.

[0056] Step S1: During the process of the task execution subject executing the target task, the posture information of the task execution subject is acquired in real time through a preset posture acquisition algorithm, and the first image frame is acquired in real time through the first camera carried by the task execution subject, and it is determined whether the image frame includes preset image information.

[0057] Step S2: If not included, obtain the current posture information of the task execution subject based on a preset iterative closest point global optimization algorithm.

[0058] Step S3, obtaining the initial position information of the task execution subject in the second coordinate system through a preset second camera, and converting the initial position information from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, to obtain the current position information of the task execution subject; wherein, the second camera is fixedly set, the first camera corresponds to the first coordinate system, and the second camera corresponds to the second coordinate system.

[0059] Step S4: Determine that the posture information and the position information are the current posture information of the task execution subject.

[0060] Step S5: After the task execution subject finishes executing the target task, the acquired posture information is arranged in time sequence to obtain posture data information, and the first image frames are arranged in time sequence to obtain visual data information.

[0061] Step S6: Control the robot to learn the process of the task execution subject performing the target task based on the posture data information and the visual data information.

[0062] Among them, the task execution subject can be a human operator or an end effector, the position information includes position information and posture information, the posture acquisition algorithm is the ORB_SLAM3 algorithm, the iterative closest point global optimization algorithm is the ICG+ algorithm in 3D ObjectTracking, and the first coordinate system and the second coordinate system are both spatial rectangular coordinate systems.

[0063] It should be noted that the ORB_SLAM3 algorithm is prone to the problem of lost tracking when the movement speed of the first camera is fast, which causes the ORB_SLAM3 algorithm to be unable to accurately confirm the posture information of the task execution subject. The method provided in this embodiment obtains the first image frame in real time through the first camera, and determines whether the image frame includes preset image information, and when the preset image information is not included in the image frame, the current posture information of the task execution subject is obtained based on the preset iterative closest point global optimization algorithm, and at the same time, the initial position information of the task execution subject in the second coordinate system is obtained through the preset second camera, and the initial position information is converted from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, and the current position information of the task execution subject is obtained. When the ORB_SLAM3 algorithm loses tracking, it can still accurately obtain the posture information of the task execution subject, which improves the application potential of the ORB_SLAM3 algorithm in complex dynamic environments. It can be understood that in the method provided in this embodiment, when the preset image information is not included in the image frame, it indicates that the ORB_SLAM3 algorithm has a problem of lost tracking.

[0064] It is understandable that when the ORB_SLAM3 algorithm loses tracking, it is impossible to accurately obtain the location information of the task execution subject. Without the auxiliary tracking of the second camera, it is difficult to obtain a clear trajectory of the task execution subject performing the task. This embodiment uses the second camera for auxiliary tracking to ensure that a clear trajectory of the task execution subject performing the task is obtained during the process of the task execution subject performing the target task.

[0065] In some embodiments, the method for obtaining the preset image information includes the following steps:

[0066] A plurality of marks are set in the execution area of ​​the target task; each mark is different from another, and the mark is an Aruco mark;

[0067] When the task execution subject executes the target task from the starting position at a preset execution speed, a second image frame is acquired in real time through the first camera; wherein the preset execution speed needs to ensure that the ORB_SLAM3 algorithm does not lose tracking problem;

[0068] Determining whether there is a common mark in each of the second image frames;

[0069] If so, determining that the image information corresponding to the shared mark is the preset image information;

[0070] If it does not exist, respectively counting the number of the second image frames corresponding to each of the marks, and arranging each of the marks in sequence based on the number of the second image frames corresponding to each of the marks, to obtain a mark sequence; wherein the number corresponding to each mark in the mark sequence decreases in sequence;

[0071] Deleting each second image frame in which a first mark in the mark sequence appears from all the second image frames, and determining the first mark as a first target mark;

[0072] Deleting each second image frame in which a second mark in the mark sequence appears from all remaining second image frames, and determining the second mark as a second target mark;

[0073] Iterate the steps of deleting each second image frame in which the first mark in the mark sequence appears from all the second image frames, and determining that the first mark is the first target mark, until all the second image frames are deleted;

[0074] The image information corresponding to each target mark is respectively determined to be the preset image information.

[0075] The method provided in this embodiment is as follows: first, multiple Aruco marks are set in the execution area of ​​the target task. The Aruco mark is a mark that can be recognized by a visual system, and each mark has a unique pattern. Secondly, when the task execution subject starts to execute the task, the first camera will acquire the second image frame in real time at a preset execution speed. After the task execution subject completes the task execution, first, it is determined whether there is a common mark in each of the second image frames. If so, it is determined that the image information corresponding to the common mark is the preset image information. If not, first, the number of the second image frames corresponding to each of the marks is counted respectively, and each of the marks is arranged in sequence based on the number of the second image frames corresponding to each of the marks to obtain a mark sequence. Then, in all the second image frames, each second image frame in which the first mark in the mark sequence appears is deleted, and the first mark is determined to be the first target mark. In all the remaining second image frames, each second image frame in which the second mark in the mark sequence appears is deleted, and the second mark is determined to be the second target mark. And so on, until all the second image frames are deleted. Finally, it is determined that the image information corresponding to each target mark is the preset image information. The method provided in this embodiment, on the one hand, ensures that during the task execution process, the first camera can continuously and stably capture sufficient image information by limiting the execution speed of the task execution subject, thereby avoiding the problem of tracking loss and obtaining effective preset image information. On the other hand, it can reduce the complexity of the preset image information as much as possible, which helps to improve the efficiency and accuracy of determining whether the image frame includes the preset image information.

[0076] In some embodiments, the step of obtaining the initial position information of the task execution subject in the second coordinate system through a preset second camera includes the following steps:

[0077] Mapping the second coordinate system to the execution area of ​​the target task; the coordinate origin of the second coordinate system is a designated point on the second camera;

[0078] The coordinate point of the designated point on the task execution subject in the second coordinate system is determined as the initial position information.

[0079] It can be understood that the second coordinate system is a rectangular coordinate system constructed in advance.

[0080] The method provided in this embodiment can accurately obtain the initial position information by mapping the second coordinate system to the execution area of ​​the target task and determining the coordinate point of the specified point on the task execution subject in the second coordinate system as the initial position information.

[0081] In some embodiments, the method for obtaining the conversion relationship between the first coordinate system and the second coordinate system comprises the following steps:

[0082] Before the task execution subject executes the target task, a first coordinate system is constructed with the position of the designated point of the task execution subject as the coordinate origin; it can be understood that at this time, the task execution subject is located at the starting position;

[0083] Constructing a second rectangular coordinate system with the position of the designated point of the second camera as the coordinate origin;

[0084] Setting a mark in the execution area of ​​the target task; the mark appears in the field of view of the first camera and the second camera;

[0085] Mapping the first coordinate system to the execution area of ​​the target task, and obtaining a first coordinate point of the center of the circumscribed circle of the mark in the first coordinate system;

[0086] Mapping the second coordinate system to the execution area of ​​the target task, and obtaining a second coordinate point of the center of the circumscribed circle of the mark in the second coordinate system;

[0087] The conversion relationship is generated based on the first coordinate point and the second coordinate point.

[0088] Among them, the method for generating the conversion relationship based on the first coordinate point and the second coordinate point is to construct a functional relationship between the numbers of the first coordinate point and the second coordinate point on the same coordinate axis. Exemplarily, the first coordinate point is A1 (20, 25, 10), and the second coordinate point is A2 (12, 15, 20). The first coordinate point A1 (20, 25, 10) can be expressed as A1 (12+8, 15+10, 20-10) using the second coordinate point A2 (12, 15, 20). Assuming A2 is (X2, Y2, Z2), A1 is (X1=X2+8, Y1=Y2+10, Z2=Z2-10).

[0089] The method provided in this embodiment, first, before the task execution subject executes the target task, a first coordinate system is constructed with the position of the designated point of the task execution subject as the coordinate origin, and a second rectangular coordinate system is constructed with the position of the designated point of the second camera as the coordinate origin, then, a mark is set in the execution area of ​​the target task, and the first coordinate system is mapped to the execution area of ​​the target task, and the first coordinate point of the center of the circumscribed circle of the mark in the first coordinate system is obtained, and the second coordinate system is mapped to the execution area of ​​the target task, and the second coordinate system is obtained. The second coordinate point of the center of the circumscribed circle of the mark in the second coordinate system is obtained, and finally, the conversion relationship is generated based on the first coordinate point and the second coordinate point, which can simply and effectively obtain the conversion relationship between the first coordinate system and the second coordinate system.

[0090] See also Figure 2 , Figure 2 A schematic block diagram of the structure of a robot learning device based on dual camera fusion provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the robot learning device based on dual camera fusion provided in the embodiment of the present application includes:

[0091] The first acquisition module 110 is used to acquire the posture information of the task execution subject in real time through a preset posture acquisition algorithm during the process of the task execution subject performing the target task, and to acquire the first image frame in real time through the first camera carried by the task execution subject, and to determine whether the image frame includes preset image information.

[0092] The second acquisition module 120 is used to acquire the current posture information of the task execution subject based on a preset iterative closest point global optimization algorithm if it is not included.

[0093] The conversion module 130 is used to obtain the initial position information of the task execution subject in the second coordinate system through a preset second camera, and convert the initial position information from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system to obtain the current position information of the task execution subject; wherein, the second camera is fixedly set, the first camera corresponds to the first coordinate system, and the second camera corresponds to the second coordinate system.

[0094] The determination module 140 is used to determine that the posture information and the position information are the current posture information of the task execution subject.

[0095] The time sequence arrangement module 150 is used to time sequence the acquired posture information to obtain posture data information after the task execution subject finishes executing the target task, and time sequence the first image frames to obtain visual data information.

[0096] The control module 160 is used to control the process in which the robot learns the task execution subject to execute the target task based on the posture data information and the visual data information.

[0097] It should be noted that technical personnel in the relevant technical field can clearly understand that, for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned embodiment of the robot learning method based on dual-camera fusion, and will not be repeated here.

[0098] The robot learning device 100 based on dual camera fusion provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in a computer system such as Figure 3 The system is run on the terminal device 200 shown.

[0099] See also Figure 3 , Figure 3 The terminal device 200 provided in the embodiment of the present application is a schematic block diagram of the structure, wherein the terminal device 200 includes a processor 201 and a memory 202, wherein the processor 201 and the memory 202 are connected via a system bus 203, wherein the memory 202 may include a non-volatile storage medium and an internal memory.

[0100] The non-volatile storage medium can store a computer program. The computer program includes program instructions, and when the program instructions are executed by the processor 201, the processor 201 can execute any of the above-mentioned robot learning methods based on dual-camera fusion.

[0101] The processor 201 is used to provide computing and control capabilities to support the operation of the entire terminal device 200.

[0102] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 201, the processor 201 can execute any of the above-mentioned robot learning methods based on dual-camera fusion.

[0103] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal device 200 involved in the scheme of the present application. The specific terminal device 200 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0104] It should be understood that the processor 201 may be a central processing unit (CPU), and the processor 201 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0105] In some embodiments, the processor 201 is used to run a computer program stored in the memory to implement the following steps:

[0106] In the process of the task execution subject performing the target task, the posture information of the task execution subject is obtained in real time through a preset posture acquisition algorithm, and a first image frame is obtained in real time through a first camera carried by the task execution subject, and it is determined whether the image frame includes preset image information;

[0107] If not included, obtaining the current posture information of the task execution subject based on a preset iterative closest point global optimization algorithm;

[0108] Acquire the initial position information of the task execution subject in the second coordinate system through a preset second camera, and convert the initial position information from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, so as to obtain the current position information of the task execution subject; wherein the second camera is fixedly arranged, the first camera corresponds to the first coordinate system, and the second camera corresponds to the second coordinate system;

[0109] Determine that the posture information and the position information are current posture information of the task execution subject;

[0110] After the task execution subject finishes executing the target task, each acquired posture information is time-sequentially arranged to obtain posture data information, and each first image frame is time-sequentially arranged to obtain visual data information;

[0111] The control robot learns the process of the task execution subject performing the target task based on the posture data information and the visual data information.

[0112] It should be noted that technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the terminal device 200 described above can refer to the corresponding process of the aforementioned robot learning method based on dual-camera fusion, and will not be repeated here.

[0113] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the one or more processors implement a robot learning method based on dual-camera fusion as provided in an embodiment of the present application.

[0114] The computer-readable storage medium may be an internal storage unit of the terminal device 200 in the aforementioned embodiment, such as a hard disk or memory of the terminal device 200. The computer-readable storage medium may also be an external storage device of the terminal device 200, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped with the terminal device 200.

[0115] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A robot learning method based on dual camera fusion, characterized in that: include: In the process of the task execution subject performing the target task, the posture information of the task execution subject is obtained in real time through a preset posture acquisition algorithm, and a first image frame is obtained in real time through a first camera carried by the task execution subject, and it is determined whether the image frame includes preset image information; If not included, obtaining the current posture information of the task execution subject based on a preset iterative closest point global optimization algorithm; Acquire the initial position information of the task execution subject in the second coordinate system through a preset second camera, and convert the initial position information from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, so as to obtain the current position information of the task execution subject; wherein the second camera is fixedly arranged, the first camera corresponds to the first coordinate system, and the second camera corresponds to the second coordinate system; Determine that the posture information and the position information are current posture information of the task execution subject; After the task execution subject finishes executing the target task, each acquired posture information is time-sequentially arranged to obtain posture data information, and each first image frame is time-sequentially arranged to obtain visual data information; The control robot learns the process of the task execution subject performing the target task based on the posture data information and the visual data information.

2. The robot learning method based on dual camera fusion according to claim 1, characterized in that: The method for obtaining the preset image information includes: Setting a plurality of marks in the execution area of ​​the target task; each mark is different from another; When the task execution subject executes the target task from the starting position at a preset execution speed, acquiring a second image frame in real time through the first camera; Determining whether there is a common mark in each of the second image frames; If so, it is determined that the image information corresponding to the shared mark is the preset image information.

3. The robot learning method based on dual camera fusion according to claim 2 is characterized in that: After determining whether there is a common mark in each of the second image frames, the method further includes: If it does not exist, respectively counting the number of the second image frames corresponding to each of the marks, and arranging each of the marks in sequence based on the number of the second image frames corresponding to each of the marks, to obtain a mark sequence; wherein the number corresponding to each mark in the mark sequence decreases in sequence; Deleting each second image frame in which a first mark in the mark sequence appears from all the second image frames, and determining the first mark as a first target mark; Deleting each second image frame in which a second mark in the mark sequence appears from all remaining second image frames, and determining the second mark as a second target mark; Iterate the steps of deleting each second image frame in which the first mark in the mark sequence appears from all the second image frames, and determining that the first mark is the first target mark, until all the second image frames are deleted; The image information corresponding to each target mark is respectively determined to be the preset image information.

4. The robot learning method based on dual camera fusion according to claim 1, characterized in that: The obtaining the initial position information of the task execution subject in the second coordinate system through a preset second camera includes: Mapping the second coordinate system to the execution area of ​​the target task; the coordinate origin of the second coordinate system is a designated point on the second camera; The coordinate point of the designated point on the task execution subject in the second coordinate system is determined as the initial position information.

5. The robot learning method based on dual camera fusion according to claim 1, characterized in that: The method for acquiring the conversion relationship between the first coordinate system and the second coordinate system includes: Before the task execution subject executes the target task, a first coordinate system is constructed with the position of the designated point of the task execution subject as the coordinate origin; Constructing a second rectangular coordinate system with the position of the designated point of the second camera as the coordinate origin; Setting a mark in the execution area of ​​the target task; the mark appears in the field of view of the first camera and the second camera; Mapping the first coordinate system to the execution area of ​​the target task, and obtaining a first coordinate point of the center of the circumscribed circle of the mark in the first coordinate system; Mapping the second coordinate system to the execution area of ​​the target task, and obtaining a second coordinate point of the center of the circumscribed circle of the mark in the second coordinate system; The conversion relationship is generated based on the first coordinate point and the second coordinate point.

6. A robot learning device based on dual camera fusion, characterized in that: include: A first acquisition module is used to acquire the posture information of the task execution subject in real time through a preset posture acquisition algorithm during the process of the task execution subject performing the target task, and to acquire a first image frame in real time through a first camera carried by the task execution subject, and to determine whether the image frame includes preset image information; A second acquisition module is used to acquire the current posture information of the task execution subject based on a preset iterative closest point global optimization algorithm if it is not included; A conversion module, used for obtaining the initial position information of the task execution subject in the second coordinate system through a preset second camera, and converting the initial position information from the second coordinate system to the first coordinate system based on the conversion relationship between the first coordinate system and the second coordinate system, so as to obtain the current position information of the task execution subject; wherein the second camera is fixedly arranged, the first camera corresponds to the first coordinate system, and the second camera corresponds to the second coordinate system; A determination module, used to determine that the posture information and the position information are the current posture information of the task execution subject; A time sequence arrangement module, used for, after the task execution subject finishes executing the target task, performing time sequence arrangement on each acquired posture information to obtain posture data information, and performing time sequence arrangement on each first image frame to obtain visual data information; A control module is used to control the robot to learn the process of the task execution subject performing the target task based on the posture data information and the visual data information.

7. A terminal device, characterized in that: The terminal device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the robot learning method based on dual-camera fusion as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: A computer program is stored on the computer-readable storage medium, wherein when the computer program is executed by a processor, the robot learning method based on dual-camera fusion as described in any one of claims 1 to 5 is implemented.

Images