Robot control method, device, storage medium and electronic device

Abstract

The present disclosure relates to a robot control method, apparatus, storage medium, and electronic device. The method includes: obtaining a real image collected by an image acquisition device disposed on an entity robot in a real scenario; obtaining a virtual image collected by a virtual robot in a virtual scenario, where the virtual scenario is a three-dimensional scenario constructed according to historical images collected from the real scenario, and the virtual robot is a digital twin of the entity robot; determining whether a difference degree between the virtual image and the real image exceeds a preset degree; and if the difference degree exceeds the preset degree, controlling the image acquisition device to perform image acquisition with a first strategy to obtain more detailed information about the real scenario. Thus, when there is a large difference between the real image and the virtual image, more detailed information about the real scenario is obtained, providing the operator with the details of the real scenario and improving the accuracy and efficiency of the operator's monitoring and assisting the robot.

Description

Technical Field

[0001] The present disclosure relates to the field of robot technology, and in particular, to a robot control method, apparatus, storage medium, and electronic device. Background Art

[0002] With the development of robot technology, currently, robots can be used to assist or even replace humans in performing various tasks. In order to better control robots, digital twin technology can be used to simulate physical robots in the real scene and display them as virtual robots in a three-dimensional virtual scene constructed artificially. In this way, operators can obtain relevant information about the physical robots in the real scene in real time through the virtual robots in the virtual scene, and then monitor and assist the physical robots. Summary of the Invention

[0003] The purpose of the present disclosure is to provide a robot control method, apparatus, storage medium, and electronic device, which can provide more detailed content about the real scene for operators and improve the accuracy and efficiency of operators in monitoring and assisting robots.

[0004] To achieve the above purpose, according to the first aspect of the present disclosure, a robot control method is provided. The method includes:

[0005] Obtain a real image collected by an image acquisition device disposed on a physical robot in a real scene;

[0006] Obtain a virtual image collected by a virtual robot in a virtual scene, where the virtual scene is a three-dimensional scene constructed based on historical images collected from the real scene, and the virtual robot is a digital twin of the physical robot;

[0007] Determine whether the degree of difference between the virtual image and the real image exceeds a preset degree;

[0008] If the degree of difference exceeds the preset degree, control the image acquisition device to perform image acquisition with a first strategy to obtain more detailed information about the real scene.

[0009] Optionally, the determining whether the degree of difference between the virtual image and the real image exceeds a preset degree includes:

[0010] Determine the number of pixel points with differences in the virtual image and the real image;

[0011] If the number is greater than a preset number, determine that the degree of difference between the virtual image and the real image exceeds the preset degree;

[0012] If the quantity is less than or equal to the preset quantity, it is determined that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0013] Optionally, determining whether the degree of difference between the virtual image and the real image exceeds the preset degree includes:

[0014] Comparing the virtual image and the real image to determine whether there is a target object, where the target object is a first object that exists in the virtual image but not in the real image or a second object that exists in the real image but not in the virtual image;

[0015] If there is the target object, it is determined that the degree of difference between the virtual image and the real image exceeds the preset degree;

[0016] If there is no such target object, it is determined that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0017] Optionally, determining whether the degree of difference between the virtual image and the real image exceeds the preset degree includes:

[0018] Comparing the virtual image and the real image to determine whether there is a target object, where the target object is a first object that exists in the virtual image but not in the real image or a second object that exists in the real image but not in the virtual image;

[0019] If there is the target object, determine the screen occupancy ratio of the target object;

[0020] If the screen occupancy ratio is greater than the preset ratio, it is determined that the degree of difference between the virtual image and the real image exceeds the preset degree;

[0021] If there is no such target object, or the screen occupancy ratio is less than or equal to the preset ratio, it is determined that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0022] Optionally, the entity robot adjusts the shooting angle by rotating the image acquisition device;

[0023] Controlling the image acquisition device to perform image acquisition with a first strategy to obtain more detailed information about the real scene includes:

[0024] Reducing the moving speed of the entity robot and / or reducing the rotation speed of the image acquisition device to obtain more detailed information about the real scene.

[0025] Optionally, controlling the image acquisition device to acquire images according to a first strategy to obtain more detailed information about the real scene, including:

[0026] If a second object that exists in the real image but does not exist in the virtual image is recognized, controlling the image acquisition device to follow and photograph the second object to obtain more detailed information about the second object.

[0027] Optionally, the image acquisition device includes multiple cameras;

[0028] The controlling the image acquisition device to follow and photograph the second object includes:

[0029] Determining the target camera closest to the second object;

[0030] Photographing the second object through the target camera.

[0031] Optionally, the method further includes:

[0032] If the degree of difference does not exceed the preset degree, controlling the image acquisition device to acquire images according to a second strategy to obtain image information different from the real image.

[0033] Optionally, the entity robot adjusts the shooting angle by rotating the image acquisition device;

[0034] The controlling the image acquisition device to acquire images according to a second strategy to obtain image information different from the real image includes:

[0035] Increasing the moving speed of the entity robot and / or increasing the rotation speed of the image acquisition device to obtain image information different from the real image.

[0036] Optionally, before the step of acquiring the real image acquired by the image acquisition device disposed on the entity robot in the real scene, the method further includes:

[0037] Recognizing that the task currently executed by the entity robot is interfered.

[0038] According to a second aspect of the present disclosure, there is provided a robot control device, and the device includes:

[0039] A first acquisition module, configured to acquire a real image acquired by an image acquisition device disposed on an entity robot in a real scene;

[0040] A second acquisition module, configured to acquire virtual images collected by a virtual robot in a virtual scene, where the virtual scene is a three-dimensional scene constructed based on historical images collected from the real scene, and the virtual robot is a digital twin of the physical robot;

[0041] A determination module, configured to determine whether the degree of difference between the virtual image and the real image exceeds a preset degree;

[0042] A first control module, configured to, if the degree of difference exceeds the preset degree, control the image acquisition device to perform image acquisition according to a first strategy to obtain more detailed information about the real scene.

[0043] According to a third aspect of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect of the present disclosure are implemented.

[0044] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0045] A memory, on which a computer program is stored;

[0046] A processor, configured to execute the computer program in the memory to implement the steps of the method described in the first aspect of the present disclosure.

[0047] Through the above technical solutions, real images collected by an image acquisition device disposed on a physical robot in a real scene are acquired, virtual images collected by a virtual robot in a virtual scene are acquired, and it is determined whether the degree of difference between the virtual image and the real image exceeds a preset degree, and, if the degree of difference exceeds the preset degree, the image acquisition device is controlled to perform image acquisition according to a first strategy to obtain more detailed information about the real scene. Wherein, the virtual scene is a three-dimensional scene constructed based on historical images collected from the real scene, and the virtual robot is a digital twin of the physical robot. Thus, based on the degree of difference between the real image of the real scene and the virtual image of the virtual scene, the image acquisition strategy of the image acquisition device on the physical robot can be adjusted in real time, so that when the difference between the real image and the virtual image is large, more detailed information about the real scene can be obtained, and further more detailed content about the real scene can be provided for the operator, improving the accuracy and efficiency of the operator's monitoring and assisting the robot.

[0048] Other features and advantages of the present disclosure will be described in detail in the subsequent specific implementation part. Description of the Drawings

[0049] The accompanying drawings are used to provide a further understanding of the present disclosure, and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the accompanying drawings:

[0050] Figure 1 is a flowchart of a robot control method provided according to an embodiment of the present disclosure;

[0051] Figure 2 is an exemplary flowchart of a step of determining whether the degree of difference between a virtual image and a real image exceeds a preset degree in the robot control method provided according to the present disclosure;

[0052] Figure 3 is a flowchart of a robot control method provided according to another embodiment of the present disclosure;

[0053] Figure 4 is a block diagram of a robot control device provided according to an embodiment of the present disclosure;

[0054] Figure 5 is a block diagram of an electronic device shown according to an exemplary embodiment;

[0055] Figure 6 is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed Description of the Embodiment

[0056] The following will provide a detailed description of the specific embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure, and are not used to limit the present disclosure.

[0057] Before introducing the solution of the present disclosure, a brief description of the application scenario related to the present disclosure will be given first. As described in the background art, in order to better control a robot, the physical robot in the real scenario can be simulated and displayed in the virtual scenario in the form of a simulated robot through the digital twin method. The virtual scenario is obtained by three-dimensional reconstruction of the real scenario where the physical robot is located. Therefore, it is necessary for the physical robot to collect the images in the real scenario and send them to the server or control platform for three-dimensional reconstruction of the scenario.

[0058] Figure 1 is a flowchart of a robot control method provided according to an embodiment of the present disclosure. The robot control method provided by the present disclosure can be applied to a server capable of communicating with a physical robot and a control platform, or the robot control method provided by the present disclosure can be applied to a physical robot.

[0059] As Figure 1 shown, the method may include Step 11 to Step 14.

[0060] In step 11, obtain a real image captured by an image acquisition device disposed on an entity robot in a real scenario.

[0061] The image acquisition device is disposed on the entity robot and is used to capture images of the environment passed by the entity robot. Exemplarily, the image acquisition device may include at least one camera, and each camera is installed at a different position of the entity robot to capture images in different directions.

[0062] In step 12, obtain a virtual image captured by a virtual robot in a virtual scenario.

[0063] Wherein, the virtual scenario is a three-dimensional scenario constructed according to historical images collected from the real scenario, and the virtual robot is a digital twin of the entity robot.

[0064] For the convenience of controlling the robot, generally, the entity robot in the real scenario is pre-simulated and displayed in the corresponding picture of the virtual scenario in the way of digital twin. The virtual scenario can be obtained by simulating the real images collected and transmitted by the entity robot in the real scenario. Specifically in implementation, the virtual scenario and the virtual robot obtained by digital twin can be displayed on a control platform for an operator to view.

[0065] It should be noted that the virtual scenario is not a real-time scenario and is constructed in advance by a certain method. For example, the virtual scenario can be a three-dimensional scenario constructed according to historical images collected from the real scenario, that is, the virtual scenario is consistent with the real scenario at a certain past time point. However, the real scenario is not static, so the images captured by the entity robot in the real scenario in real time are not always consistent with the virtual scenario.

[0066] In step 13, determine whether the degree of difference between the virtual image and the real image exceeds a preset degree.

[0067] In a possible implementation manner, step 13 may include the following steps:

[0068] Determine the number of pixel points with differences in the virtual image and the real image;

[0069] If the number is greater than a preset number, determine that the degree of difference between the virtual image and the real image exceeds the preset degree;

[0070] If the number is less than or equal to the preset number, determine that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0071] Since the virtual robot is the digital twin of the physical robot, the size of the virtual image captured by the virtual robot is the same as the size of the real image captured by the physical robot. Therefore, it is possible to determine whether the pixel points at the corresponding positions in the real image and the virtual image are the same, that is, to compare whether the pixel points with the same coordinates in the real image and the virtual image are the same, and then determine the number of pixel points that are different between the virtual image and the real image.

[0072] If the number of pixel points with differences between the virtual image and the real image is greater than the preset number, it means that there are many pixel points with differences between the virtual image and the real image. Thus, it can be determined that the degree of difference between the virtual image and the real image exceeds the preset degree. On the contrary, if the number is less than or equal to the preset number, it means that there are not many pixel points with differences between the virtual image and the real image. Therefore, it can be determined that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0073] In another possible implementation, step 13 may include the following steps:

[0074] Compare the virtual image and the real image to determine whether there is a target object;

[0075] If there is a target object, determine that the degree of difference between the virtual image and the real image exceeds the preset degree;

[0076] If there is no target object, determine that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0077] Wherein, the target object is a first object that exists in the virtual image but not in the real image or a second object that exists in the real image but not in the virtual image.

[0078] Exemplarily, image recognition technology can be used to identify the objects (such as people or objects) in the virtual image and the real image, and determine whether the objects in the virtual image and the real image can correspond to each other. If they cannot correspond to each other, it is determined that there is a target object. And if an object exists in the virtual image but not in the real image, it means that the target object includes the first object, or if an object exists in the real image but not in the virtual image, it means that the target object includes the second object.

[0079] If there is a target object, it means that there are new objects or missing objects in the real scene compared to the virtual scene, that is, there are already differences between the real scene and the virtual scene. Therefore, it can be determined that the degree of difference between the virtual image and the real image exceeds the preset degree. On the contrary, if there is no target object, it means that there are no new or missing objects in the real scene compared to the virtual scene. Therefore, it can be determined that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0080] In another possible implementation, step 13 may include the following steps, as Figure 2 shown.

[0081] In step 21, the virtual image and the real image are compared to determine whether there is a target object.

[0082] Wherein, the target object is a first object that exists in the virtual image but not in the real image or a second object that exists in the real image but not in the virtual image.

[0083] Exemplarily, the objects (such as people or objects) in the virtual image and the real image can be recognized by image recognition technology, and it can be determined whether the objects in the virtual image and the real image can correspond to each other. If they cannot correspond to each other, it is determined that there is a target object. Moreover, if an object exists in the virtual image but not in the real image, it indicates that the target object includes the first object, or, if an object exists in the real image but not in the virtual image, it indicates that the target object includes the second object.

[0084] In step 22, if there is a target object, determine the screen occupancy ratio of the target object.

[0085] If there is a target object, it indicates that there is a difference between the virtual image and the real image. Therefore, in order to confirm the degree of difference between the two, it is necessary to further determine the proportion of this difference in the image, that is, the screen occupancy ratio of the target object.

[0086] Exemplarily, if the target object is the first object, the occupancy ratio of the first object in the virtual image can be determined as the screen occupancy ratio of the target object. For another example, if the target object is the second object, the occupancy ratio of the second object in the real image can be determined as the screen occupancy ratio of the target object.

[0087] In step 23, if the screen occupancy ratio is greater than the preset ratio, determine that the degree of difference between the virtual image and the real image exceeds the preset degree.

[0088] If the screen occupancy ratio is greater than the preset ratio, it indicates that there are relatively large differences between the virtual image and the real image. Therefore, it can be determined that the degree of difference between the virtual image and the real image exceeds the preset degree.

[0089] In step 24, if there is no target object, or the screen occupancy ratio is less than or equal to the preset ratio, determine that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0090] If there is no target object, it indicates that the objects included in the virtual image and the real image are the same, and there is no large difference between the two. Therefore, it can be determined that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0091] If the screen occupation ratio is less than or equal to the preset ratio, it indicates that although there are differences between the virtual image and the real image, the occupation ratio is relatively small. Therefore, it can be determined that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0092] In step 14, if the degree of difference exceeds the preset degree, control the image acquisition device to perform image acquisition with the first strategy to obtain more detailed information about the real scene.

[0093] If the degree of difference between the real image and the virtual image exceeds the preset degree, it indicates that there are significant differences between the virtual scene and the real scene. That is to say, some content in the virtual scene has lost its meaning. Therefore, in order to ensure the control efficiency and accuracy of the operator for the physical robot and improve the task completion efficiency of the physical robot, it is necessary to obtain more detailed information about the real scene so that the operator can know what kind of content is included in the part where there are differences between the real image and the virtual image.

[0094] In one possible implementation, step 14 may include the following steps:

[0095] Reduce the moving speed of the physical robot and / or reduce the rotation speed of the image acquisition device to obtain more detailed information about the real scene.

[0096] Generally, the physical robot moves in the real scene and acquires image information of the real scene during the movement. Among them, the physical robot adjusts the shooting angle by rotating the image acquisition device. Since the acquisition frequency of the image acquisition device remains unchanged, the faster the moving speed of the physical robot, the fewer the number of images taken after passing the same distance, and the less detailed information about the real scene is obtained. Similarly, the faster the rotation speed of the image acquisition device, the fewer the number of images taken within the same time period, and the less detailed information about the real scene is obtained.

[0097] Therefore, reducing the moving speed of the physical robot can take more images within the same distance compared to before reducing the speed, thereby obtaining more detailed information about the real scene. Reducing the rotation speed of the image acquisition device can take more images within the same angular rotation range compared to before reducing the rotation speed, thereby obtaining more detailed information about the real scene.

[0098] In another possible implementation, step 14 may include the following steps:

[0099] If a second object that exists in the real image but does not exist in the virtual image is recognized, control the image acquisition device to follow and shoot the second object to obtain more detailed information about the second object.

[0100] If the second object is recognized, it indicates that a second object has been newly added in the real scenario, which does not exist in the virtual scenario. Therefore, the second object can be followed for shooting to obtain detailed information about the second object, so as to provide as much information as possible about the second object to the operator.

[0101] In a possible embodiment, the image acquisition device may include multiple cameras. Correspondingly, controlling the image acquisition device to follow and shoot the second object may include the following steps:

[0102] Determine the target camera closest to the second object;

[0103] Shoot the second object through the target camera.

[0104] Exemplarily, the movement trend of the second object can be determined through the images containing the second object acquired by the image acquisition device within a period of time, and then, according to the positional distribution relationship of each camera, the target camera closest to the second object can be determined. For example, if it is determined that the second object moves to the left and the second object disappears from the shooting field of view of the current camera, it can be determined that the camera to the left of the current camera is the target camera.

[0105] For another example, the distance between the second object and each camera can be detected by a ranging device (such as ultrasonic, lidar, etc.), and then the camera with the shortest distance to the second object can be determined as the target camera.

[0106] After determining the target camera, continue to shoot the second object through the target camera to obtain more detailed information about the second object. At the same time, the current camera can also be turned off to save energy.

[0107] Optionally, the method provided by the present disclosure can take effect in a specific scenario. Therefore, before step 11, the method provided by the present disclosure may further include the following steps:

[0108] Recognize that the task currently executed by the entity robot is interfered.

[0109] When the task currently executed by the entity robot is interfered, it indicates that the real scenario may have changed and affected the execution of the robot task. Therefore, it is necessary to compare the virtual image and the real image to confirm whether more details of the real scenario need to be obtained. Thus, unnecessary data processing can be avoided and computing resources can be saved.

[0110] Through the above technical solution, a real image collected by an image acquisition device disposed on an entity robot in a real scenario is obtained, a virtual image collected by a virtual robot in a virtual scenario is obtained, and it is determined whether the degree of difference between the virtual image and the real image exceeds a preset degree. Moreover, if the degree of difference exceeds the preset degree, the image acquisition device is controlled to perform image acquisition with a first strategy to obtain more detailed information about the real scenario. Among them, the virtual scenario is a three-dimensional scenario constructed based on historical images collected from the real scenario, and the virtual robot is a digital twin of the entity robot. Thus, based on the degree of difference between the real image of the real scenario and the virtual image of the virtual scenario, the image acquisition strategy of the image acquisition device on the entity robot can be adjusted in real time. Therefore, when the difference between the real image and the virtual image is large, more detailed information about the real scenario can be obtained, and then more detailed content about the real scenario can be provided to the operator, improving the accuracy and efficiency of the operator's monitoring and assisting the robot.

[0111] In a possible implementation manner, on the basis of the steps shown in Figure 1 The method provided by the present disclosure may further include the following steps, as shown in Figure 3 shown.

[0112] In step 31, if the degree of difference does not exceed the preset degree, the image acquisition device is controlled to perform image acquisition with a second strategy to obtain image information different from the real image.

[0113] If the degree of difference between the real image and the virtual image does not exceed the preset degree, it indicates that the difference between the virtual scenario and the real scenario is not significant. That is to say, the environment where the entity robot is currently located is consistent with the one constructed in the virtual scenario, and there is no need to continue shooting and analyzing. Therefore, the image acquisition device can be controlled to perform image acquisition with a second strategy to obtain image information different from the real image, that is, quickly switch to other areas outside the area corresponding to the real image in the real scenario.

[0114] In a possible implementation manner, step 31 may include the following steps:

[0115] Increase the moving speed of the entity robot and / or increase the rotation speed of the image acquisition device to obtain image information different from the real image.

[0116] Increasing the moving speed of the entity robot can cover more distances when shooting the same number of images compared with before increasing the speed, so as to obtain image information different from the real image, that is, image information of other areas outside the area corresponding to the real image in the real scenario. In this way, it can.

[0117] Increasing the rotation speed of the image acquisition device can turn through a greater angle when capturing a certain number of images compared to before increasing the rotation speed, so as to obtain image information different from the real image as soon as possible.

[0118] Figure 4 It is a block diagram of a robot control device provided according to an embodiment of the present disclosure. As Figure 4 shown, the device 40 includes:

[0119] A first acquisition module 41, configured to acquire a real image captured by an image acquisition device disposed on an entity robot in a real scene;

[0120] A second acquisition module 42, configured to acquire a virtual image captured by a virtual robot in a virtual scene, where the virtual scene is a three-dimensional scene constructed according to historical images captured from the real scene, and the virtual robot is a digital twin of the entity robot;

[0121] A determination module 43, configured to determine whether the degree of difference between the virtual image and the real image exceeds a preset degree;

[0122] A first control module 44, configured to, if the degree of difference exceeds the preset degree, control the image acquisition device to perform image acquisition with a first strategy to obtain more detailed information about the real scene.

[0123] Optionally, the determination module 43 includes:

[0124] A first determination sub-module, configured to determine the number of pixel points with differences in the virtual image and the real image;

[0125] A second determination sub-module, configured to, if the number is greater than a preset number, determine that the degree of difference between the virtual image and the real image exceeds the preset degree;

[0126] A third determination sub-module, configured to, if the number is less than or equal to the preset number, determine that the degree of difference between the virtual image and the real image does not exceed the preset degree.

[0127] Optionally, the determination module 43 includes:

[0128] A comparison sub-module, configured to compare the virtual image and the real image to determine whether there is a target object, where the target object is a first object that exists in the virtual image but does not exist in the real image or a second object that exists in the real image but does not exist in the virtual image;

[0129] A fourth determination sub-module, configured to determine that the degree of difference between the virtual image and the real image exceeds a preset degree if the target object exists;

[0130] A fifth determination sub-module, configured to determine that the degree of difference between the virtual image and the real image does not exceed a preset degree if the target object does not exist.

[0131] Optionally, the determination module 43 includes:

[0132] A comparison sub-module, configured to compare the virtual image and the real image to determine whether there is a target object, where the target object is a first object that exists in the virtual image but does not exist in the real image or a second object that exists in the real image but does not exist in the virtual image;

[0133] A sixth determination sub-module, configured to determine the screen occupancy ratio of the target object if the target object exists;

[0134] A seventh determination sub-module, configured to determine that the degree of difference between the virtual image and the real image exceeds a preset degree if the screen occupancy ratio is greater than a preset ratio;

[0135] An eighth determination sub-module, configured to determine that the degree of difference between the virtual image and the real image does not exceed a preset degree if the target object does not exist, or the screen occupancy ratio is less than or equal to the preset ratio.

[0136] Optionally, the entity robot adjusts the shooting angle by rotating the image acquisition device;

[0137] The first control module 44 includes:

[0138] A first control sub-module, configured to reduce the moving speed of the entity robot and / or reduce the rotation speed of the image acquisition device to obtain more detailed information about the real scene.

[0139] Optionally, the first control module 44 includes:

[0140] A second control sub-module, configured to control the image acquisition device to follow and shoot the second object if it recognizes a second object that exists in the real image but does not exist in the virtual image, so as to obtain more detailed information about the second object.

[0141] Optionally, the image acquisition device includes a plurality of cameras;

[0142] The second control sub-module includes:

[0143] A ninth determination sub-module, configured to determine the target camera closest to the second object;

[0144] A shooting sub-module, configured to shoot the second object through the target camera.

[0145] Optionally, the device 40 further includes:

[0146] A second control module, configured to control the image acquisition device to perform image acquisition with a second strategy to obtain image information different from the real image if the degree of difference does not exceed the preset degree.

[0147] Optionally, the entity robot adjusts the shooting angle by rotating the image acquisition device;

[0148] The second control module includes:

[0149] A third control sub-module, configured to increase the moving speed of the entity robot and / or increase the rotating speed of the image acquisition device to obtain image information different from the real image.

[0150] Optionally, the first acquisition module 41 is configured to acquire a real image collected by an image acquisition device disposed on the entity robot in a real scene after identifying that the task currently executed by the entity robot is interfered.

[0151] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0152] Figure 5 is a block diagram of an electronic device 700 shown according to an exemplary embodiment. As Figure 5 shown, the electronic device 700 may include: a processor 701, a memory 702. The electronic device 700 may further include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0153] Among them, the processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned robot control method. The memory 702 is used to store various types of data to support the operation of the electronic device 700. These data may include, for example, instructions for any application or method operating on the electronic device 700, as well as application-related data, such as contact data, received and sent messages, pictures, audio, video, and so on. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, magnetic disk or optical disc. The multimedia component 703 may include a screen and an audio component. The screen can be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal can be further stored in the memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules, and the above-mentioned other interface modules can be a keyboard, a mouse, buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, and so on.

[0154] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components, and is used to execute the above-mentioned robot control method.

[0155] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above-mentioned robot control method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 702 including program instructions, and the above-mentioned program instructions can be executed by the processor 701 of the electronic device 700 to complete the above-mentioned robot control method.

[0156] Figure 6 FIG. is a block diagram of an electronic device 1900 shown according to an exemplary embodiment. For example, the electronic device 1900 can be provided as a server. Referring to Figure 6 , the electronic device 1900 includes a processor 1922, the number of which can be one or more, and a memory 1932 for storing computer programs executable by the processor 1922. The computer programs stored in the memory 1932 can include one or more modules each corresponding to a set of instructions. In addition, the processor 1922 can be configured to execute the computer program to execute the above-mentioned robot control method.

[0157] In addition, the electronic device 1900 can further include a power supply component 1926 and a communication component 1950. The power supply component 1926 can be configured to perform power management of the electronic device 1900, and the communication component 1950 can be configured to implement communication of the electronic device 1900, for example, wired or wireless communication. In addition, the electronic device 1900 can further include an input / output (I / O) interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM and so on.

[0158] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above-described robot control method are implemented. For example, the computer-readable storage medium may be the above-described memory 1932 including program instructions, and the above program instructions may be executed by the processor 1922 of the electronic device 1900 to complete the above-described robot control method.

[0159] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above-described robot control method when executed by the programmable device.

[0160] The preferred embodiments of the present disclosure have been described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0161] In addition, it should be noted that, in the above specific embodiments, the various specific technical features described can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present disclosure does not separately describe various possible combination manners.

[0162] Furthermore, any combination can be made between various different embodiments of the present disclosure as long as it does not violate the idea of the present disclosure, and it should also be regarded as the content disclosed by the present disclosure.

Claims

1. A robot control method, characterized in that, The method includes: Obtaining a real image collected by an image acquisition device disposed on an entity robot in a real scenario; Obtaining a virtual image collected by a virtual robot in a virtual scenario, where the virtual scenario is a three-dimensional scenario constructed based on historical images collected from the real scenario, and the virtual robot is a digital twin of the entity robot; Determining whether the degree of difference between the virtual image and the real image exceeds a preset degree; If the degree of difference exceeds the preset degree, controlling the image acquisition device to perform image acquisition with a first strategy to obtain more detailed information about the real scenario.

2. The method according to claim 1, characterized in that, The determining whether the degree of difference between the virtual image and the real image exceeds a preset degree includes: Determining the number of pixel points with differences in the virtual image and the real image; If the number is greater than a preset number, determining that the degree of difference between the virtual image and the real image exceeds the preset degree; If the number is less than or equal to the preset number, determining that the degree of difference between the virtual image and the real image does not exceed the preset degree.

3. The method according to claim 1, wherein The determining whether the degree of difference between the virtual image and the real image exceeds a preset degree includes: Comparing the virtual image and the real image to determine whether there is a target object, where the target object is a first object that exists in the virtual image but does not exist in the real image or a second object that exists in the real image but does not exist in the virtual image; If there is the target object, determining that the degree of difference between the virtual image and the real image exceeds the preset degree; If there is no such target object, determining that the degree of difference between the virtual image and the real image does not exceed the preset degree.

4. The method according to claim 1, wherein The determining whether the degree of difference between the virtual image and the real image exceeds a preset degree includes: Comparing the virtual image and the real image to determine whether there is a target object, where the target object is a first object that exists in the virtual image but does not exist in the real image or a second object that exists in the real image but does not exist in the virtual image; If there is the target object, determining the screen occupancy ratio of the target object; If the screen occupancy ratio is greater than a preset ratio, determining that the degree of difference between the virtual image and the real image exceeds the preset degree; If there is no such target object, or the screen occupancy ratio is less than or equal to the preset ratio, determining that the degree of difference between the virtual image and the real image does not exceed the preset degree.

5. The method according to claim 1, wherein The entity robot adjusts the shooting angle by rotating the image acquisition device; The controlling the image acquisition device to perform image acquisition with a first strategy to obtain more detailed information about the real scenario includes: Reducing the moving speed of the entity robot and / or reducing the rotation speed of the image acquisition device to obtain more detailed information about the real scenario.

6. The method according to claim 1, wherein The controlling the image acquisition device to perform image acquisition with a first strategy to obtain more detailed information about the real scenario includes: If a second object that exists in the real image but does not exist in the virtual image is recognized, control the image acquisition device to follow and photograph the second object to obtain more detailed information about the second object.

7. The method according to claim 6, characterized in that, The image acquisition device includes a plurality of cameras; The controlling the image acquisition device to follow and photograph the second object includes: Determine the target camera closest to the second object; Photograph the second object through the target camera.

8. The method according to claim 1, wherein The method further includes: If the degree of difference does not exceed the preset degree, control the image acquisition device to perform image acquisition with a second strategy to obtain image information different from the real image.

9. The method according to claim 8, wherein The entity robot adjusts the shooting angle through the rotation of the image acquisition device; The controlling the image acquisition device to perform image acquisition with a second strategy to obtain image information different from the real image includes: Increase the moving speed of the entity robot and / or increase the rotation speed of the image acquisition device to obtain image information different from the real image.

10. The method according to any one of claims 1-9, characterized in that, Before the step of acquiring the real image collected by the image acquisition device disposed on the entity robot in the real scene, the method further includes: Recognize that the task currently executed by the entity robot is interfered.

11. A robot control device, characterized in that, The device includes: A first acquisition module, configured to acquire a real image collected by an image acquisition device disposed on an entity robot in a real scene; A second acquisition module, configured to acquire a virtual image collected by a virtual robot in a virtual scene, where the virtual scene is a three-dimensional scene constructed according to historical images collected from the real scene, and the virtual robot is a digital twin of the entity robot; A determination module, configured to determine whether the degree of difference between the virtual image and the real image exceeds a preset degree; A first control module, configured to, if the degree of difference exceeds the preset degree, control the image acquisition device to perform image acquisition with a first strategy to obtain more detailed information about the real scene.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-10.

13. An electronic device, characterized in that, Including: A memory, on which a computer program is stored; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1-10.

Images