Method for operating target object, and electronic device, medium and computer program product
By generating control signals in a virtual environment, the robot performs operations in a real-world scenario, solving the problems of low efficiency and insufficient safety in traditional power distribution cabinet operations, and achieving efficient and safe remote operation.
Patent Information
- Application Number
- PCT/CN2023/113837
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-08-18
- Publication Date
- 2025-11-27
AI Technical Summary
Traditional power distribution cabinet operation methods require manual intervention, which is inefficient and unsafe. They cannot be operated remotely, leading to untimely fault handling and potentially serious consequences such as power outages.
By creating a digital twin in a virtual environment and using mixed reality technology to generate control signals, the robot operates in the real scene according to the signals, including adjusting the posture of the robotic arm and recognizing and manipulating target objects.
It enables remote, efficient, and safe operation of the power distribution cabinet, enhances the capabilities of the robot-assisted system, and ensures the safe and efficient operation of the target object.
Smart Images
Figure CN2023113837_27112025_PF_FP_ABST
Abstract
Description
Method for operating target object, electronic device, medium and computer program product TECHNICAL FIELD
[0001] Embodiments of the present application mainly relate to the field of remote control, and in particular to a method for operating a target object, an electronic device, a medium and a computer program product. BACKGROUND
[0002] A power distribution cabinet is a general term for a motor control center that receives energy from a power source and distributes it to different areas or devices, usually as a backbone device in a production manufacturing environment and located in a designated workspace, such as a control room or electrical room. In the traditional method of monitoring, maintaining and operating the power distribution cabinet, the participation of an electrical engineer or technician is usually required, however, this traditional manual method is not only very inefficient but also cannot guarantee safety, in addition, if the engineer is not on site, complex situations such as overload or short circuit failure may occur, which may cause the power distribution cabinet to directly power off, thereby causing a huge financial impact.
[0003] SUMMARY
[0004] Embodiments of the present application provide a method for operating a target object, an electronic device, a medium and a computer program product, which can help users to accurately remotely operate the target object.
[0005] In a first aspect, a method for operating a target object is provided, including: receiving a control signal of a user operating a target object in a preset metaverse environment; wherein the preset metaverse environment includes a digital twin of a real scene; the control signal includes: an action source, time information corresponding to the occurrence of the action, a name of the target object, and action position information; moving to a preset position of the corresponding target object in the real scene according to the name of the target object; adjusting the pose of a mechanical arm according to the action position information; operating the target object in the real scene through the adjusted mechanical arm.
[0006] In a second aspect, an apparatus for operating a target object is provided, including: a receiving module configured to receive a control signal of a user operating a target object in a preset metaverse environment; wherein the preset metaverse environment includes a digital twin of a real scene; the control signal includes: an action source, time information corresponding to the occurrence of the action, a name of the target object, and action position information; a moving module configured to move to a preset position of the corresponding target object in the real scene according to the name of the target object; an adjusting module configured to adjust the pose of a mechanical arm according to the action position information; and an operating module configured to operate the target object in the real scene through the adjusted mechanical arm.
[0007] In a third aspect, an electronic device is provided, comprising: at least one memory configured to store computer readable code; and at least one processor configured to invoke the computer readable code to perform the steps of the method of the first aspect.
[0008] In a fourth aspect, a computer readable medium is provided, having stored thereon computer readable instructions which, when executed by a processor, cause the processor to perform the steps of the method of the first aspect.
[0009] In a fifth aspect, a computer program product is provided, tangibly stored on a computer readable medium and comprising computer executable instructions which, when executed, cause at least one processor to perform the steps of the method of the first aspect. BRIEF DESCRIPTION OF DRAWINGS
[0010] The following drawings are merely schematic and intended to aid in understanding the present embodiments and are not limiting in scope. In the drawings:
[0011] FIG. 1 is a schematic diagram of a digital twin of a real scene according to an embodiment of the present application;
[0012] FIG. 2 is a flowchart of a method of operating a target object according to an embodiment of the present application;
[0013] FIG. 3 is a schematic diagram of an apparatus for operating a target object according to an embodiment of the present application;
[0014] FIG. 4 is a schematic diagram of an electronic device according to an embodiment of the present application.
[0015] REFERENCE NUMERALS 200: method of operating a target object 201-204: method steps 30: apparatus for operating a target object 31: receiving module 32: moving module 33: adjusting module 34: operating module 400: electronic device 401: processor 402: communication interface 403: memory 404: communication bus 405: program DETAILED DESCRIPTION
[0016] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that these implementations are discussed so that a better appreciation of the subject matter described herein can be attained, and are not intended to limit the scope of protection, applicability, or examples set forth in the claims. Changes in the function and arrangement of elements can be made without departing from the scope of the examples contemplated herein. Various examples can omit, substitute, or add various procedures or components as appropriate. For instance, the methods described can be performed in an order different than that described, and / or various steps can be added, omitted, or combined. Also, features described with respect to some examples can be combined in other examples.
[0017] As used herein, the terms "includes," "including," "has," "having," "contains," "containing," "comprises," "comprising," "including," "contains," "containing" and the like are open-ended terms that are intended to mean "including but not limited to." The term "based on" means "based, at least in part, on." The terms "one embodiment" and "an embodiment" mean "at least one embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "a first," "a second," etc. do not require that there be only one of the indicated items. The foregoing description, for purposes of clarity, only describes a select number of example implementations. However, the subject matter described herein is not limited to the implementations described above. Rather, it will be apparent to one skilled in the art from this disclosure that numerous other implementations are also possible.
[0018] Embodiments of the present application create a digital twin of a scene corresponding to an actual environment in a virtual game engine, as shown in FIG. 1, to create a corresponding meta-universe. The meta-universe can be presented using various mixed reality technologies, including but not limited to augmented reality (AR), virtual reality technology (VR), mixed reality (MR), etc., to enable users to interact with the meta-universe on related devices (such as head-mounted devices, portable devices, etc.). Specifically, users can use related devices to simulate controlling target objects such as power equipment in the meta-universe, generate control signals, and these control signals will be received by robots and converted into specific operations of target objects in the real environment, thereby achieving the purpose of remote control.
[0019] Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0020] FIG. 2 is a flowchart of a method of operating a target object according to an embodiment of the present application. As shown in FIG. 2, the method 200 of operating a target object includes:
[0021] In step 201, a control signal of a user operating a target object in a preset metaverse environment is received. The preset metaverse environment includes a digital twin of a real scene. The control signal includes an action source, time information corresponding to an action occurrence, a name of the target object, and action position information.
[0022] Step 201 can be applied to the robot side. Optionally, before step 201, a simulator can also be used to simulate the user's operation of the target object in the preset metaverse to ensure that the movement of the robot in the real scene does not conflict with the surrounding environment, thereby further improving the safety of the machine device and protecting the overall environment. If there is a related conflict, for example, the mechanical arm of the robot may collide with the surrounding equipment, the user is immediately notified of the operation failure through the head-mounted device or other receiving device, and the forwarding of the control signal is suspended at the same time. Conversely, if the simulation shows that the operation can be successfully performed, the control signal of the user operating the target object in the preset metaverse is sent to the server and forwarded to the robot by the server to perform the actual target object operation. Optionally, the conflict between the movement of the robot and the surrounding environment can include but is not limited to: (1) physical collision: the robot or mechanical arm may collide with the surrounding equipment or mechanism. Such a conflict can be due to the mismatch between the position or action path of the robot and the spatial layout of the surrounding environment; (2) safety conflict: the action of the robot can cause the equipment to be overworked, resulting in the action being unable to be implemented.
[0023] Optionally, when the server receives the control signal of the user operating the target object in the preset metaverse and the signal of activating the robot for remote operation, the server sends the control signal of the user operating the target object in the preset metaverse to the robot.
[0024] Optionally, the robot can be composed of an automatic guided vehicle and a mechanical arm. The automatic guided vehicle is responsible for positioning and moving within the site, and the mechanical arm is responsible for fine operation. Optionally, the server can be in the cloud or in the edge to meet different network environments and real-time requirements.
[0025] In step 202, according to the name of the target object, the preset position of the corresponding target object in the real scene is moved to.
[0026] In an embodiment, before step 202, the name of the target object in the real scene and the corresponding preset position information are stored.
[0027] In an embodiment, before step 202, a navigation control system is preset in the real scene to guide the movement of the robot. Then, according to the name of the target object, the guide information in the preset navigation control system is determined. According to the guide information, the preset position of the corresponding target object in the real scene is moved to.
[0028] Optionally, the preset navigation control system in the real scene can be composed of different navigation markers or devices, such as QR (Quick Response) codes, lasers, or magnetic tapes, etc.
[0029] QR code: QR codes can be printed and pasted in specific locations as navigation markers for robots. Each QR code can encode specific location information. When the robot scans the QR code through the camera on it, it can identify the location information corresponding to the QR code, thereby understanding its specific position in the scene.
[0030] Laser: Lasers are commonly used in laser navigation systems. In such systems, lasers emit multiple beams of light into the surrounding environment, and by measuring the time or angle of the reflected light, the robot can determine its own position and the positions of surrounding objects. This method is commonly used for obstacle avoidance, path planning, and precise positioning of robots.
[0031] Magnetic tape: Magnetic tapes are usually laid on the ground to indicate the movement path of the robot. The robot is usually equipped with sensors for sensing magnetic fields. When the robot travels on the magnetic tape, it can track and adjust its travel path by sensing the changes in the magnetic field.
[0032] Step 203, adjusting the pose of the mechanical arm according to the action position information.
[0033] Optionally, according to the action position information, the 6D pose of the mechanical arm is calculated. According to the calculated 6D pose, the mechanical arm is adjusted. Through the visual system on the adjusted mechanical arm, it is judged whether the object pointed to by the user in the preset metaverse environment is the object corresponding to the object pointed to by the user in the real scene. When it is judged that the object pointed to by the user in the preset metaverse environment is the object corresponding to the object pointed to by the user in the real scene, the center position of the object is detected through the visual system. According to the detected center position, the 6D pose of the mechanical arm is adjusted again.
[0034] Optionally, the corresponding 6D pose can be preset according to the positions of several buttons in the real scene. In this way, after receiving the action position information, the mechanical arm can be directly adjusted according to the preset 6D pose, thereby simplifying the operation process and improving the control efficiency.
[0035] Step 204, operating the target object in the real scene through the adjusted mechanical arm.
[0036] In a scene, after adjusting the pose of the mechanical arm, the mechanical arm can press the real button corresponding to the button pressed by the user in the metaverse environment of the power equipment in the real scene, thereby completing the operation in the metaverse.
[0037] Through the embodiments of the present application, an operator can remotely control a target object in a real scene with the assistance of a robot in the metaverse. The embodiments of the present application not only enhance the capabilities of the robot-assisted system, but also promote more effective cooperation between operators and automation technology, thereby ensuring the safe and efficient operation of the target object.
[0038] FIG. 3 is a schematic diagram of an apparatus for operating a target object according to an embodiment of the present application. As shown in FIG. 3, the apparatus 30 for operating a target object includes:
[0039] The receiving module 31 is configured to receive a control signal of a user operating a target object in a preset metaverse environment. The preset metaverse environment includes a digital twin of a real scene. The control signal includes an action source, time information corresponding to the occurrence of the action, the name of the target object, and action position information.
[0040] The moving module 32 is configured to move to a preset position of the corresponding target object in the real scene according to the name of the target object.
[0041] The adjusting module 33 is configured to adjust the pose of the mechanical arm according to the action position information.
[0042] The operating module 34 is configured to operate the target object in the real scene through the adjusted mechanical arm.
[0043] The embodiments of the present application not only enhance the capabilities of the robot-assisted system, but also promote more effective cooperation between operators and automation technology, thereby ensuring the safe and efficient operation of the target object.
[0044] FIG. 4 is a schematic diagram of an electronic device according to an embodiment of the present application. The specific implementation of the electronic device is not limited in the embodiments of the present application. As shown in FIG. 4, the electronic device 400 can include a processor 401, a communications interface 402, a memory 403, and a communications bus 404. Wherein:
[0045] The processor 401, the communications interface 402, and the memory 403 complete mutual communication through the communications bus 404.
[0046] The communications interface 402 is configured to communicate with other electronic devices or servers.
[0047] The processor 401 is configured to execute the program 402, and specifically can execute the related steps in any of the method embodiments.
[0048] Specifically, the program 405 can include program codes including computer operation instructions.
[0049] The processor 401 can be a CPU, or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application. The one or more processors included in the smart device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.
[0050] The memory 403 is configured to store the program 405. The memory 403 can include a high-speed RAM memory, and can further include a non-volatile memory, such as at least one disk memory.
[0051] The program 405 can be specifically configured to cause the processor 401 to perform any one of the plurality of method embodiments in the foregoing embodiments.
[0052] The specific implementation of each step in the program 405 can refer to the corresponding description in the corresponding steps and units of the method embodiments of the foregoing operation target objects, and will not be described herein. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working processes of the devices and modules described above can refer to the corresponding process descriptions in the foregoing method embodiments, and will not be described herein.
[0053] The present application also provides a computer-readable storage medium storing instructions for causing a machine to perform any one of the plurality of method embodiments as described herein. Specifically, a system or apparatus equipped with a storage medium can be provided, and the storage medium stores software program codes implementing the functions of any one of the foregoing embodiments, and causes the computer (or CPU or MPU) of the system or apparatus to read and execute the program codes stored in the storage medium.
[0054] In this case, the program codes read from the storage medium can themselves implement the functions of any one of the foregoing embodiments, and thus the program codes and the storage medium storing the program codes constitute a part of the present application.
[0055] The storage medium for providing the program codes includes a floppy disk, a hard disk, a magneto-optical disk (such as a CD-ROM, a CD-R, a CD-RW, a DVD-ROM, a DVD-RAM, a DVD-RW, a DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program codes can be downloaded from a server computer via a communication network.
[0056] The embodiments of the present application further provide a computer program product comprising computer instructions for instructing a computing device to perform any corresponding operation of the above-mentioned method embodiments.
[0057] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.
[0058] The above-mentioned method according to the embodiments of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium such as a CD ROM, a RAM, a floppy disk, a hard disk or an optical disk, or be downloaded from a network and stored in a local recording medium, so that the method described herein can be processed by such software on a recording medium using a general computer, a special processor or programmable or special hardware such as an ASIC or an FPGA. It can be understood that the computer, the processor, the microprocessor controller or the programmable hardware includes a storage component (for example, RAM, ROM, flash memory, etc.) that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor or the hardware, the method described herein is implemented. In addition, when a general computer accesses the code for implementing the method shown herein, the execution of the code will convert the general computer into a special computer for executing the method shown herein.
[0059] It should be noted that not all steps and modules in the above-mentioned flowcharts and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted according to needs. The system structure described in each embodiment above can be a physical structure or a logical structure, that is, some modules can be implemented by the same physical entity, or some modules can be implemented by multiple physical entities, or can be implemented by some components in multiple independent devices.
[0060] In each of the above embodiments, a hardware module can be implemented mechanically or with electrical means. For example, a hardware module can include a permanent, dedicated circuit or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. A hardware module can also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor) that can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily configured circuit) can be determined based on cost and time considerations.
[0061] The application is illustrated and described in detail above by the accompanying drawings and preferred embodiments, however the application is not limited to these disclosed embodiments, based on the above-mentioned embodiments, those skilled in the art can know that the code review means in different embodiments can be combined to obtain more embodiments of the application, and these embodiments are also within the protection scope of the application.
[0062] The terms and pronouns referring to a person in this patent application are not limited to a specific gender.
Claims
1. A method of operating a target object, comprising: - receiving (201) a control signal of a user operating a target object in a preset metaverse environment; wherein the preset metaverse environment comprises a digital twin of a real scene; the control signal comprises: a motion source, time information corresponding to the occurrence of the motion, a name of the target object, and motion position information; - moving (202) to a preset position of a corresponding target object in the real scene according to the name of the target object; - adjusting (203) the pose of a robot arm according to the motion position information; - operating (204) the target object in the real scene through the adjusted robot arm.
2. The method of claim 1, wherein, Before the receiving (201) a control signal of a user operating a target object in a preset metaverse, the method further comprises: - when the server receives a control signal of a user operating a target object in a preset metaverse, and a signal of activating a robot for remote operation, the server sends the control signal of the user operating the target object in the preset metaverse to the robot.
3. The method of claim 1, wherein, Before the moving (202) to a preset position of a corresponding target object in the real scene according to the name of the target object, the method further comprises: - storing the name of the target object in the real scene and the corresponding preset position information.
4. The method of claim 1, wherein, Before the moving (202) to a preset position of a corresponding target object in the real scene according to the name of the target object, the method further comprises: - presetting a navigation control system in the real scene to guide the movement of the robot; The moving (202) to a preset position of a corresponding target object in the real scene according to the name of the target object comprises: - determining the guide information in the preset navigation control system according to the name of the target object; - moving to the preset position of the corresponding target object in the real scene according to the guide information.
5. The method of claim 1, wherein, The adjusting (203) the pose of a robot arm according to the motion position information comprises: - calculating the 6D pose of the robot arm according to the motion position information; - adjusting the robot arm according to the calculated 6D pose; - determining whether the object pointed by the robot arm is the corresponding object of the first object pointed by the user in the preset metaverse environment in the real scene through a vision system on the adjusted robot arm; When it is determined that the object pointed by the robot arm is the corresponding object of the first object pointed by the user in the preset metaverse environment, - detecting the center position of the object through the vision system; - adjusting the 6D pose of the robot arm again according to the detected center position.
6. An apparatus for operating a target object, comprising: - a receiving module (31) configured to receive a control signal of a user operating a target object in a preset metaverse environment; wherein the preset metaverse environment comprises a digital twin of a real scene; the control signal comprises an action source, time information corresponding to an action occurrence time, a name of the target object, and action position information; - a moving module (32) configured to move to a preset position of a corresponding target object in the real scene according to the name of the target object; - an adjusting module (33) configured to adjust a pose of a mechanical arm according to the action position information; - an operating module (34) configured to operate the target object in the real scene by the adjusted mechanical arm.
7. An electronic device (400) comprising: The processor (401), the communication interface (402), the memory (403), and the communication bus (404) complete communication with each other through the communication bus (404); The memory (403) is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation of the target object method corresponding to any one of claims 1-5.
8. A computer storage medium having a computer program stored thereon, the program being executed by a processor to implement the method for operating a target object according to any one of claims 1-5.
9. A computer program product tangibly stored on a computer readable medium and comprising computer executable instructions that, when executed, cause at least one processor to perform the method for operating a target object according to any one of claims 1-5.