Virtual reality-based robot interaction method, device, equipment and storage medium

Abstract

The application relates to a virtual reality-based robot interaction method, device, equipment and storage medium. The method comprises the following steps: acquiring operation information, controlling a robot to perform corresponding operations on a virtual object according to the operation information; performing virtual-real registration on at least one display device respectively to determine the poses of the display devices relative to a real three-dimensional space; in the process of controlling the robot to perform the corresponding operations, operation animations matching the poses of the display devices are respectively generated according to the poses of the display devices, wherein the operation animations reflect the state change process of the virtual object after receiving the corresponding operations under the corresponding poses; and the operation animations corresponding to the display devices are transmitted to the corresponding display devices for display. Multiple users can see the operation animations matching the poses of the users under different poses, so that the learning efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a method, apparatus, device, and storage medium for robot interaction based on virtual reality. Background Technology

[0002] With the development of computer technology, virtual reality (VR) technology has emerged. VR is a computer simulation system that can create and allow users to experience virtual worlds. It uses computers to generate a simulated environment, immersing users in that environment. It is widely used in industries such as industrial manufacturing, medical applications, aerospace and military, and real estate. VR-based interaction is often involved in scenarios that require displaying virtual objects or observing changes in virtual objects after performing specified operations on them. VR technology provides a risk-free operating environment, reducing security risks.

[0003] Traditional interaction methods typically involve users performing actions on virtual objects, which then provide corresponding feedback. This feedback is usually based on the operator's visual perspective, showing changes in the virtual object. This method only allows interaction between the operator and the virtual object, and other viewers cannot participate in the interaction, resulting in low interaction efficiency. Summary of the Invention

[0004] Therefore, it is necessary to address the aforementioned technical problems by providing a virtual reality-based robot interaction method, device, equipment, and storage medium that enables multiple users to see operation animations matching their own positions from different poses, thereby improving interaction efficiency.

[0005] In a first aspect, this application provides a robot interaction method based on virtual reality, the method comprising:

[0006] Acquire operational information and control the robot to perform corresponding operations on virtual objects based on the operational information;

[0007] At least one display device is registered in both virtual and real modes to determine the pose of each display device relative to the real three-dimensional space.

[0008] During the process of controlling the robot to perform corresponding operations, operation animations matching each pose are generated according to the poses of each display device. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation in the corresponding pose.

[0009] The operation animations corresponding to each display device are transmitted to the corresponding display device for display.

[0010] In one embodiment, virtual-real registration is performed on at least one display device to determine the pose of each display device relative to the real three-dimensional space, including:

[0011] Based on multiple registration markers posted in different locations, at least one display device is registered in both real and virtual modes to determine the registration markers associated with each display device.

[0012] The pose of each display device relative to the real three-dimensional space is calculated based on the orientation of the registration markers associated with each display device.

[0013] In one embodiment, the display device includes a camera, and supplementary markers are disposed around the periphery of the display device. At least one display device is subjected to virtual-real registration based on multiple registration markers affixed to different locations to determine the registration markers associated with each display device, including:

[0014] Acquire images captured by the cameras of each display device, and determine the unobstructed display devices and the obstructed display devices based on the content in the images;

[0015] The image corresponding to the unobstructed display device is processed by registration marker recognition to obtain a first front mark. The registration marker pointed to by the first front mark is used as the registration marker associated with the unobstructed display device.

[0016] The image corresponding to the obscured display device is processed by supplementary marker recognition to obtain a second front marker. The target display device to which the supplementary marker pointed by the second front marker is located is determined. If the target display device is not obscured, the registration marker associated with the target display device is used as the registration marker associated with the obscured display device.

[0017] In one embodiment, the image corresponding to the unobstructed display device is subjected to registration and marker recognition processing to obtain a first front-facing marker, including:

[0018] Image thresholding is performed on the image to obtain multiple connected components;

[0019] A perspective transformation is performed on connected components whose corner numbers and sizes meet preset conditions to obtain the first frontal marker.

[0020] In one embodiment, the display device includes an unobstructed display device, and the pose of each display device relative to the real three-dimensional space is calculated based on the orientation of a registration marker associated with each display device, including:

[0021] Acquire images captured by an unobstructed display device and identify the first two-dimensional position information of the corner points of the registration markers in the images captured by the unobstructed display device;

[0022] Based on the first two-dimensional position information and the first three-dimensional position information of the corner points of the registration marker in the real three-dimensional space, determine the first position transformation matrix;

[0023] Based on the first position transformation matrix, the pose of the unobstructed display device relative to the real three-dimensional space is determined.

[0024] In one embodiment, determining a first position transformation matrix based on the first two-dimensional position information and the first three-dimensional position information of the corner points of the registration marker in real three-dimensional space includes:

[0025] Using the center of the camera on the unobstructed display device as the origin and the plane of the registration marker as the XY plane, a three-dimensional coordinate system is established to obtain the first three-dimensional position information of the corner points of the registration marker in the three-dimensional coordinate system.

[0026] Based on the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the first two-dimensional position information, and the first three-dimensional position information, the first position transformation matrix is ​​determined.

[0027] In one embodiment, the display device further includes an obscured display device, and the pose of each display device relative to the real three-dimensional space is calculated based on the orientation of a registration marker associated with each display device, further including:

[0028] Acquire images captured by an obstructed display device and identify second two-dimensional position information of corner points of supplementary markers in the images captured by the obstructed display device;

[0029] The second position transformation matrix is ​​determined based on the second two-dimensional position information and the second three-dimensional position information of the corner points of the supplementary markers in the real three-dimensional space;

[0030] Based on the first position transformation matrix and the second position transformation matrix, the pose of the occluded display device relative to the real three-dimensional space is determined.

[0031] In one embodiment, determining the second position transformation matrix based on the second two-dimensional position information and the second three-dimensional position information of the corner points of the supplementary markers in the real three-dimensional space includes:

[0032] A three-dimensional coordinate system is established with the center of the camera of the obscured display device as the origin and the plane of the supplementary marker as the XY plane, so as to obtain the second three-dimensional position information of the corner point of the supplementary marker in the three-dimensional coordinate system;

[0033] Based on the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the second two-dimensional position information, and the second three-dimensional position information, the second position transformation matrix is ​​determined.

[0034] Secondly, this application also provides a robot interaction device based on virtual reality, the device comprising:

[0035] The acquisition module is used to acquire operation information and control the robot to perform corresponding operations on virtual objects based on the operation information.

[0036] The registration module is used to perform virtual-real registration on at least one display device to determine the pose of each display device relative to the real three-dimensional space.

[0037] The operation animation generation module is used to generate operation animations that match the poses of each display device during the process of controlling the robot to perform corresponding operations. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation in the corresponding pose.

[0038] The display module is used to transmit the operation animations corresponding to each display device to the corresponding display device for display.

[0039] In one embodiment, the registration module is further configured to perform virtual-real registration on at least one display device based on a plurality of registration markers affixed to different locations, so as to determine the registration markers associated with each display device respectively.

[0040] The pose of each display device relative to the real three-dimensional space is calculated based on the orientation of the registration markers associated with each display device.

[0041] In one embodiment, the display device includes a camera, and supplementary markers are disposed around the periphery of the display device. The registration module is also used to acquire images captured by the cameras of each display device and to determine the unobstructed display devices and the obstructed display devices based on the content in the images.

[0042] The image corresponding to the unobstructed display device is processed by registration marker recognition to obtain a first front mark. The registration marker pointed to by the first front mark is used as the registration marker associated with the unobstructed display device.

[0043] The image corresponding to the obscured display device is processed by supplementary marker recognition to obtain a second front marker. The target display device to which the supplementary marker pointed by the second front marker is located is determined. If the target display device is not obscured, the registration marker associated with the target display device is used as the registration marker associated with the obscured display device.

[0044] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0045] Acquire operational information and control the robot to perform corresponding operations on virtual objects based on the operational information;

[0046] At least one display device is registered in both virtual and real modes to determine the pose of each display device relative to the real three-dimensional space.

[0047] During the process of controlling the robot to perform corresponding operations, operation animations matching each pose are generated according to the poses of each display device. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation in the corresponding pose.

[0048] The operation animations corresponding to each display device are transmitted to the corresponding display device for display.

[0049] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0050] Acquire operational information and control the robot to perform corresponding operations on virtual objects based on the operational information;

[0051] At least one display device is registered in both virtual and real modes to determine the pose of each display device relative to the real three-dimensional space.

[0052] During the process of controlling the robot to perform corresponding operations, operation animations matching each pose are generated according to the poses of each display device. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation in the corresponding pose.

[0053] The operation animations corresponding to each display device are transmitted to the corresponding display device for display.

[0054] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0055] Acquire operational information and control the robot to perform corresponding operations on virtual objects based on the operational information;

[0056] At least one display device is registered in both virtual and real modes to determine the pose of each display device relative to the real three-dimensional space.

[0057] During the process of controlling the robot to perform corresponding operations, operation animations matching each pose are generated according to the poses of each display device. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation in the corresponding pose.

[0058] The operation animations corresponding to each display device are transmitted to the corresponding display device for display.

[0059] The aforementioned virtual reality-based robot interaction method, device, equipment, and storage medium acquire operation information and control the robot to perform corresponding operations on virtual objects based on this information. Virtual-real registration is performed on at least one display device to determine the pose of each display device in real three-dimensional space. Thus, during the process of controlling the robot to perform corresponding operations, operation animations matching the poses of each display device can be generated and transmitted to the corresponding display devices for display. This allows users in different positions to see operation animations matching their own location on the display devices, enabling them to truly understand the state changes of the virtual object after receiving corresponding operations, improving interaction efficiency and enriching interaction methods. Attached Figure Description

[0060] Figure 1 This is an application environment diagram of a virtual reality-based robot interaction method in one embodiment;

[0061] Figure 2 This is a flowchart illustrating a virtual reality-based robot interaction method in one embodiment.

[0062] Figure 3 This is a schematic diagram of the operating device and sensing device of a robot interaction method based on virtual reality in one embodiment;

[0063] Figure 4 This is a schematic diagram illustrating the real-time updating of virtual object data in another embodiment of a robot interaction method based on virtual reality.

[0064] Figure 5 This is a flowchart illustrating the instructional process in one embodiment;

[0065] Figure 6 This is a flowchart illustrating the process of determining the pose of each display device relative to real three-dimensional space in one embodiment.

[0066] Figure 7 This is a schematic diagram of the virtual-to-real registration process of a display device in one embodiment;

[0067] Figure 8 This is a schematic diagram of the Hamming coding principle in one embodiment;

[0068] Figure 9 This is a schematic diagram of registration mark posting in one embodiment;

[0069] Figure 10This is a schematic diagram of the process for determining registration identifiers associated with each display device in one embodiment;

[0070] Figure 11 This is a schematic diagram of supplementary identifiers for the display device in one embodiment;

[0071] Figure 12 This is a schematic diagram of the process for identifying the first frontal marker of the registration identifier in one embodiment;

[0072] Figure 13 This is a flowchart illustrating the pose of an unobstructed display device relative to real three-dimensional space in one embodiment.

[0073] Figure 14 This is a schematic diagram of a camera imaging model in one embodiment;

[0074] Figure 15 This is a schematic diagram illustrating the principle of pose estimation in one embodiment;

[0075] Figure 16 This is a flowchart illustrating the process of calculating the pose of an occluded display device relative to real three-dimensional space in one embodiment.

[0076] Figure 17 This is a detailed flowchart of a virtual reality-based robot interaction method in one embodiment;

[0077] Figure 18 This is a schematic diagram of the operation prompt function in one embodiment;

[0078] Figure 19 This is a schematic diagram of the operation instruction function in one embodiment;

[0079] Figure 20 This is a schematic diagram of the operation warning function in one embodiment;

[0080] Figure 21 This is a structural block diagram of a virtual reality-based robot interaction device in one embodiment;

[0081] Figure 22 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0082] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0083] The robot interaction method based on virtual reality provided in this application can be applied to, for example... Figure 1In the application environment shown, the operating device 102 is installed on the intelligent robot or other device independent of the intelligent robot. When a user wearing a display device 108 operates the operating device 102, the operating device 102 sends the operation result and / or operation information to the processing unit 104 via the network. The sensing device 106 is installed on the intelligent robot or other location that can sense the user's operation information. The sensing device 106 collects the user's operation information and transmits it to the processing unit 104. The processing unit 104 obtains the operation information and controls the robot to perform corresponding operations on the virtual object according to the operation information. Virtual-real registration is performed on at least one display device to determine the pose of each display device relative to the real three-dimensional space. During the process of controlling the robot to perform corresponding operations, operation animations matching each pose are generated according to the pose of each display device. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation under the corresponding pose. The operation animations corresponding to each display device are transmitted to the corresponding display devices for display. Processing unit 104 establishes a connection with remote server 110 and sends an access request to remote server 110. After successful authentication, remote server 110 transmits the operation instructions from the remote service engineer to processing unit 104. Upon receiving the operation instructions, processing unit 104 controls the robot to perform corresponding operations on the virtual object and transmits the corresponding operation animations to the respective display devices for display. Processing unit 104 is a processor in a computer device, which can be, but is not limited to, various personal computers, laptops, smartphones, tablets, intelligent robots, and portable wearable devices. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. For example, when the computer device is an intelligent robot, the processing unit can specifically be the central control unit of the intelligent robot. Remote server 110 can be implemented using a standalone server or a server cluster composed of multiple servers.

[0084] Existing methods for operating robots using virtual reality typically involve the user performing an action, followed by corresponding feedback from the virtual robot. For example, in medical robot teaching scenarios, current methods usually involve the user performing actions on a virtual medical robot, which then provides feedback, or the robot supplier provides teaching materials. These existing interactive methods cannot actually operate a real medical robot, are limited in scope, and have low efficiency. To address these issues, in one embodiment, such as... Figure 2 As shown, a robot interaction method based on virtual reality is provided, which can be applied to... Figure 1 Taking the processing unit 104 as an example, the following steps are included:

[0085] S202, Obtain operation information, and control the robot to perform corresponding operations on the virtual object according to the operation information.

[0086] When a user performs instructional operations through the operating device, operational information is generated. This operating device can provide prompts, instructions, and warnings, displayed in formats including images, text, sound, and vibration. In this embodiment, the operating device can be a control handle or control unit for the intelligent robot, or it can be an independent control unit installed on the intelligent robot.

[0087] like Figure 3 As shown, this embodiment acquires operation information through a sensing device, which includes a voice recognition unit, a machine vision unit, and a force feedback unit. The voice recognition unit is used to acquire the user's voice commands; the machine vision unit is used to determine the operation completed by the user; and the force feedback unit is used to detect the force applied by the user when operating the device. The sensing device communicates with the processing unit and transmits the user's operation information to the processing unit. In this embodiment, the sensing device can be a sensing unit composed of a camera, microphone, and force sensor attached to the operating device, capable of sensing and capturing the user's actions, voice, and controls.

[0088] Specifically, when a user performs instructional operations through the operating device, the processing unit acquires operation information through the sensing device and feeds this information back to the processing unit. The processing unit then updates the user interaction content, operation status, and virtual object status in real time based on the user's operation information. The virtual object status includes the virtual object's pose and operation state. The operation status includes the pre-operation planning state and the in-operation guidance state.

[0089] S204, Perform virtual-real registration on at least one display device to determine the pose of each display device relative to the real three-dimensional space.

[0090] The display device is connected to the processing unit and provides the user with virtual visual information highly coupled with the real environment. The display device can be a head-mounted display, worn by the user. When the user's position changes, the display device needs to be reconfigured to determine its pose relative to the real three-dimensional space. The number of display devices is not limited and can be set according to actual needs.

[0091] Virtual-real registration means: calculating the position and posture information of the user's viewpoint relative to the real world in real time and accurately, and using this information to correctly overlay the virtual scene onto its proper position.

[0092] Specifically, the processing unit identifies the image captured by the display device, including the registration marker, and obtains the front marking of the registration marker, thus completing the virtual-real registration of the display device. The orientation of the display device can be determined based on the front marking of the registration marker. For example, if the registration marker is affixed to the front of the processing unit, and the display device and the registration marker on the front of the processing unit are virtual-real registered, it indicates that the display device is located within a position range where the front of the processing unit can be captured.

[0093] S206, During the process of controlling the robot to perform corresponding operations, operation animations matching each pose are generated according to the poses of each display device. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation in the corresponding pose.

[0094] Among them, such as Figure 4 As shown, the virtual images at different positions and angles are different during the process of the processing unit controlling the robot to perform corresponding operations. Therefore, the processing unit generates operation animations that match the poses of the display devices at different positions.

[0095] It is important to note that the process of generating operation animations that match the poses of the display devices at different locations is as follows: During the process of controlling the robot to perform the corresponding operation, the processing unit generates a global 3D change map of the virtual object, determines the visual range corresponding to each display device based on the pose of each display device, and displays the state changes of the virtual object within the visual range of each display device.

[0096] S208, transmits the operation animation corresponding to each display device to the corresponding display device for display.

[0097] Specifically, the communication architecture between the display device, the operating device, the sensing device, and the processing unit is as follows: Figure 1 As shown, the processing unit calculates the pose of each display device relative to the real space and generates virtual guidance information and virtual object images that match each pose. The virtual guidance information and virtual object images are sent to the corresponding display devices. The user controls the operation device according to the virtual guidance information on the display device. The operation device and the sensing device capture the user's operation information and position, and feed the operation information back to the processing unit. The processing unit generates operation animations that match each pose according to the operation information and pose of each display device, and transmits the operation animations corresponding to each display device to the corresponding display device for display.

[0098] In the aforementioned virtual reality-based robot interaction method, operation information is acquired, and the robot is controlled to perform corresponding operations on virtual objects based on this information. Virtual-real registration is performed on at least one display device to determine the pose of each device in real 3D space. Thus, during the process of controlling the robot to perform corresponding operations, operation animations matching the poses of each display device can be generated and transmitted to the corresponding display devices for display. This allows users in different locations to see operation animations matching their own location on the display devices, enabling them to truly understand the state changes of the virtual object after receiving corresponding operations, improving interaction efficiency and enriching interaction methods.

[0099] In one embodiment, the processing unit updates the user interaction content, operation status, and virtual object status in real time based on the user's operation information. The virtual object status includes the virtual object pose and virtual object operation status. The operation status includes the pre-operation planning state and the in-operation guidance state. The pre-operation planning state and the in-operation guidance state are as follows: Figure 5 As shown. Before operation, the user sequentially completes the steps of determining the operation type, identifying virtual object information, and planning the operation. During operation, the processing unit guides the user step-by-step through the robot's actions, and warns the user when necessary that the operation is approaching safety limits. After operation, the processing unit summarizes the operation and saves the information, including audio, video, and the user's actions at each step. Additionally, the user can request a remote engineer to participate in the virtual operation at any stage.

[0100] In one embodiment, such as Figure 6 As shown, virtual-real registration is performed on at least one display device to determine the pose of each display device relative to the real three-dimensional space, including the following steps:

[0101] S602, performing virtual-real registration on at least one display device based on multiple registration marks affixed to different locations, to determine the registration marks associated with each display device respectively.

[0102] The flowchart for virtual-real registration is as follows: Figure 7 As shown, the processing unit acquires images captured by the camera of the display device or extracts video frames from the video recorded by the camera. The processing unit detects registration markers in the images or video frames, determines the position of the display device relative to the registration markers, identifies the front markings of the registration markers to complete virtual-real registration, and finally calculates the pose of each display device relative to the registration markers. According to the pose of each display device, the processing unit generates operation animations that match each pose and transmits the operation animations corresponding to each display device to the corresponding display device for display.

[0103] The registration marker can use Hamming codes based on the ARTag tag system, or other graphic codes such as QR codes and barcodes; this application does not limit this. ARTag uses edge detection to locate the marker, providing some resistance to occlusion. The feature graphic uses CRC (Cyclic Redundancy Check) encoding, and the recognition process is actually a decoding process, greatly improving recognition efficiency. Drawing inspiration from ARTag, the digital 0 / 1 codes are converted into square markers composed of black and white blocks as the registration marker. Figure 8 As shown, the registration marker is divided into a 7x7 grid, with black and white squares represented by 0 and 1 respectively. A black ring around the registration marker facilitates its detection. The internal 5x5 grid is the valid area. Each row consists of 5 bits: bits 1, 3, and 5 are used for error checking to determine the correct orientation of the marker; bits 2 and 4 are information bits, storing valid data. Therefore, each row of 5 bits represents 4 types of data (00 / 01 / 10 / 11), for a total of 4... ^5 = 1024. To prevent a completely black rectangle from becoming a valid marker, the first bit of the Hamming code is inverted. If the code is 0 (Hamming code is 00000), then the front marker ID is 10000. The Hamming code does not have rotational symmetry; the positive orientation of the registration marker can be determined by calculating the Hamming distance. Therefore, the decoded information is unique. Figure 8 The valid ID of the registration identifier shown is 0000001010, which is 10.

[0104] Registration markers should be affixed to locations that are not easily obstructed, such as on navigation carts, robots, or processing units. Figure 9 As shown, the registration markers are affixed to the three vertex faces of the processing unit, the three adjacent faces of the navigation vehicle, or other suitable affixing locations. Each affixed face has a fixed positional relationship, and the relative positions between the affixed faces also have a fixed positional relationship.

[0105] Specifically, multiple registration markers are affixed to locations on the robot or processing unit that are not easily obscured, such as... Figure 9As shown, registration markers are affixed to the three apex faces of the processing unit, the three adjacent faces of the navigation vehicle, or other suitable locations. Each registration marker has a different marking. The processing unit can determine the location of each display device relative to the registration marker by identifying the front marking of the registration marker in the images captured by each display device, and the registration marker pointed to by the front marking is the registration marker associated with the display device. For example, if three registration markers are affixed to the front, top, and right sides of the processing unit, and the front markings of the registration markers on the front, top, and right sides are the numbers 10, 20, and 30 respectively, and the processing unit identifies the front marking of the registration marker in the image captured by display device A as the number 10, then it indicates that display device A is located within the range where the front of the processing unit can be captured.

[0106] S604 calculates the pose of each display device relative to the real three-dimensional space based on the orientation of the registration markers associated with each display device.

[0107] In this process, after the processing unit determines the orientation of the registration marker associated with each display device, it can determine the two-dimensional and three-dimensional coordinates of the same point within that orientation, as well as the transformation relationship between the two-dimensional and three-dimensional coordinate systems, to know the pose of each display device relative to the real three-dimensional space.

[0108] In this embodiment, multiple registration markers are affixed at different locations, each with a different front marking. By identifying the front markings of the registration markers in the images captured by each display device, the location of each display device relative to the registration markers can be determined. Then, by combining the two-dimensional and three-dimensional coordinates of the same point within that location, as well as the transformation relationship between the two-dimensional and three-dimensional coordinate systems, the pose of each display device relative to the real three-dimensional space can be determined, providing coordinate information for generating operation animations that match each pose.

[0109] In one embodiment, in a multi-person teaching scenario, a display device may be obstructed by an obstacle, preventing it from capturing registration markers or causing the registration markers to be obstructed. In this case, the obstructed display device cannot determine its associated registration marker, thus preventing the processing unit from knowing the pose of the obstructed display device and thus preventing the transmission of the operation animation matching each pose to the display device. Therefore, to solve the above problem, such as Figure 10 As shown, at least one display device is registered in both real and virtual modes based on multiple registration markers posted in different locations to determine the registration markers associated with each display device, including the following steps:

[0110] S1002, acquire images captured by the cameras of each display device, and determine the unobstructed display devices and the obstructed display devices based on the content in the images.

[0111] The display device includes a camera. The processing unit determines whether the display device is obstructed by judging whether the image captured by the camera of each display device contains registration markers. If the image captured by the camera of the display device contains registration markers, it is determined that the display device is not obstructed; if the image captured by the camera of the display device does not contain registration markers, it is determined that the display device is obstructed.

[0112] S1004, perform registration marker recognition processing on the image corresponding to the unobstructed display device to obtain a first front mark, and use the registration marker pointed to by the first front mark as the registration marker associated with the unobstructed display device.

[0113] When the image captured by the display device contains only one complete registration marker, it means that the display device is positioned in front of the registration marker. Identifying the registration marker in the image captured by the display device allows us to obtain the first frontal marking of the registration marker. When the image captured by the display device contains multiple or incomplete registration markers, it means that the current position of the display device cannot be determined. The display device needs to be moved until only one complete registration marker is visible in the image captured by the display device.

[0114] Specifically, the processing unit performs registration marker recognition processing on the image corresponding to the unobstructed display device. If the image corresponding to the unobstructed display device contains multiple or incomplete registration markers, the unobstructed display device is moved until there is only one complete registration marker in the image captured by the unobstructed display device. The first front mark of the registration marker in the image corresponding to the unobstructed display device is identified. The processing unit determines the position of the unobstructed display device on the registration marker based on the first front mark, and the registration marker pointed to by the first front mark is the registration marker associated with the unobstructed display device.

[0115] S1006, perform supplementary marker recognition processing on the image corresponding to the obscured display device to obtain a second front marker, determine the target display device where the supplementary marker pointed to by the second front marker is located, and if the target display device is not obscured, use the registration marker associated with the target display device as the registration marker associated with the obscured display device.

[0116] In multi-person scenarios, if a display device is obstructed by other objects during the registration process, preventing it from capturing the registration marker, the obstructed display device cannot determine the associated registration marker. Consequently, the processing unit cannot determine the pose of the obstructed display device and cannot transmit the operation animation matching each pose to the display device. To solve this problem, such as... Figure 11As shown, a supplementary marker 112 is provided on the periphery of the display device. The front markings of the supplementary marker are different from those of the registration marker, and the supplementary markers are also different for different display devices. The marker is located on the display device, and the display position is within the visible range of the obscured display device.

[0117] In some embodiments, after determining that a display device is obstructed, the obstructed display device immediately sends a signal, which may be an electromagnetic wave, sound, or other form of signal. Upon receiving the signal, the indicator lights of the unobstructed display devices illuminate. The user wearing the obstructed display device can determine the target display device based on the indicator lights and adjust their current position according to the location of the target display device until the obstructed display device can capture a supplementary marker on the target display device. The captured image, including the supplementary marker affixed to the unobstructed display device, is then transmitted to the processing unit. The processing unit identifies the front marking of the supplementary marker and determines the target display device where the supplementary marker is located based on the second front marking. If the target display device is not obstructed, the registration marker associated with the target display device is used as the registration marker associated with the obstructed display device.

[0118] In some embodiments, after determining that the display device is obstructed, the obstructed display device transmits an image including a supplementary marker to the processing unit. The processing unit identifies the front marking of the supplementary marker and determines the target display device where the supplementary marker is located based on the second front marking. The processing unit determines whether the target display device is obstructed based on the state of the target display device. If the target display device is obstructed, the user wearing the obstructed display device is prompted to move the camera's viewing angle and re-capture an image including the supplementary marker. The processing unit re-acquires images including the supplementary marker taken from different angles and identifies the front marking of the supplementary marker. The processing unit determines the target display device where the supplementary marker is located based on the second front marking. The processing unit again determines whether the target display device is obstructed, and repeats the above process until the target display device is not obstructed. If the target display device is not obstructed, the front marking of the supplementary marker is identified, and the processing unit determines the target display device where the supplementary marker is located based on the second front marking. If the target display device is not obstructed, the registration marker associated with the target display device is used as the registration marker associated with the obstructed display device.

[0119] By supplementing the front markings of the registration markers, not only can further registration with the obstructed display device be achieved—that is, the position of the obstructed display device relative to the registration markers can be calculated using the registration markers on the unobstructed display device—but also the orientation of the obstructed display device relative to the registration markers can be determined. In this embodiment, eight fixed positions are provided on each display device for displaying the registration markers, evenly distributed around the perimeter of the display device.

[0120] Specifically, when a display device is obstructed, the obstructed display device immediately sends a signal. Upon receiving the signal, the indicator lights of the unobstructed display devices illuminate. The processing unit acquires an image captured by the obstructed display device, including supplementary markers on the unobstructed display devices. If the image corresponding to the obstructed display device contains multiple or incomplete supplementary markers, the user is prompted to move. The position of the obstructed display device changes as the user moves until only one complete supplementary marker appears in the image captured by the obstructed display device. The processing unit identifies the supplementary marker and obtains a second frontal mark of the supplementary marker. Based on the second frontal mark, the processing unit determines the target display device where the supplementary marker is located. If the target display device is not obstructed, the registration marker associated with the target display device is used as the registration marker associated with the obstructed display device. For example, if the image captured by the obscured display device A includes a supplementary identifier for the unobscured display device, the supplementary identifier is identified, and a second front mark of the supplementary identifier is obtained. This second front mark has a value of 40. Based on the pre-set attachment relationship between the supplementary identifier and the display device, the target display device where the supplementary identifier is located is determined to be the unobscured display device B. According to the above embodiment, the registration identifier associated with the unobscured display device B is used as the registration identifier associated with the obscured display device A.

[0121] In this embodiment, multiple supplementary markers are affixed to the display device. Each supplementary marker has a unique front marking. When the display device is obstructed, the image corresponding to the obstructed display device is processed for supplementary marker recognition to obtain a second front marking. The target display device to which the supplementary marker pointed by the second front marking is located is determined. If the target display device is not obstructed, the registration marker associated with the target display device is used as the registration marker associated with the obstructed display device. Through the above method, not only can the obstructed display device indirectly achieve virtual-real registration with the registration marker through the supplementary marker of the unobstructed display device, that is, based on the virtual-real registration between the unobstructed display device and the registration marker, and the virtual-real registration between the obstructed display device and the supplementary marker of the unobstructed display device, the orientation of the obstructed display device relative to the registration marker can also be determined, thereby knowing the pose of the obstructed display device. The operation animation matching each pose is transmitted to the obstructed display device for display.

[0122] In one embodiment, the methods for identifying the registration marker and the supplementary marker can be the same or different. This embodiment sets the methods for identifying the registration marker and the supplementary marker to be the same; therefore, only the method for identifying the registration marker is described here. For example... Figure 12 As shown, the image corresponding to the unobstructed display device is registered and identified to obtain a first front-facing mark, including the following steps:

[0123] S1202, perform image thresholding segmentation on the image to obtain multiple connected components.

[0124] Here, the image refers to an image including registration markers captured by an unobstructed display device. This embodiment uses Hamming codes based on the ARTag tagging system as the registration markers. The principle of Hamming codes has been explained in the above embodiments and therefore will not be repeated here.

[0125] Specifically, the processing unit acquires an image including registration markers taken by the camera of the unobstructed display device, preprocesses the image (e.g., smoothing, edge enhancement), and performs threshold segmentation on the preprocessed image to obtain multiple connected components.

[0126] S1204, perform perspective transformation on connected regions whose corner numbers and sizes meet preset conditions to obtain the first frontal marker.

[0127] In some cases, images captured by the display device at certain angles may contain multiple or incomplete registration markers, making it difficult to identify them. Therefore, before identifying the front markings of the registration markers, images that do not meet the requirements need to be filtered to obtain registration markers that can be used for identification. This embodiment determines whether a connected component meets the requirements based on whether the number and size of its corner vertices meet preset conditions. The preset conditions are that the connected component has 4 corner vertices and its size conforms to a preset range. The lower limit of the preset range is the minimum area of ​​the connected component with 4 corner vertices, and the upper limit is the maximum area of ​​the connected component with 4 corner vertices.

[0128] Specifically, the processing unit performs connected component analysis and contour detection on connected components. For connected components whose corner count and size meet preset conditions, perspective transformation is performed. The connected components are imaged in the camera, and the perspective transformation matrix is ​​obtained. This perspective transformation is then applied to the connected components to obtain a front view of the registration marker. This front view is then binarized to obtain a binarized image for extracting information from the registration marker. During decoding, the processing unit first determines whether the registration marker is complete. This can be done by detecting whether the outer ring of the registration marker is completely black. The standard for detection is whether the encoding of the outer ring of the registration marker is 0. If it is, the outer ring of the registration marker is completely black, and the registration marker is complete; otherwise, the registration marker is incomplete, and the image is not a registration marker. Then, the central encoding region of the complete registration marker is identified, and the Hamming distance of the encoding region is calculated to obtain the positive direction of the registration marker. Finally, the registration marker is decoded by traversing each row of the central encoding region to obtain the first frontal marker.

[0129] In this embodiment, preset conditions are set according to the number and size of corner points of the connected domain to filter images captured by the front of the display device that include a complete registration marker. By recognizing the image, the first front mark of the registration marker can be obtained. By filtering the registration markers in the images captured by the display device, the registration efficiency of the registration markers is improved.

[0130] In one embodiment, the display device includes an unobstructed display device. After obtaining the first frontal marker of the registration identifier in the image corresponding to the unobstructed display device through the above embodiment, as shown... Figure 13 As shown, based on the orientation of the registration markers associated with each display device, the pose of each display device relative to the real three-dimensional space is calculated, including:

[0131] S1302, acquire an image captured by an unobstructed display device, and identify the first two-dimensional position information of the corner point of the registration marker in the image captured by the unobstructed display device.

[0132] The registration markers use Hamming codes, and the effective area of ​​the Hamming code is rectangular, with the corner points of the registration markers being the corner points of the rectangle. For example... Figure 14 As shown, the camera coordinate system is a coordinate system with the optical center of the camera as the origin. The z-axis is the principal optical axis, generally pointing in the direction of the camera's shooting; the horizontal direction is the x-axis, pointing to the right; and the vertical direction is the y-axis, satisfying the right-hand rule. The coordinate system is denoted as (X... m Y m Z m ) ^T The camera coordinate system is denoted as (X). c Y c Z c ) ^T The transformation matrix between the camera coordinate system and the actual screen coordinate system is called the intrinsic parameter matrix, denoted by K. It is obtained through camera calibration, and the actual screen coordinate system is a planar coordinate system. In other words, the first two-dimensional position information can be determined through the intrinsic parameter matrix K. The first two-dimensional position information is the two-dimensional coordinate of the corner point of the registration marker in the image captured by the unobstructed display device in the actual screen coordinate system of the camera. The first two-dimensional position information represents the row and column information of the corner point image.

[0133] Specifically, the processing unit acquires an image captured by an unobstructed display device, performs image thresholding on the image to obtain multiple connected components, performs perspective transformation on connected components whose corner point count and size meet preset conditions to obtain a front view of the registration marker, performs binarization processing on the front view of the registration marker to obtain a binarized image, traverses the binarized image to extract the effective area of ​​the Hamming code, determines the first two-dimensional position information of the four corner points of the effective area in the actual screen coordinate system of the camera based on the intrinsic parameter matrix K, and determines the target corner point from the four corner points.

[0134] S1304, determine the first position transformation matrix based on the first two-dimensional position information and the first three-dimensional position information of the corner points of the registration marker in the real three-dimensional space.

[0135] The first position transformation matrix represents the transformation relationship between the camera coordinate system and the marker coordinate system of the unobstructed display device. The first position transformation matrix is ​​also called the extrinsic parameter matrix, denoted as T. cm This includes the rotation matrix R and the translation vector T, T cm The mathematical expression is as follows:

[0136]

[0137] This step includes the following steps:

[0138] S1. Using the center of the camera on the unobstructed display device as the origin and the plane of the registration marker as the XY plane, establish a three-dimensional coordinate system to obtain the first three-dimensional position information of the corner points of the registration marker in the three-dimensional coordinate system.

[0139] S2, determine the first position transformation matrix based on the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the first two-dimensional position information, and the first three-dimensional position information.

[0140] The transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system is: m = KT cm M. Where m is the camera planar coordinate and M is the 3D spatial coordinate. The pose from 2D to 3D space can be estimated using the known corner information from the registration markers, i.e., finding the first position transformation matrix T. cm .like Figure 15 As shown, O is the camera center, M i It is the corner point of the registration marker in the coordinate system, m i It is M i The corresponding point projected onto the actual camera screen G can be determined by using the corner point information of the registration marker. i The first two-dimensional position information; a three-dimensional coordinate system is established with the camera center as the origin and the plane of the registration marker as the XY plane, and the corner points M of the registration marker are obtained. i The first three-dimensional position information is obtained in the three-dimensional coordinate system. Finally, based on the known transformation relationship between the three-dimensional and two-dimensional coordinate systems, the first two-dimensional position information, and the first three-dimensional position information, the first position transformation matrix is ​​determined.

[0141] S1306, determine the pose of the unobstructed display device relative to the real three-dimensional space according to the first position transformation matrix.

[0142] Given the first two-dimensional position information, the first three-dimensional position information, and the first position transformation matrix of the corner points of the registration marker, the pose of the unobstructed display device relative to the real three-dimensional space can be calculated based on the multi-point perspective PnP function.

[0143] In this embodiment, the first two-dimensional position information of the corner points of the registration marker in the image captured by the unobstructed display device is identified. Based on the first two-dimensional position information and the first three-dimensional position information of the corner points of the registration marker in the real three-dimensional space, the first position transformation matrix is ​​determined. Based on the known first two-dimensional position information, first three-dimensional position information and first position transformation matrix of the corner points of the registration marker, the pose of the unobstructed display device relative to the real three-dimensional space can be calculated based on the multi-point perspective PnP function.

[0144] In one embodiment, the display device includes an obscured display device. After obtaining a first front mark of the registration identifier and a second front mark of the supplementary identifier associated with the obscured display device through the above embodiments, as follows: Figure 16 As shown, the pose of each display device relative to the real three-dimensional space is calculated based on the orientation of the registration markers associated with each display device, including the following steps:

[0145] S1602, acquire the image captured by the obstructed display device, and identify the second two-dimensional position information of the corner point of the supplementary marker in the image captured by the obstructed display device.

[0146] The image captured by the obstructed display device includes supplementary markers from the unobstructed display device. In this embodiment, the supplementary markers also employ Hamming encoding. The principle of Hamming encoding has been explained in the above embodiments and will not be repeated here. The method for obtaining the second two-dimensional location information is the same as the method for obtaining the first two-dimensional location information, and will not be repeated here.

[0147] It should be noted that the second two-dimensional position information is the two-dimensional coordinates of the corner point of the supplementary marker on the unobstructed display device.

[0148] S1604. Determine the second position transformation matrix based on the second two-dimensional position information and the second three-dimensional position information of the corner points of the supplementary markers in the real three-dimensional space.

[0149] The second position transformation matrix represents the transformation relationship between the camera coordinate system of the obscured display device and the camera coordinate system of the unobscured display device. The mathematical expression of the second position transformation matrix is ​​the same as that of the first position transformation matrix, and will not be repeated here.

[0150] This step includes the following steps:

[0151] S1: Establish a three-dimensional coordinate system with the center of the camera of the obscured display device as the origin and the plane of the supplementary marker as the XY plane, and obtain the second three-dimensional position information of the corner point of the supplementary marker in the three-dimensional coordinate system.

[0152] It should be noted that, unlike the embodiment described above for determining the first position transformation matrix, the origin of the three-dimensional coordinate system in this step is the center of the camera of the obscured display device, and the XY plane is the plane of the supplementary marker.

[0153] S2. Based on the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the second two-dimensional position information, and the second three-dimensional position information, determine the second position transformation matrix.

[0154] It should be noted that the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system is the same as that in the above embodiment, and will not be repeated here. This embodiment differs from the above embodiment for determining the first position transformation matrix in that the second two-dimensional position information and the second three-dimensional position information in this embodiment are different from the first two-dimensional position information and the first three-dimensional position information.

[0155] S1606, Based on the first position transformation matrix and the second position transformation matrix, determine the pose of the occluded display device relative to the real three-dimensional space.

[0156] The product of the first position transformation matrix and the second position transformation matrix is ​​the position transformation matrix of the occluded display device relative to the registration marker. Given the second two-dimensional position information and the second three-dimensional position information of the corner points of the supplementary marker, as well as the position transformation matrix of the occluded display device relative to the registration marker, the pose of the occluded display device relative to the real three-dimensional space can be calculated according to the multi-point perspective PnP function.

[0157] In this embodiment, the obscured display device indirectly achieves virtual-real registration with the registration marker through a supplementary marker of the unobscured display device. After determining the orientation of the obscured display device relative to the registration marker, the second two-dimensional position information of the corner points of the supplementary marker in the image captured by the obscured display device is obtained. A three-dimensional coordinate system is established with the center of the camera of the obscured display device as the origin and the plane of the supplementary marker as the XY plane. The second three-dimensional position information of the corner points of the supplementary marker in the three-dimensional coordinate system is obtained. Based on the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the second two-dimensional position information, and the second three-dimensional position information, a second position transformation matrix is ​​determined. Based on the first position transformation matrix and the second position transformation matrix, the pose of the obscured display device relative to the real three-dimensional space is determined. Through the above method, the position transformation matrix between the obscured display device and the registration marker can be indirectly calculated using the first position transformation matrix between the unobscured display device and the registration marker, and the second position transformation matrix between the obscured display device and the supplementary marker of the unobscured display device, thereby determining the pose of the obscured display device relative to the real three-dimensional space.

[0158] In one embodiment, such as Figure 17 As shown, this embodiment provides a robot interaction method based on virtual reality, specifically including the following steps:

[0159] S1702, Obtain operation information, and control the robot to perform corresponding operations on the virtual object according to the operation information.

[0160] S1704: Obtain images captured by the cameras of each display device, and determine whether the display device is an obstructed display device based on the content in the images; if the display device is an unobstructed display device, then execute S1706; if the display device is an obstructed display device, then execute S1716.

[0161] S1706, perform registration marker recognition processing on the image corresponding to the unobstructed display device to obtain a first front mark, and use the registration marker pointed to by the first front mark as the registration marker associated with the unobstructed display device.

[0162] S1708, Identifying the first two-dimensional position information of the corner point of the registration marker in an image captured by an unobstructed display device.

[0163] S1710, with the center of the camera of the unobstructed display device as the origin and the plane of the registration mark as the XY plane, establish a three-dimensional coordinate system to obtain the first three-dimensional position information of the corner point of the registration mark in the three-dimensional coordinate system.

[0164] S1712, Based on the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the first two-dimensional position information, and the first three-dimensional position information, determine the first position transformation matrix.

[0165] S1714, determine the pose of the unobstructed display device relative to the real three-dimensional space according to the first position transformation matrix, and execute S1726.

[0166] S1716, perform supplementary marker recognition processing on the image corresponding to the obscured display device to obtain a second front marker, determine the target display device where the supplementary marker pointed to by the second front marker is located, and if the target display device is not obscured, then use the registration marker associated with the target display device as the registration marker associated with the obscured display device.

[0167] S1718, identifying second two-dimensional position information of corner points of supplementary markers in an image captured by an obscured display device.

[0168] S1720: A three-dimensional coordinate system is established with the center of the camera of the obscured display device as the origin and the plane of the supplementary marker as the XY plane, so as to obtain the second three-dimensional position information of the corner point of the supplementary marker in the three-dimensional coordinate system.

[0169] S1722, Based on the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the second two-dimensional position information, and the second three-dimensional position information, determine the second position transformation matrix.

[0170] S1724, based on the first position transformation matrix and the second position transformation matrix, determine the pose of the occluded display device relative to the real three-dimensional space, and execute S1726.

[0171] S1726, In the process of controlling the robot to perform corresponding operations, according to the poses corresponding to each display device, operation animations matching each pose are generated respectively. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation under the corresponding pose.

[0172] S1728, transmit the operation animation corresponding to each display device to the corresponding display device for display.

[0173] In this embodiment, users in different locations can see operation animations that match their own location through the display device, so as to truly understand the change process of the virtual object after receiving the corresponding operation, thus enriching the interaction.

[0174] By affixing multiple supplementary markers to the display device, each marker has a unique front marking. When the display device is obstructed, the image corresponding to the obstructed display device is processed for supplementary marker recognition to obtain a second front marking. The target display device to which the supplementary marker pointed by the second front marking is located is determined. If the target display device is not obstructed, the registration marker associated with the target display device is used as the registration marker associated with the obstructed display device. Through the above method, the obstructed display device indirectly achieves virtual-real registration with the registration marker. The processing unit determines the orientation of the obstructed display device relative to the registration marker based on the virtual-real registration between the unobstructed display device and the registration marker, as well as the virtual-real registration between the obstructed display device and the supplementary marker of the unobstructed display device. This allows the unit to determine the pose of the obstructed display device and transmit the operation animation matching each pose to the obstructed display device for display.

[0175] In one embodiment, such as Figure 18 The diagram illustrates an operation prompt function. When the user is not active, or in other scenarios requiring prompting, the operation prompt function page is displayed on the display page of the processing unit or the display device. For example, if the user has not performed any operation for an extended period, the operation prompt function page will indicate that the display page of the processing unit will enter a sleep state.

[0176] In one embodiment, such as Figure 19As shown, a schematic diagram of the operation instruction function is provided. An operation instruction function page is formed on the display page of the processing unit or the display page of the display device. Before, during, or after operation, the operation instruction function page guides the user through each step of the operation. For example, before operation, the operation instruction function page displays the operation steps and information about the virtual object. The information about the virtual object may include the virtual identity information and virtual lesion information of the virtual object. Figure 19 The "Please click xxx in the box" control is used to guide users to trigger the operation instruction page.

[0177] In one embodiment, such as Figure 20 The diagram illustrates the operation warning function. During virtual operation, a warning message is displayed when the operation triggers a warning condition. For example, if the safety boundary is exceeded, the display page on the processing unit or display device turns red, and simultaneously emits a warning sound and vibrates the head-mounted device to warn the user.

[0178] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0179] Based on the same inventive concept, this application also provides a virtual reality-based robot interaction device for implementing the virtual reality-based robot interaction method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the virtual reality-based robot interaction device provided below can be found in the limitations of the virtual reality-based robot interaction method described above, and will not be repeated here.

[0180] In one embodiment, such as Figure 21 As shown, a virtual reality-based robot interaction device is provided, comprising:

[0181] The acquisition module 100 is used to acquire operation information and control the robot to perform corresponding operations on the virtual object based on the operation information.

[0182] The registration module 200 is used to perform virtual-real registration on at least one display device to determine the pose of each display device relative to the real three-dimensional space.

[0183] The operation animation generation module 300 is used to generate operation animations that match the poses of each display device during the process of controlling the robot to perform corresponding operations. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation in the corresponding pose.

[0184] Display module 400 is used to transmit the operation animations corresponding to each display device to the corresponding display device for display.

[0185] In one embodiment, the registration module 200 is further configured to: perform virtual-real registration on at least one display device based on a plurality of registration markers posted at different locations, so as to determine the registration markers associated with each display device respectively; and calculate the pose of each display device relative to the real three-dimensional space based on the orientation of the registration markers associated with each display device respectively.

[0186] In one embodiment, the display device includes a camera, and supplementary markers are disposed around the periphery of the display device. The registration module 200 is further configured to: acquire images captured by the cameras of each display device; determine unobstructed display devices and obstructed display devices based on the content in the images; perform registration marker recognition processing on the image corresponding to the unobstructed display device to obtain a first frontal marker; and use the registration marker pointed to by the first frontal marker as the registration marker associated with the unobstructed display device; perform supplementary marker recognition processing on the image corresponding to the obstructed display device to obtain a second frontal marker; determine the target display device where the supplementary marker pointed to by the second frontal marker is located; and if the target display device is unobstructed, use the registration marker associated with the target display device as the registration marker associated with the obstructed display device.

[0187] In one embodiment, the registration module 200 is further configured to: perform image thresholding on the image to obtain multiple connected components; and perform perspective transformation on the connected components whose corner count and size meet preset conditions to obtain a first frontal marker.

[0188] In one embodiment, the display device includes an unobstructed display device, and the registration module 200 is further configured to: acquire an image captured by the unobstructed display device, and identify first two-dimensional position information of the corner points of the registration markers in the image captured by the unobstructed display device; determine a first position transformation matrix based on the first two-dimensional position information and the first three-dimensional position information of the corner points of the registration markers in the real three-dimensional space; and determine the pose of the unobstructed display device relative to the real three-dimensional space based on the first position transformation matrix.

[0189] In one embodiment, the registration module 200 is further configured to: establish a three-dimensional coordinate system with the center of the camera of the unobstructed display device as the origin and the plane of the registration marker as the XY plane, obtain the first three-dimensional position information of the corner point of the registration marker in the three-dimensional coordinate system; and determine the first position transformation matrix according to the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the first two-dimensional position information and the first three-dimensional position information.

[0190] In one embodiment, the display device further includes an occluded display device, and the registration module 200 is further configured to: acquire an image captured by the occluded display device, and identify second two-dimensional position information of the corner points of supplementary markers in the image captured by the occluded display device; determine a second position transformation matrix based on the second two-dimensional position information and the second three-dimensional position information of the corner points of supplementary markers in real three-dimensional space; and determine the pose of the occluded display device relative to real three-dimensional space based on the first position transformation matrix and the second position transformation matrix.

[0191] In one embodiment, the registration module 200 is further configured to: establish a three-dimensional coordinate system with the center of the camera of the obscured display device as the origin and the plane of the supplementary marker as the XY plane, obtain the second three-dimensional position information of the corner point of the supplementary marker in the three-dimensional coordinate system; and determine the second position transformation matrix according to the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the second two-dimensional position information and the second three-dimensional position information.

[0192] The modules in the aforementioned virtual reality-based robot interaction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0193] In one embodiment, a computer device is provided, which may be a robot, and its internal structure diagram may be as follows: Figure 22As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a virtual reality-based robot interaction method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0194] Those skilled in the art will understand that Figure 22 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0195] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0196] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0197] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0198] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0199] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0200] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0201] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot interaction method based on virtual reality, characterized in that, The method includes: Obtain operation information and control the robot to perform corresponding operations on the virtual object based on the operation information; At least one display device is registered in both virtual and real modes to determine the pose of each display device relative to the real three-dimensional space. During the process of controlling the robot to perform corresponding operations, operation animations matching each pose are generated according to the poses of each display device. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation in the corresponding pose. The operation animations corresponding to each display device are transmitted to the corresponding display devices for display. The step of performing virtual-real registration on at least one display device to determine the pose of each display device relative to the real three-dimensional space includes: Based on multiple registration markers posted in different locations, at least one display device is registered in both real and virtual modes to determine the registration markers associated with each display device. The pose of each display device relative to the real three-dimensional space is calculated based on the orientation of the registration marker associated with each display device. The display device includes a camera. The step of performing virtual-real registration on at least one display device based on multiple registration markers posted at different locations to determine the registration markers associated with each display device includes: Images captured by the cameras of each display device are acquired, and the front markings of the registration markers in the images captured by each display device are identified. The registration markers pointed to by the front markings are determined to be the registration markers associated with the corresponding display devices.

2. The method according to claim 1, characterized in that, The display device is surrounded by supplementary markers. The process of performing virtual-real registration on at least one display device based on multiple registration markers affixed to different locations to determine the registration markers associated with each display device includes: Images captured by the cameras of each of the display devices are acquired, and the unobstructed display devices and the obstructed display devices are determined based on the content of the images. The image corresponding to the unobstructed display device is processed by registration marker recognition to obtain a first front mark. The registration marker pointed to by the first front mark is used as the registration marker associated with the unobstructed display device. The image corresponding to the obscured display device is processed by supplementary marker recognition to obtain a second front marker. The target display device to which the supplementary marker pointed by the second front marker is located is determined. If the target display device is not obscured, the registration marker associated with the target display device is used as the registration marker associated with the obscured display device.

3. The method according to claim 2, characterized in that, The process of registering and identifying markers on the image corresponding to the unobstructed display device to obtain a first front-facing marker includes: The image is subjected to image thresholding to obtain multiple connected components; A perspective transformation is performed on the connected regions whose corner numbers and sizes meet preset conditions to obtain a first frontal marker.

4. The method according to claim 1, characterized in that, The display devices include unobstructed display devices, and the calculation of the pose of each display device relative to the real three-dimensional space based on the orientation of a registration marker associated with each display device includes: Acquire images captured by an unobstructed display device and identify the first two-dimensional position information of the corner points of the registration markers in the images captured by the unobstructed display device; Based on the first two-dimensional position information and the first three-dimensional position information of the corner points of the registration marker in the real three-dimensional space, a first position transformation matrix is ​​determined; Based on the first position transformation matrix, the pose of the unobstructed display device relative to the real three-dimensional space is determined.

5. The method according to claim 4, characterized in that, The step of determining the first position transformation matrix based on the first two-dimensional position information and the first three-dimensional position information of the corner points of the registration marker in real three-dimensional space includes: A three-dimensional coordinate system is established with the center of the camera of the unobstructed display device as the origin and the plane of the registration marker as the XY plane, and the first three-dimensional position information of the corner point of the registration marker in the three-dimensional coordinate system is obtained. Based on the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the first two-dimensional position information, and the first three-dimensional position information, the first position transformation matrix is ​​determined.

6. The method according to claim 4, characterized in that, The display device also includes an obscured display device, and the step of calculating the pose of each display device relative to the real three-dimensional space based on the orientation of a registration marker associated with each display device further includes: Acquire images captured by an obstructed display device and identify second two-dimensional position information of corner points of supplementary markers in the images captured by the obstructed display device; Based on the second two-dimensional position information and the second three-dimensional position information of the corner points of the supplementary markers in the real three-dimensional space, a second position transformation matrix is ​​determined; Based on the first position transformation matrix and the second position transformation matrix, the pose of the occluded display device relative to the real three-dimensional space is determined.

7. The method according to claim 6, characterized in that, The step of determining the second position transformation matrix based on the second two-dimensional position information and the second three-dimensional position information of the corner points of the supplementary markers in the real three-dimensional space includes: A three-dimensional coordinate system is established with the center of the camera of the obscured display device as the origin and the plane of the supplementary marker as the XY plane, and the second three-dimensional position information of the corner point of the supplementary marker in the three-dimensional coordinate system is obtained. Based on the transformation relationship between the three-dimensional coordinate system and the two-dimensional coordinate system, the second two-dimensional position information, and the second three-dimensional position information, the second position transformation matrix is ​​determined.

8. The method according to claim 1, characterized in that, The step of calculating the pose of each display device relative to the real three-dimensional space based on the orientation of the registration marker associated with each display device includes: The orientation of the registration marker associated with each display device is determined, and the pose of each display device relative to the real three-dimensional space is determined by combining the two-dimensional and three-dimensional coordinates of the same point within the orientation, as well as the transformation relationship between the two-dimensional and three-dimensional coordinate systems.

9. A robot interaction device based on virtual reality, characterized in that, The device includes: The acquisition module is used to acquire operation information and control the robot to perform corresponding operations on the virtual object based on the operation information. The registration module is used to perform virtual-real registration on at least one display device to determine the pose of each display device relative to the real three-dimensional space. The operation animation generation module is used to generate operation animations that match each pose according to the pose of each display device during the process of controlling the robot to perform corresponding operations. The operation animations reflect the state change process of the virtual object after receiving the corresponding operation in the corresponding pose. The display module is used to transmit the operation animations corresponding to each display device to the corresponding display device for display. The registration module is also used to perform virtual-real registration on at least one display device based on multiple registration markers posted in different locations, so as to determine the registration markers associated with each display device respectively. The pose of each display device relative to the real three-dimensional space is calculated based on the orientation of the registration marker associated with each display device. The display device includes a camera, and the registration module is further configured to acquire images captured by the cameras of each display device, identify the front markings of registration markers in the images captured by each display device, and determine that the registration markers pointed to by the front markings are registration markers associated with the corresponding display device.

10. The apparatus according to claim 9, characterized in that, The display device includes a camera, and supplementary markers are provided around the display device. The registration module is also used to acquire images captured by the cameras of each display device, and to determine the unobstructed display devices and the obstructed display devices based on the content in the images. The image corresponding to the unobstructed display device is processed by registration marker recognition to obtain a first front mark. The registration marker pointed to by the first front mark is used as the registration marker associated with the unobstructed display device. The image corresponding to the obscured display device is processed by supplementary marker recognition to obtain a second front marker. The target display device to which the supplementary marker pointed by the second front marker is located is determined. If the target display device is not obscured, the registration marker associated with the target display device is used as the registration marker associated with the obscured display device.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images