Virtual reality interaction method, device, equipment and computer readable storage medium

Abstract

The present application relates to the technical field of virtual reality, and especially relates to a virtual reality interaction method, device, equipment and computer readable storage medium, the method comprises the following steps: collecting a first environment image of an external environment in which a user is located through an image sensor; obtaining a target object specified by the user in the external environment, and extracting a first object image of the target object from the first environment image; generating a target image according to the first object image, and displaying the target image in a virtual display picture of a head-mounted display device. The present application improves the interactivity between the virtual world and the real world of the head-mounted display device.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and in particular to a virtual reality interaction method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the development of technology, head-mounted display devices are gradually being widely used in people's daily lives. Head-mounted display devices can provide users with interactive virtual scenes; for example, VR (Virtual Reality) devices can provide users with immersive virtual scenes. However, when using head-mounted display devices, users cannot observe objects in the real world within the virtual scene, which reduces the interactivity between the virtual and real worlds. Summary of the Invention

[0003] The main objective of this invention is to provide a virtual reality interaction method, apparatus, device, and computer-readable storage medium, which aims to improve the interactivity of head-mounted display devices between the virtual and real worlds.

[0004] To achieve the above objectives, the present invention provides a virtual reality interaction method, which includes the following steps:

[0005] The image sensor captures the first environmental image of the user's surroundings.

[0006] Acquire the target object specified by the user in the external environment, and extract the first object image of the target object from the first environment image;

[0007] A target image is generated based on the first object image, and the target image is displayed in the virtual display screen of the head-mounted display device.

[0008] Optionally, an infrared sensor is provided on the head-mounted display device, and before the step of acquiring a first environmental image of the user's external environment through an image sensor, the method further includes:

[0009] The infrared sensor emits infrared rays arranged in a preset pattern.

[0010] The step of obtaining the user-specified target object in the external environment includes:

[0011] The object containing the preset pattern in the first environmental image of each frame is identified as the positioning object;

[0012] If it is detected that the preset pattern falls on the same positioning object in multiple consecutive frames of the first environment image collected within a first preset time period, then the positioning object on which the preset pattern falls is determined as the target object specified by the user.

[0013] Optionally, after the step of obtaining the user-specified target object in the external environment, the method further includes:

[0014] Within a second preset time period starting from the moment the target object is acquired, detect whether the user has moved;

[0015] The step of generating a target image based on the first object image includes:

[0016] If the user does not move, the first object image is determined as the target image;

[0017] If the user moves, the image sensor acquires multiple frames of second environmental images of the external environment during the user's movement, extracts second object images of the target object from each frame of the second environmental images, and generates the target image based on the first object image and the second object image.

[0018] Optionally, the step of generating the target image based on the first object image and the second object image includes:

[0019] Determine a reference image as the scaling reference from the first object image and the second object image;

[0020] The images to be processed, excluding the reference image, from the first object image and the second object image are scaled according to the size of the reference image to obtain the processed image;

[0021] The coordinates of each pixel in the processed image and the reference image are converted into three-dimensional points in three-dimensional space.

[0022] A stereoscopic image of the target object is constructed according to each of the stereoscopic points, and the stereoscopic image is used as the target image.

[0023] Optionally, after the step of acquiring multiple frames of second environmental images of the user's external environment during movement using the image sensor, the method further includes:

[0024] Detect whether the target object is present in each frame of the second environment image;

[0025] If each frame of the second environment image contains the target object, then the process of extracting the second object image from each frame of the second environment image is performed.

[0026] Optionally, the step of displaying the target image in the virtual display screen of the head-mounted display device includes:

[0027] Store the target image and enter the virtual mode of the head-mounted display device;

[0028] In virtual mode, the target image is imported and displayed in the virtual display screen of the virtual mode.

[0029] Optionally, before the step of storing the target image and entering the virtual mode of the head-mounted display device, the method further includes:

[0030] When the head-mounted display device is in perspective mode, the target image is displayed in the perspective display screen in perspective mode, and it is detected whether the target image in the perspective display screen and the virtual handle in the perspective display screen overlap.

[0031] If the target image in the perspective display screen overlaps with the virtual handle in the perspective display screen, then the step of storing the target image and entering the virtual mode of the head-mounted display device is executed.

[0032] To achieve the above objectives, the present invention also provides a virtual reality interaction device, the virtual reality interaction device comprising:

[0033] The acquisition module is used to acquire a first environmental image of the user's external environment through an image sensor;

[0034] The determination module is used to acquire the target object specified by the user in the external environment and extract the first object image of the target object from the first environment image;

[0035] The display module is used to generate a target image based on the first object image and display the target image in the virtual display screen of the head-mounted display device.

[0036] To achieve the above objectives, the present invention also provides a virtual reality interaction device, the virtual reality interaction device comprising: a memory, a processor, and a virtual reality interaction program stored in the memory and executable on the processor, wherein the virtual reality interaction program, when executed by the processor, implements the steps of the virtual reality interaction method as described above.

[0037] Furthermore, to achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a virtual reality interaction program, which, when executed by a processor, implements the steps of the virtual reality interaction method as described above.

[0038] In this invention, a first environmental image of the user's external environment is acquired by an image sensor, a target object specified by the user in the external environment is obtained, a first object image of the target object is extracted from the first environmental image, a target image is generated based on the first object image, and the target image is displayed in the virtual display screen of the head-mounted display device.

[0039] This invention enables the display of real-world objects within a virtual display screen provided by a head-mounted display device, allowing users to observe real objects of interest in the virtual world. Compared to devices that only provide an immersive virtual world, this invention improves the interactivity between the virtual and real worlds of head-mounted display devices.

[0040] Furthermore, compared to providing users with pre-stored images of real objects through applications in head-mounted display devices, the real object images obtained by this invention are more in line with the actual needs and preferences of users, expanding the range of choices available to users and improving the user experience of using head-mounted display devices. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the hardware operating environment involved in the embodiments of the present invention;

[0042] Figure 2 This is a flowchart illustrating the first embodiment of the virtual reality interaction method of the present invention;

[0043] Figure 3 This is a schematic diagram of the structure of a head-mounted display device according to an embodiment of the present invention;

[0044] Figure 4 This refers to a virtual mode screen overlaid with a target image, as described in an embodiment of the present invention.

[0045] Figure 5 This is a schematic diagram of the functional modules of a preferred embodiment of the virtual reality interaction device of the present invention.

[0046] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0047] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0048] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0049] It should be noted that the virtual reality interaction device in the embodiments of the present invention can be a head-mounted display device, such as a virtual reality device, an augmented reality device, a mixed reality device, etc.; or it can be a device that establishes a communication connection with the head-mounted display device, such as a mobile phone, a computer, etc., without specific limitations.

[0050] like Figure 1 As shown, the virtual reality interaction device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0051] Those skilled in the art will understand that Figure 1 The device structure shown does not constitute a limitation on virtual reality interactive devices and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0052] like Figure 1 As shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a virtual reality interaction program. The operating system is a program that manages and controls the hardware and software resources of the device, supporting the operation of the virtual reality interaction program and other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the virtual reality interaction program stored in the memory 1005 and perform the following operations:

[0053] The image sensor captures the first environmental image of the user's surroundings.

[0054] Acquire the target object specified by the user in the external environment, and extract the first object image of the target object from the first environment image;

[0055] A target image is generated based on the first object image, and the target image is displayed in the virtual display screen of the head-mounted display device.

[0056] Furthermore, an infrared sensor is provided on the head-mounted display device. Before the operation of acquiring the first environmental image of the user's external environment through the image sensor, the processor 1001 can also call the virtual reality interaction program stored in the memory 1005 and perform the following operations:

[0057] The infrared sensor emits infrared rays arranged in a preset pattern.

[0058] The step of obtaining the user-specified target object in the external environment includes:

[0059] The object containing the preset pattern in the first environmental image of each frame is identified as the positioning object;

[0060] If it is detected that the preset pattern falls on the same positioning object in multiple consecutive frames of the first environment image collected within a first preset time period, then the positioning object on which the preset pattern falls is determined as the target object specified by the user.

[0061] Furthermore, after acquiring the user-specified target object in the external environment, the processor 1001 can also call the virtual reality interaction program stored in the memory 1005 and perform the following operations:

[0062] Within a second preset time period starting from the moment the target object is acquired, detect whether the user has moved;

[0063] The step of generating a target image based on the first object image includes:

[0064] If the user does not move, the first object image is determined as the target image;

[0065] If the user moves, the image sensor acquires multiple frames of second environmental images of the external environment during the user's movement, extracts second object images of the target object from each frame of the second environmental images, and generates the target image based on the first object image and the second object image.

[0066] Further, the operation of generating the target image based on the first object image and the second object image includes:

[0067] Determine a reference image as the scaling reference from the first object image and the second object image;

[0068] The images to be processed, excluding the reference image, from the first object image and the second object image are scaled according to the size of the reference image to obtain the processed image;

[0069] The coordinates of each pixel in the processed image and the reference image are converted into three-dimensional points in three-dimensional space.

[0070] A stereoscopic image of the target object is constructed according to each of the stereoscopic points, and the stereoscopic image is used as the target image.

[0071] Furthermore, after acquiring multiple frames of second environmental images of the user's external environment during movement via the image sensor, the processor 1001 can also call the virtual reality interaction program stored in the memory 1005 and perform the following operations:

[0072] Detect whether the target object is present in each frame of the second environment image;

[0073] If each frame of the second environment image contains the target object, then the process of extracting the second object image from each frame of the second environment image is performed.

[0074] Furthermore, the operation of displaying the target image in the virtual display screen of the head-mounted display device includes:

[0075] Store the target image and enter the virtual mode of the head-mounted display device;

[0076] In virtual mode, the target image is imported and displayed in the virtual display screen of the virtual mode.

[0077] Furthermore, before storing the target image and entering the virtual mode of the head-mounted display device, the processor 1001 can also call the virtual reality interaction program stored in the memory 1005 and perform the following operations:

[0078] When the head-mounted display device is in perspective mode, the target image is displayed in the perspective display screen in perspective mode, and it is detected whether the target image in the perspective display screen and the virtual handle in the perspective display screen overlap.

[0079] If the target image in the perspective display screen overlaps with the virtual handle in the perspective display screen, then the step of storing the target image and entering the virtual mode of the head-mounted display device is executed.

[0080] Based on the above structure, various embodiments of the virtual reality interaction method are proposed.

[0081] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the virtual reality interaction method of the present invention.

[0082] This invention provides embodiments of a virtual reality interaction method. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order. In this embodiment, the executing entity of the virtual reality interaction method can be a virtual reality interaction device, which can be a head-mounted display device, such as a virtual reality device, augmented reality device, mixed reality device, etc.; it can also be a device that establishes a communication connection with the head-mounted display device, such as a mobile phone, computer, etc. No limitation is made in this embodiment. For ease of description, the execution entity is omitted from the description of each embodiment. In this embodiment, the virtual reality interaction method includes:

[0083] Step S10: Acquire a first environmental image of the user's external environment using an image sensor;

[0084] In this embodiment, an image sensor is used to acquire an image of the user's external environment (hereinafter referred to as the first environmental image for distinction). Specifically, the image sensor may be installed on the head-mounted display device or on a device that establishes a communication connection with the head-mounted display device (hereinafter referred to as the connection device for distinction), and there is no limitation thereto.

[0085] In one specific embodiment, the image sensor can be disposed on the head-mounted display device, specifically in front of the head-mounted display device, aligned with the user's facial orientation, so that the acquired first environmental image matches the user's viewing angle. This ensures that the first environmental image conforms to the user's actual visual perception, thereby enhancing the user's sense of interaction and immersion when using the head-mounted display device for real-world and virtual-world interaction.

[0086] Step S20: Obtain the target object specified by the user in the external environment, and extract the first object image of the target object from the first environment image;

[0087] Users can specify real objects from the external environment to be displayed in the virtual screen of a head-mounted display device. These objects will be referred to as target objects for distinction. In this embodiment, the target objects specified by the user in the external environment are acquired.

[0088] Specifically, in one embodiment, a microphone can be provided on the virtual reality interaction device to acquire target objects by receiving voice commands from the user; in another embodiment, a display screen can be provided on the virtual reality interaction device to acquire target objects by receiving control operations or input commands from the user; in yet another embodiment, an infrared sensor can be provided on the virtual reality interaction device. In this embodiment, the target object can be determined based on the area of ​​the external environment illuminated by the infrared light emitted by the infrared sensor. The specific settings can be configured according to actual needs and are not limited here.

[0089] In this embodiment, after determining the target object, the image corresponding to the target object is extracted from the first environment image, hereinafter referred to as the first object image for distinction. In one embodiment, the target image can be obtained by target segmentation and extraction of the target object using OpenCV (a cross-platform computer vision and machine learning software library); in another embodiment, the target image can also be obtained by target segmentation and extraction based on image segmentation technology, which will not be elaborated here.

[0090] Step S30: Generate a target image based on the first object image, and display the target image in the virtual display screen of the head-mounted display device.

[0091] In this embodiment, after obtaining the first object image, an image of the target object (hereinafter referred to as the first object image for convenience) can be generated based on the first object image, and the target image can be displayed in the virtual mode display screen of the head-mounted display device (hereinafter referred to as the virtual display screen for convenience).

[0092] In a specific implementation, the system can determine whether to generate a planar image of the target object based on the first object image or a three-dimensional image of the target object according to the user's command triggered by voice or action, so that the obtained target image meets the user's actual needs and improves the user's experience of using the head-mounted display device.

[0093] Furthermore, in one embodiment, the target image may be displayed on the virtual display screen of the head-mounted display device after receiving the import command; in another embodiment, the target image may be displayed on the virtual display screen of the head-mounted display device immediately after it is generated, and the specific implementation is not limited here. For example, in one embodiment, after obtaining the target image, the target image can be stored in the virtual reality interaction device; when the user needs to display the target image, the user can issue an import command to import the target image, which can be triggered by control operation or voice command; the virtual reality interaction device responds to the import command by displaying the target image on the head-mounted display device.

[0094] Furthermore, in a feasible implementation, the virtual reality interaction device may store multiple target images of the target object (hereinafter referred to as associated images for distinction). In this implementation, before displaying the target image in the virtual display screen of the head-mounted display device in step S30, the method further includes:

[0095] The target image and related images are provided to the user for selection; the image selected by the user from the target image and related images is used as the target image and displayed in the virtual display screen of the head-mounted display device.

[0096] Furthermore, in a feasible implementation, in step S30: after generating the target image based on the first object image, the virtual reality interaction device can detect whether the target image is complete. Specifically, the detection process can be: acquiring a network image of the target object from the internet, comparing the network image with the target image to detect whether the target image is complete; alternatively, a deep learning model can be trained to detect the target image, details of which will not be elaborated here. If the target image is incomplete, images similar to the target image can be searched from locally stored images or images stored on a server, and the target image can be filled in using the image with the highest similarity. The filled-in target image is then displayed in the virtual display screen of the head-mounted display device. This implementation allows the target image displayed by the head-mounted display device to be complete and clear, improving the immersive experience for users interacting with the head-mounted display device.

[0097] Furthermore, in one feasible implementation, after the target image is displayed in the virtual display screen, it can be scaled and moved in the virtual display screen according to the user's operation, so as to improve the user's experience of using the head-mounted display device.

[0098] In this embodiment, a first environmental image of the user's external environment is acquired by an image sensor, a target object specified by the user in the external environment is obtained, a first object image of the target object is extracted from the first environmental image, a target image is generated based on the first object image, and the target image is displayed in the virtual display screen of the head-mounted display device.

[0099] This embodiment enables the display of real-world objects within a virtual display screen provided by a head-mounted display device, allowing users to observe real objects of interest in the virtual world. Compared to simply providing an immersive virtual world, this embodiment improves the interactivity of the head-mounted display device between the virtual and real worlds.

[0100] Furthermore, compared to providing users with pre-stored images of real objects through applications within the head-mounted display device, the real object images obtained in this embodiment better meet the user's actual needs and preferences, expanding the user's range of choices and improving the user's experience of using the head-mounted display device.

[0101] Furthermore, based on the first embodiment described above, a second embodiment of the virtual reality interaction algorithm of the present invention is proposed. In this embodiment, an infrared sensor is provided on the head-mounted display device. In this embodiment, before step S10, the following steps are also included:

[0102] Step S40: The infrared sensor emits infrared rays arranged in a preset pattern.

[0103] In this embodiment, an infrared sensor is installed on the head-mounted display device to locate target objects using infrared rays emitted by the sensor. In a specific implementation, the infrared rays emitted by the sensor can form a special, easily recognizable shape, such as a five-pointed star, to avoid false detections and improve the accuracy of target object identification.

[0104] Specifically, in this embodiment, infrared rays are emitted by an infrared sensor according to a preset special pattern (hereinafter referred to as the preset pattern for distinction).

[0105] In a specific implementation, refer to Figure 3 , Figure 3 This is a schematic diagram of a head-mounted display device according to an embodiment of the present invention. The image sensor is an infrared camera disposed on the head-mounted display device, such as... Figure 3 As shown in the figure, 3001 is the fixing strap of the head-mounted display device, 3002 is the display of the head-mounted display device, 3003 is the infrared sensor, and 3004 is the infrared camera. In this embodiment, while emitting infrared rays through the infrared sensor, the infrared camera captures the current image of the current external environment. The object to be located by infrared irradiation is determined based on the position of the infrared rays in the current image (hereinafter referred to as the positioning object for distinction).

[0106] In this embodiment, step S20, obtaining the user-specified target object in the external environment, includes:

[0107] Step S201: Determine the object containing the preset pattern in the first environmental image of each frame as the positioning object;

[0108] In this embodiment, the object containing the preset pattern in each frame of the first environmental image is determined as the positioning object.

[0109] Specifically, in one embodiment, target detection or image segmentation can be performed on the first environmental image to determine the images of each object in the first environmental image, and the object corresponding to the object image containing the preset pattern in the first environmental image is determined as the positioning object. In a specific embodiment, infrared light may illuminate one object or multiple objects. Therefore, all objects illuminated by infrared light can be determined as positioning objects, that is, the object corresponding to the object image containing the entire preset pattern in the first environmental image is determined as the positioning object.

[0110] In another embodiment, the image region containing the preset pattern in the first environmental image (hereinafter referred to as the positioning region image for distinction) can be determined, and the object in the positioning region image can be determined as the positioning object. Further, there may be multiple objects in the image region containing the preset pattern (hereinafter referred to as region objects for distinction). In this case, the image area of ​​the region object in the positioning region image can be calculated, and the proportion of the region object's image area in the first environmental image can be calculated (hereinafter referred to as the positioning ratio for distinction). The region object with the largest positioning ratio among all region objects can be determined as the positioning object.

[0111] Step S202: When it is detected that in multiple consecutive frames of the first environment image collected within a first preset time period, the preset pattern falls on the same positioning object, the positioning object on which the preset pattern falls is determined as the target object specified by the user.

[0112] In this embodiment, after determining the location object, if it is detected that in multiple consecutive frames of the first environmental image collected within a first preset time period, the preset pattern falls on the same location object, then the location object on which the preset pattern falls is determined as the user-specified target object. The first preset time period can be set according to actual needs; for example, in one embodiment, the first preset time period can be set to 3 seconds.

[0113] It should be noted that, compared to directly identifying the located object as the target object, this embodiment identifies the located object as the user-specified target object by detecting that a preset pattern continuously falls on the located object in multiple consecutive frames of the first environmental image collected within a first preset time period. This embodiment can make the identified target object more accurate, thereby improving the user's experience of using the head-mounted display device for virtual reality interaction.

[0114] Furthermore, compared to directly identifying the located object as the target object, this embodiment avoids the situation where the user accidentally identifies the target object and needs to reposition, reducing the user's operation process and improving the user's experience and immersion when using the head-mounted display device.

[0115] Furthermore, in one embodiment, the target object can be determined by the point on the object illuminated by the infrared light emitted by the infrared sensor (hereinafter referred to as the positioning point for convenience). Specifically, if the user wants to extract the target object, the positioning point of the infrared sensor can be used as a drawing tool, and the range of the target object to be captured can be determined by the movement of the head. Therefore, in this embodiment, the target object can be determined by detecting the movement trajectory of the positioning point of the infrared sensor.

[0116] Further, in a feasible implementation, step S30: displaying the target image in the virtual display screen of the head-mounted display device includes:

[0117] Step S301: Store the target image and enter the virtual mode of the head-mounted display device;

[0118] In this embodiment, the head-mounted display device may not have virtual mode enabled. For example, when the head-mounted display device is a virtual reality device, in order to enable the user to see the external environment and more accurately locate the target object, the virtual reality device may be in a perspective mode that allows the user to see the external environment, or the virtual reality device may not have its display enabled.

[0119] Therefore, in this embodiment, it is necessary to first store the target image and enter the virtual mode of the head-mounted display device. Specifically, the target image can be stored in the head-mounted display device or in a device that establishes a communication connection with the head-mounted display device (hereinafter referred to as the connection device for convenience). No specific limitation is made here.

[0120] Step S302: In virtual mode, import the target image and display the target image in the virtual display screen of the virtual mode.

[0121] In this embodiment, the target image is imported in virtual mode and displayed on the virtual mode display screen, i.e., in the virtual reality screen. Specifically, in one embodiment, displaying the target image in the virtual screen can be done by referring to... Figure 4 , Figure 4 This refers to a virtual display screen displaying a target image, as described in an embodiment of the present invention.

[0122] Specifically, in one embodiment, when the virtual reality interaction device is a head-mounted display device, the head-mounted display device can activate the simulation mode after obtaining the target image and display the target image in the virtual display screen. Compared with the virtual reality interaction device being a connection device, this embodiment can reduce the interaction process between the head-mounted display device and the connection device, thereby reducing the time required to display the target image in the screen of the head-mounted display device. This can improve the timeliness of virtual and display interaction for users using the head-mounted display device, thereby improving the user experience of using the head-mounted display device.

[0123] In another embodiment, when the virtual reality interaction device is a device that communicates with a head-mounted display device, the connecting device may be a device that sends an instruction carrying a target image to the head-mounted display device, and the head-mounted display device displays the target image in a virtual display screen according to the instruction.

[0124] Furthermore, in a feasible implementation, before step S301, the method further includes:

[0125] Step S303: When the head-mounted display device is in perspective mode, the target image is displayed in the perspective display screen in perspective mode, and it is detected whether the target image in the perspective display screen and the virtual handle in the perspective display screen overlap.

[0126] In this embodiment, in perspective mode, the head-mounted display device presents the real external environment onto its display screen through a camera. Typically, a virtual controller is present in the perspective mode display screen (hereinafter referred to as the perspective display screen for distinction). Therefore, in this embodiment, the user can trigger input commands through the operation of the virtual controller, reducing user operations and allowing for a more immersive experience using the head-mounted display device.

[0127] Specifically, in this embodiment, when the head-mounted display device is in perspective mode, the target image is displayed in the perspective display screen in perspective mode, and it is detected whether the target image in the perspective display screen and the virtual handle in the perspective display screen overlap.

[0128] Specifically, in one embodiment, it can be detected whether there are overlapping parts between the coordinates of the target image in the display screen of the head-mounted display device and the coordinates of the virtual handle in the display screen, so as to detect whether the target image in the perspective display screen and the virtual handle in the perspective display screen overlap. In this embodiment, if there are overlapping parts in the coordinates, it is determined that the target image in the perspective display screen and the virtual handle in the perspective display screen overlap. In another embodiment, it can also be detected whether the target image and the virtual handle overlap by image detection technology, which can be set according to actual needs.

[0129] Step S304: If the target image in the perspective display screen overlaps with the virtual handle in the perspective display screen, then the step of storing the target image and entering the virtual mode of the head-mounted display device is executed.

[0130] In this embodiment, if the target image in the perspective display screen overlaps with the virtual handle in the perspective display screen, it is determined that the user needs to display the target image in the virtual mode screen. At this time, the target image is stored and the virtual mode is entered. The target image is imported in the virtual mode and displayed in the virtual mode screen.

[0131] It should be noted that in this embodiment, the user can control the display of the target image by controlling the virtual controller. Compared with the user triggering import commands by pressing buttons or by voice, this embodiment allows the user to immerse themselves in the use of the head-mounted display device.

[0132] In this embodiment, infrared rays are emitted by an infrared sensor in a preset pattern; the object containing the preset pattern in each frame of the first environmental image is identified as the positioning object; when it is detected that the preset pattern continuously falls on the positioning object in multiple consecutive frames of the first environmental images collected within a first preset time period, the positioning object is identified as the target object specified by the user.

[0133] Compared to determining the target object based on the user's voice commands or spatial operations, this embodiment can determine the target object based on infrared illumination, making the determined target object more accurate, thereby improving the user's experience of using the head-mounted display device for virtual reality interaction.

[0134] Furthermore, based on the first and / or second embodiments described above, a third embodiment of the virtual reality interaction algorithm of the present invention is proposed. In this embodiment, after obtaining the target object specified by the user in the external environment in step S20, the algorithm further includes:

[0135] Step S50: Within a second preset time period starting from the moment the target object is acquired, detect whether the user has moved;

[0136] In this embodiment, if the user requests a stereoscopic image of a target object, the virtual reality interaction device will prompt the user to move around the target object to obtain images of the target object from various angles. Therefore, in this embodiment, the generation of a planar image or a stereoscopic image of the target object can be determined based on the user's movements.

[0137] Specifically, within a second preset time period starting from the moment the target object is acquired, it is detected whether the user has moved. In one embodiment, this can be achieved by using a six-axis sensor on the virtual reality device to detect whether the user has moved within the second preset time period starting from the moment the target object is acquired. In another embodiment, when the virtual reality interaction device is a connected device and cannot be worn by the user, it can be achieved by using a distance sensor on the virtual reality interaction device to detect whether the distance between the user and the virtual reality interaction device has changed, thus detecting whether the user has moved within the second preset time period starting from the moment the target object is acquired. In yet another embodiment, when the virtual reality interaction device is a head-mounted display device and an infrared sensor is provided on the head-mounted display device, it can detect whether the user has moved based on whether the infrared positioning point has moved. No specific limitations are imposed here.

[0138] Step S60: If the user does not move, then the first object image is determined as the target image;

[0139] In this embodiment, if the user does not move, a planar image of the target object is generated, and the first object image is determined as the target image.

[0140] In a specific implementation, when there are multiple frames of the first object image, the first frame of the first object image can be determined as the target image, or the frame with the highest clarity among the first object images can be determined as the target image. The specific settings can be made according to actual needs, and no restrictions are imposed here.

[0141] In a specific implementation, noise reduction and other processing can be applied to the first object image to make the obtained target image clearer and improve the user experience of using the head-mounted display device.

[0142] Step S70: If the user moves, the image sensor acquires multiple frames of second environmental images of the external environment during the user's movement, extracts second object images of the target object from each frame of the second environmental images, and generates the target image based on the first object image and the second object image.

[0143] In this embodiment, if the user moves, it is determined that the user needs to generate a stereoscopic image of the target object. At this time, multiple frames of images of the external environment during the user's movement are acquired by an image sensor (hereinafter referred to as the second environment image for distinction). The image of the target object (hereinafter referred to as the second object image for distinction) is extracted from each frame of the second environment image, and the target image is generated based on the first object image and the second object image.

[0144] Furthermore, in a feasible implementation, it is also possible to determine whether to generate a planar image or a stereo image of the target object by detecting the user's head movement. For example, if the user's head moves on the z-axis of the six-axis sensor, it is determined to generate a stereo image of the target object; if the user's head moves on the x-axis of the six-axis sensor, it is determined to generate a planar image of the target object.

[0145] Further, in a feasible implementation, step S70: generating the target image based on the first object image and the second object image includes:

[0146] Step S701: Determine a reference image as a scaling reference from the first object image and the second object image;

[0147] In this embodiment, the user requests the generation of a stereoscopic image of the target object. The virtual reality interactive device will prompt the user to move around the target object to obtain images of the target object from various angles. Since the user cannot maintain the same distance from the target object throughout the movement, the sizes of the first object image and the second object images in each frame may differ. In this embodiment, it is necessary to adjust each frame image to the same size before generating the target image based on the adjusted image.

[0148] Specifically, in this embodiment, a reference image is determined from the first object image and the second object image as the scaling reference. In a specific embodiment, the first frame image in the first object image and the second object image can be determined as the reference image, or the frame image with the highest clarity in the target image can be determined as the reference image. The specific setting can be set according to actual needs, and there is no limitation here.

[0149] Step S702: The images to be processed in the first object image and the second object image, excluding the reference image, are scaled according to the size of the reference image to obtain the processed image;

[0150] The images to be processed, excluding the reference image, in the first and second object images are scaled according to the size of the reference image to obtain the processed image.

[0151] In one embodiment, the scaling factor can be determined based on the size difference between the image to be processed and the reference image. In another embodiment, when the virtual reality device is a head-mounted display device, a distance sensor and a camera are provided on the head-mounted display device. As the user moves around the target object, the distance sensor can obtain the distance between the user and the target object in real time. In this embodiment, the scaling factor can be determined based on the distances corresponding to the first object image and the second object image, respectively. The specific scaling factor can be set according to actual needs and is not limited here.

[0152] Step S703: Convert the coordinates of each pixel in the processed image and the reference image into solid points in three-dimensional space;

[0153] The coordinates of each pixel in the processed image and the reference image are converted into points in three-dimensional space (hereinafter referred to as three-dimensional points for distinction), and a three-dimensional image of the target object is constructed according to each three-dimensional point.

[0154] Step S704: Construct a stereoscopic image of the target object according to each of the stereoscopic points, and use the stereoscopic image as the target image.

[0155] In this embodiment, a stereoscopic image of the target object is constructed according to each stereoscopic point, and the stereoscopic image is used as the target image.

[0156] Specifically, in this embodiment, the specific process of generating a stereoscopic image of the target object can be as follows: converting the coordinates of each pixel in the processed image and the reference image into points in stereoscopic space, and forming a stereoscopic model from the points in stereoscopic space; rendering the stereoscopic model into a stereoscopic image of the target object.

[0157] Furthermore, in another embodiment, the processed image and the reference image can be converted into a 3D mesh model using an image processing algorithm; then, the coordinates of each grid point in the 3D mesh model can be converted into points in 3D space using computer vision technology, and the points in 3D space can be used to form a 3D model; finally, the 3D model can be rendered into a 3D image using computer vision technology.

[0158] Furthermore, in a feasible implementation, after acquiring multiple frames of second environmental images of the user's external environment during movement using the image sensor in step S70, the method further includes:

[0159] Step S80: Detect whether each frame of the second environment image contains the target object;

[0160] In this embodiment, after acquiring multiple frames of second environmental images of the user's external environment during movement using an image sensor, it is detected whether each frame of the second environmental image contains the target object, in order to determine whether the user wants to obtain a stereoscopic image of the target object or does not need to generate a target image.

[0161] Furthermore, in one embodiment, when an infrared sensor is provided on the head-mounted display device, it can detect whether a preset pattern continuously exists on the target object in each frame of the second environmental image, in order to determine whether the user wants to obtain a stereoscopic image of the target object or does not need to generate a target image.

[0162] Step S90: If each frame of the second environment image contains the target object, then the second object image of the target object is extracted from each frame of the second environment image.

[0163] If each frame of the second environment image contains the target object, then the user needs to generate a stereoscopic image of the target object. At this time, the second object image of the target object is extracted from each frame of the second environment image, and the target image is generated based on the first object image and the second object image.

[0164] It should be noted that this implementation method makes the generated target image more in line with the user's actual needs and preferences, thereby improving the user's experience of using the head-mounted display device.

[0165] In this embodiment, the user is detected to have moved within a second preset time period from the moment the target object is acquired. If the user does not move, the first object image is determined as the target image. If the user moves, multiple frames of second environment images of the external environment during the user's movement are acquired by the image sensor. The second object image of the target object is extracted from each frame of the second environment image, and the target image is generated based on the first object image and the second object image.

[0166] This embodiment generates target images based on the user's actual needs. Compared to directly generating planar or stereoscopic images, the target images obtained in this embodiment better meet the user's actual needs and preferences, thus improving the user's experience when using the head-mounted display device.

[0167] Furthermore, in a feasible embodiment, the virtual reality interaction method is applied to a head-mounted display device. The infrared sensor installed on the head-mounted display device is an infrared distance sensor. This infrared distance sensor can be used to locate a target object and test the actual distance between the target object and the head-mounted display device. The devices emitting infrared optical fibers in the infrared distance sensor are arranged according to a preset pattern, so that the point on the object illuminated by the emitted infrared light, i.e., the positioning point, is within the preset pattern, thus facilitating camera acquisition. In this embodiment, the camera on the head-mounted display device is an infrared camera, used to see through the surrounding environment and capture the positioning point of the infrared distance sensor.

[0168] In this embodiment, the specific process of virtual reality interaction is as follows:

[0169] When a user needs to obtain an image of a real-world object, the head-mounted display device will switch to perspective mode, and the infrared camera and infrared distance sensor will start working. The infrared camera will capture a first environmental image of the user's surroundings, and based on the first environmental image, the infrared distance sensor will identify the object A that is being illuminated by infrared rays arranged in a preset pattern, and the object A will be identified as the positioning object.

[0170] From the moment the object is identified, when the positioning point of the infrared light emitted by the infrared distance sensor stays on object A for a period of time that reaches a first preset duration, that is, when the preset pattern continuously falls on the positioning object in multiple consecutive frames of the first environmental image collected within the first preset duration, object A is identified as the target object.

[0171] From the moment the target object is determined, if the viewing angle of the head-mounted display device remains unchanged within a second preset time period, that is, if no user movement is detected within the second preset time period from the moment the target object is acquired, a planar image of the target object, i.e., a planar image of object A, is generated. Furthermore, if the object A that the user wants to capture is planar, the focal point of the infrared distance sensor can be used as a drawing tool, and the range of the target object to be captured can be drawn by moving the user's head. In this case, the planar image of the target object can be determined based on the movement trajectory of the positioning point.

[0172] From the moment the target object is determined, if the angle of the head-mounted display device changes within a second preset time period, that is, if user movement is detected within the second preset time period from the moment the target object is acquired, meaning the positioning point is still on object A, then it is determined that the user wants to obtain a stereoscopic image of the target object. At this time, the generation of the stereoscopic image of the target object begins when the user stops moving. Specifically, the process of generating the stereoscopic image of the target object can be as follows: acquiring multiple frames of second environmental images of the external environment during the user's movement using an image sensor, extracting the second object image of the target object from each frame of the second environmental image; determining a reference image and an image to be processed from the first object image and the second object image as the scaling reference; determining the scaling ratio of the image to be processed relative to the reference image based on the distance value from the head-mounted display device to object A uploaded by the infrared distance sensor; converting the coordinates of each pixel of the scaled image to be processed (i.e., the processed image) and the reference image into stereoscopic points in stereoscopic space, and constructing the stereoscopic image of the target object, i.e., the stereoscopic image of object A, according to each stereoscopic point.

[0173] In this embodiment, the obtained target image is displayed in perspective mode. When the user wants to bring the target image from real space to virtual space, the user needs to touch the target image with the right handle. At this time, the head-mounted display device stores the target image and enters virtual mode from perspective mode. In virtual mode, the target image is imported and displayed on the virtual mode screen. In virtual mode, the user can adjust the size and angle of the target image and place it at any position on the virtual mode screen.

[0174] For example, in one embodiment, the head-mounted display device is a virtual reality device, equipped with an infrared distance sensor. If a user wants to place a painting they like in the real world into the virtual space of the virtual reality environment, the virtual reality device first switches to perspective mode. The user moves the infrared distance sensor's positioning point to the painting and waits for 3 seconds (a first preset duration). At this point, it is determined that the user wants an image of the painting, i.e., the painting is the target object. The user then waits still for another 3 seconds (a second preset duration), determining that the user wants a planar view of the painting (i.e., a planar image of the target object is generated). Image processing is then initiated to generate a planar view of the painting, which is then suspended in the perspective mode screen. Specifically, it can be suspended at the user's right virtual controller for easy access. The user touches the planar view of the painting with the right virtual controller and switches the scene to the virtual space; the planar view of the painting is then brought into the virtual space. The system can also accept zooming operations on the planar view of the painting, adjusting it to the user's desired size.

[0175] Furthermore, embodiments of the present invention also propose a virtual reality interaction device, referring to... Figure 5 The virtual reality interaction device includes:

[0176] The acquisition module 10 is used to acquire a first environmental image of the user's external environment through an image sensor;

[0177] The determination module 20 is used to acquire the target object specified by the user in the external environment and extract the first object image of the target object from the first environment image;

[0178] Display module 30 is used to generate a target image based on the first object image and display the target image in the virtual display screen of the head-mounted display device.

[0179] Furthermore, an infrared sensor is provided on the head-mounted display device, and the virtual reality interaction device further includes an infrared module, which is used for:

[0180] The infrared sensor emits infrared rays arranged in a preset pattern.

[0181] The determining module 20 is further configured to:

[0182] The object containing the preset pattern in the first environmental image of each frame is identified as the positioning object;

[0183] If it is detected that the preset pattern falls on the same positioning object in multiple consecutive frames of the first environment image collected within a first preset time period, then the positioning object on which the preset pattern falls is determined as the target object specified by the user.

[0184] Furthermore, the virtual reality interaction device also includes a first detection module, which is used for:

[0185] Within a second preset time period starting from the moment the target object is acquired, detect whether the user has moved;

[0186] The display module 30 is also used for:

[0187] If the user does not move, the first object image is determined as the target image;

[0188] If the user moves, the image sensor acquires multiple frames of second environmental images of the external environment during the user's movement, extracts second object images of the target object from each frame of the second environmental images, and generates the target image based on the first object image and the second object image.

[0189] Furthermore, the display module 30 is also used for:

[0190] Determine a reference image as the scaling reference from the first object image and the second object image;

[0191] The images to be processed, excluding the reference image, from the first object image and the second object image are scaled according to the size of the reference image to obtain the processed image;

[0192] The coordinates of each pixel in the processed image and the reference image are converted into three-dimensional points in three-dimensional space.

[0193] A stereoscopic image of the target object is constructed according to each of the stereoscopic points, and the stereoscopic image is used as the target image.

[0194] Furthermore, the virtual reality interaction device also includes a second detection module, which is used for:

[0195] Detect whether the target object is present in each frame of the second environment image;

[0196] If each frame of the second environment image contains the target object, then the process of extracting the second object image from each frame of the second environment image is performed.

[0197] Furthermore, the display module 30 is also used for:

[0198] Store the target image and enter the virtual mode of the head-mounted display device;

[0199] In virtual mode, the target image is imported and displayed in the virtual display screen of the virtual mode.

[0200] Furthermore, the display module 30 is also used for:

[0201] When the head-mounted display device is in perspective mode, the target image is displayed in the perspective display screen in perspective mode, and it is detected whether the target image in the perspective display screen and the virtual handle in the perspective display screen overlap.

[0202] If the target image in the perspective display screen overlaps with the virtual handle in the perspective display screen, then the step of storing the target image and entering the virtual mode of the head-mounted display device is executed.

[0203] All embodiments of the virtual reality interaction device of the present invention can refer to the various embodiments of the virtual reality interaction method of the present invention, and will not be repeated here.

[0204] Furthermore, embodiments of the present invention also propose a computer-readable storage medium storing a virtual reality interaction program, wherein the virtual reality interaction program, when executed by a processor, implements the steps of the virtual reality interaction method described below.

[0205] The various embodiments of the virtual reality interaction device and computer-readable storage medium of the present invention can be referred to the various embodiments of the virtual reality interaction method of the present invention, and will not be repeated here.

[0206] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0207] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0208] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0209] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A virtual reality interaction method, characterized in that, The virtual reality interaction method includes the following steps: The image sensor captures the first environmental image of the user's surroundings. Acquire the target object specified by the user in the external environment, and extract the first object image of the target object from the first environment image; A target image is generated based on the first object image, and the target image is displayed in the virtual display screen of the head-mounted display device; The head-mounted display device is equipped with an infrared sensor. Before the step of acquiring a first environmental image of the user's external environment through an image sensor, the following method is also included: The infrared sensor emits infrared rays arranged in a preset pattern. The step of obtaining the user-specified target object in the external environment includes: The object containing the preset pattern in the first environmental image of each frame is identified as the positioning object; If it is detected that the preset pattern falls on the same positioning object in multiple consecutive frames of the first environment image collected within a first preset time period, then the positioning object on which the preset pattern falls is determined as the target object specified by the user.

2. The virtual reality interaction method as described in claim 1, characterized in that, After the step of obtaining the user-specified target object in the external environment, the method further includes: Within a second preset time period starting from the moment the target object is acquired, detect whether the user has moved; The step of generating a target image based on the first object image includes: If the user does not move, the first object image is determined as the target image; If the user moves, the image sensor acquires multiple frames of second environmental images of the external environment during the user's movement, extracts second object images of the target object from each frame of the second environmental images, and generates the target image based on the first object image and the second object image.

3. The virtual reality interaction method as described in claim 2, characterized in that, The step of generating the target image based on the first object image and the second object image includes: Determine a reference image as the scaling reference from the first object image and the second object image; The images to be processed, excluding the reference image, from the first object image and the second object image are scaled according to the size of the reference image to obtain the processed image; The coordinates of each pixel in the processed image and the reference image are converted into three-dimensional points in three-dimensional space. A stereoscopic image of the target object is constructed according to each of the stereoscopic points, and the stereoscopic image is used as the target image.

4. The virtual reality interaction method as described in claim 2, characterized in that, After the step of acquiring multiple frames of second environmental images of the user's external environment during movement using the image sensor, the method further includes: Detect whether the target object is present in each frame of the second environment image; If each frame of the second environment image contains the target object, then the process of extracting the second object image from each frame of the second environment image is performed.

5. The virtual reality interaction method as described in any one of claims 1 to 4, characterized in that, The step of displaying the target image in the virtual display screen of the head-mounted display device includes: Store the target image and enter the virtual mode of the head-mounted display device; In virtual mode, the target image is imported and displayed in the virtual display screen of the virtual mode.

6. The virtual reality interaction method as described in claim 5, characterized in that, Before the step of storing the target image and entering the virtual mode of the head-mounted display device, the method further includes: When the head-mounted display device is in perspective mode, the target image is displayed in the perspective display screen in perspective mode, and it is detected whether the target image in the perspective display screen and the virtual handle in the perspective display screen overlap. If the target image in the perspective display screen overlaps with the virtual handle in the perspective display screen, then the step of storing the target image and entering the virtual mode of the head-mounted display device is executed.

7. A virtual reality interactive device, characterized in that, The virtual reality interaction device includes: The acquisition module is used to acquire a first environmental image of the user's external environment through an image sensor; The determination module is used to acquire the target object specified by the user in the external environment and extract the first object image of the target object from the first environment image; The display module is used to generate a target image based on the first object image and display the target image in the virtual display screen of the head-mounted display device; The head-mounted display device is equipped with an infrared sensor, and the virtual reality interaction device further includes an infrared module for: emitting infrared rays arranged according to a preset pattern through the infrared sensor; the step of acquiring the user-specified target object in the external environment includes: determining the object containing the preset pattern in each frame of the first environmental image as the positioning object; when it is detected that in multiple consecutive frames of the first environmental image collected within a first preset time period, the preset pattern falls on the same positioning object, then the positioning object on which the preset pattern falls is determined as the user-specified target object.

8. A virtual reality interactive device, characterized in that, The virtual reality interaction device includes: a memory, a processor, and a virtual reality interaction program stored in the memory and executable on the processor, wherein the virtual reality interaction program, when executed by the processor, implements the steps of the virtual reality interaction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a virtual reality interaction program, which, when executed by a processor, implements the steps of the virtual reality interaction method as described in any one of claims 1 to 6.

Images