Virtual interaction method based on image recognition and head-mounted display equipment

Abstract

The invention relates to the technical field of VR / AR (Virtual Reality / Augmented Reality), and provides a virtual interaction method based on image recognition and head-mounted display equipment, which are used for realizing extended interaction of 3D (Three-Dimensional) application. According to the method, an image recognition function is added in a target 3D application, the function is turned on or turned off through an intelligent interaction switch, when the intelligent interaction switch is turned on, if the parking time of a focus on a current display page on a sky box exceeds a preset time threshold value, it is indicated that a target user is interested in an object on the sky box, and at the moment, the target user is interested in the object on the sky box. According to the embodiment of the invention, image recognition can be performed on the target area on the sky box, and the interactive operation of the target user is responded according to the recognized target object, so that the interactive operation of the target user is responded according to the target model or UI option on the current display page selected according to the focus; and the interaction operation of the target user can be responded according to the target object on the sky box selected by the focus, so that the expandability of interaction in the 3D scene is improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) / augmented reality (AR) technology, and provides a virtual interaction method based on image recognition and a head-mounted display device. Background Art

[0002] The 3D space displayed by head-mounted display devices (HMD) such as VR / AR, taking U3D applications as an example, is generally composed of a skybox and a 3D model, where the skybox serves as the background, such as Figure 1A As shown in, the 3D model is used as the target of attention, such as Figure 1B As shown in the figure, combined with the UI panel mainly used for interaction, a common typical 3D interaction scene is formed, such as Figure 1C shown.

[0003] At present, the interaction in the 3D space displayed by head-mounted display devices such as VR / AR is usually triggered by pre-setting. That is to say, the head-mounted display device will only respond to the user operation when the user operates the pre-set operation items (such as options in the UI panel, 3D objects, etc.), and there are certain interaction restrictions. Summary of the invention

[0004] The embodiments of the present application provide a virtual interaction method and a head-mounted display device based on image recognition, which are used to improve the scalability of virtual interaction.

[0005] On the one hand, an embodiment of the present application provides a virtual interaction method based on image recognition, which is applied to a head-mounted display device, including:

[0006] Start the target 3D application and turn on the smart interaction switch;

[0007] Get the focus status of the current display page, where the current display page includes the skybox, 3D model and UI interface;

[0008] When the focus is on the skybox, do the following:

[0009] Start the timer;

[0010] If the dwell time of the focus exceeds a preset time threshold, image recognition is performed on the target area on the sky box;

[0011] Responding to the target user's interactive operation based on the identified target object;

[0012] When the focus is located on the 3D model or the UI interface, the interactive operation of the target user is responded to according to the target model or UI option selected by the focus.

[0013] On the other hand, an embodiment of the present application provides a head mounted display device, comprising a processor, a memory and a display, wherein the display, the memory and the processor are connected via a bus;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Start the target 3D application and turn on the smart interaction switch;

[0016] Obtaining a focus state on a current display page displayed on the display, wherein the current display page includes a skybox, a 3D model, and a UI interface;

[0017] When the focus is on the skybox, do the following:

[0018] Start the timer;

[0019] If the dwell time of the focus exceeds a preset time threshold, image recognition is performed on the target area on the sky box;

[0020] Responding to the target user's interactive operation based on the identified target object;

[0021] When the focus is located on the 3D model or the UI interface, the interactive operation of the target user is responded to according to the target model or UI option selected by the focus.

[0022] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a virtual interaction method based on image recognition provided in an embodiment of the present application.

[0023] The beneficial effects of the virtual interaction method based on image recognition provided by the embodiment of the present application are as follows:

[0024] The image recognition function is added to the target 3D application, and the image recognition function is turned on and off through the smart interaction switch. When the smart interaction switch is turned on, if the focus on the current display page stays on the sky box for more than the preset time threshold, it indicates that the target user is interested in the object on the sky box. At this time, the target area on the sky box can be image recognized, and the target user's interactive operation can be responded to according to the recognized target object, which expands the interactive content on the display page. In addition to responding to the target user's interactive operation according to the target model or UI option on the current display page selected by the focus, the target user's interactive operation can also be responded to according to the target object on the sky box selected by the focus, which improves the scalability of interaction in the 3D scene, can meet the user's interest in objects in the background, and thus improve the user's interactive experience.

[0025] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0027] Figure 1A A schematic diagram of a skybox in a 3D scene provided in an embodiment of the present application;

[0028] Figure 1B A schematic diagram of a 3D model in a 3D scene provided in an embodiment of the present application;

[0029] Figure 1C A schematic diagram of a complete interactive page in a 3D scene provided in an embodiment of the present application;

[0030] Figure 2 A flowchart of a virtual interaction method based on image recognition provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of a currently displayed page of a target 3D application provided in an embodiment of the present application;

[0032] Figure 4 A flow chart of an extended interaction method based on image recognition provided in an embodiment of the present application;

[0033] Figure 5 The image recognition effect diagram provided by the embodiment of the present application;

[0034] Figure 6 A page jump effect diagram based on a target object provided in an embodiment of the present application;

[0035] Figure 7 A complete 3D interaction method flow chart provided in the embodiment of the present application;

[0036] Figure 8 A structural diagram of a head-mounted display device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0038] At present, when interacting in the 3D space displayed by head-mounted display devices such as VR / AR, in addition to operating the preset UI panel and the 3D object itself, the skybox is displayed as a background and usually cannot respond to different interactive operations. For some objects or scenes that need to be highlighted in the user's visible area, it is impossible to select them locally, and the dynamic scalability is poor. At the same time, for manufacturers, in addition to installing their own preset 3D applications on the head-mounted display device, the skybox in the installed third-party application cannot respond to the user's interactive operations.

[0039] In view of this, the embodiment of the present application provides a virtual interaction method based on image recognition. During the interaction process, when the focus in the page is located on a non-UI interface or a non-preset (i.e., capable of generating an action response) 3D object, image recognition is triggered to identify the target object in the area near the focus, and a prompt of the target object is given by querying the preset interaction library. At this time, the user can operate the target object according to the prompt to achieve extended interaction with the head-mounted display device. This method can improve the diversity of interactive content in 3D space, especially the scalability of the interactive skybox background, and can meet the user's exploration of different objects in the field of view, thereby enhancing the user's immersive experience.

[0040] See also Figure 2 , is a flow chart of a virtual interaction method based on image recognition provided in an embodiment of the present application. The process is executed by a head mounted display device, including but not limited to a VR device and an AR device, and mainly includes the following steps:

[0041] S201: Start the target 3D application and turn on the smart interaction switch.

[0042] The intelligent interactive switch is used to control whether the image recognition function is turned on, and the intelligent interactive switch can be a hardware switch or a software switch.

[0043] S202: Obtain the focus status on the current display page, where the current display page includes a sky box, a 3D model, and a UI interface.

[0044] At present, the common interaction modes of head-mounted display devices include handle control and head control. In the case of handle control, the rays emitted by the handle form a focus on the display page of the head-mounted display device, and the focus on the display page is moved by moving the handle; in the case of head control, the focus on the display page is moved by shaking the head.

[0045] Take handle control as an example. Figure 3 As shown, it is a schematic diagram of the current display page of the target 3D application provided in the embodiment of the present application, and the display page includes a skybox, a 3D model (such as a recliner, a pavilion, a telescope, etc.) and a UI interface. Among them, the skybox as a whole is displayed as a background, and the 3D model and the UI interface can respond to the interactive operation triggered by the user. The user can change the focus of the handle ray and the display page by operating the handle.

[0046] S203: Determine whether the focus is on the sky box, if so, execute S204, if it is on the 3D model or UI interface, execute S207.

[0047] As the background of the 3D scene, the skybox occupies most of the display area of ​​the page, and in order to improve the richness and beauty of the 3D scene, it usually also contains some foils, such as Figure 3 Hot air balloon in the sky.

[0048] During the interaction, the focus can be moved across the entire display page and can fall on the skybox, the 3D model, or the UI interface.

[0049] S204-S205: Start the timer to determine whether the focus stay time exceeds a preset time threshold. If so, execute S206; if not, return to S202.

[0050] When the focus is on the sky box, a timer is started to count the time the focus stays on the sky box, and the stay time is compared with a preset time threshold. If the stay time exceeds the preset time threshold, it indicates that the target user may be interested in the object on the sky box, and the image recognition function can be started to expand the interactive content of the display page.

[0051] Optionally, the preset time threshold is 6 seconds.

[0052] It should be noted that the embodiment of the present application does not impose any restrictions on the time threshold and can be set according to actual needs.

[0053] S206: Perform image recognition on the target area on the sky box, and respond to the interactive operation of the target user according to the recognized target object.

[0054] When the focus stays on the sky box for longer than a preset time threshold, an artificial intelligence (AI) algorithm is called to perform image recognition on the sky box to expand the interactive content on the display page.

[0055] In specific implementation, the process of extending interaction is as follows Figure 4 As shown, it mainly includes the following steps:

[0056] S2061: Determine a target area on the sky box with the focus as the center and a preset number of pixels as the radius.

[0057] Since the area of ​​the skybox is large, in order to improve the response speed of the interaction, a target area can be determined according to the focus to narrow the scope of image recognition.

[0058] In one example, the target area may be a circle, such as a circle with the focus as the center and a preset number of pixels as the radius. Optionally, the preset number of pixels may be adjusted according to actual needs, such as being set to 100 pixels in this application.

[0059] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the shape of the target area. For example, it can also be a square, rectangle, ellipse or irregular polygon.

[0060] S2062: Extracting features of objects in the target area, matching the features of the target object with features of candidate objects in a preset interaction library, and identifying the category of the target object.

[0061] In one example, an AI algorithm is used to extract object features in the target area (such as the object's outline, color, shape, and other information), and calculate their feature similarity with the candidate objects in the preset interaction library. The higher the similarity, the more matched they are. Therefore, the category of the candidate object with the highest similarity is used as the category of the target object.

[0062] For example, Figure 5 As shown, by performing image recognition on the target area where the focus is located, it can be identified that the target object selected by the focus is a hot air balloon and is circled by a circular frame.

[0063] S2063: Based on the category of the target object, an interactive prompt pop-up window is provided near the target object.

[0064] After identifying the category of the target object, an interactive prompt pop-up window is given near the target object according to the category of the target object, such as Figure 5 shown.

[0065] S2064: Determine whether the target user's operation on the interactive prompt pop-up window is received within the preset time period. If so, execute S2065; if not, execute S2066.

[0066] Optionally, the preset duration is 4 seconds, which can be adjusted according to actual needs in the application.

[0067] S2065: Jump to a new 3D scene corresponding to the target object.

[0068] When an operation triggered by the target user on the interactive prompt pop-up window is received within a preset time period, in response to the operation triggered by the target user on the interactive prompt pop-up window, jump to a new 3D scene corresponding to the target object.

[0069] In specific implementation, a new 3D scene pre-set for the target object can be loaded from the local system and displayed, and a new 3D scene corresponding to the target object can be downloaded from the application platform and displayed.

[0070] For example, when the target user clicks Figure 5 When a hot air balloon pops up in the head mounted display, the head mounted display jumps to Figure 6 The 3D scene page of the hot air balloon shown in the figure realizes the extended interaction of the sky box.

[0071] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the interactive content corresponding to the target object on the sky box. In addition to displaying the new 3D scene corresponding to the target object, the video of the target object can also be played.

[0072] S2066: Hide the interactive prompt pop-up window.

[0073] When no operation is received from the target user on the interactive prompt pop-up window within the preset time, it may be because the target user temporarily leaves or is distracted, causing the focus to stay on the skybox. At this time, no response is required and the interactive prompt pop-up window can be hidden.

[0074] S207: Respond to the target user's interactive operation according to the target model or UI option selected by focus.

[0075] In the target 3D application, by pre-configuring interactive response behaviors for the 3D model and UI interface in the display page, when the focus is on the 3D model or UI interface, it can respond to the target user's interactive operations according to the target model or UI option selected by the focus.

[0076] For example, when the "screenshot" icon on the UI interface gains focus, when the target user triggers a confirmation operation through a handle button, the head-mounted display device begins to take a screenshot of the currently displayed page.

[0077] For another example, when the "telescope" model in the 3D model gains focus, when the target user triggers a confirmation operation through a handle button, the head-mounted display device jumps to the starry sky scene for display.

[0078] In one example, when the target user turns off the smart interaction switch, the extended interaction function based on image recognition is turned off, and normal interaction is performed according to the preset content in the target 3D application. Specifically, the head mounted display device directly responds to the target user's interactive operation according to the target model or UI option selected by the focus on the current display page.

[0079] See also Figure 7 , is a complete interactive method flow chart provided in the embodiment of the present application, and the process mainly includes the following steps:

[0080] S701: Obtain the type of the currently started target application.

[0081] S702: Determine whether the target application is a 3D application, if so, execute S703, if it is a 2D application, execute S711.

[0082] S703: Determine whether the intelligent interaction switch is turned on, if so, execute S704, if not, execute S710.

[0083] S704: Obtain the focus status on the currently displayed page.

[0084] S705: Determine whether the focus is located on the sky box of the currently displayed page. If so, execute S706; if it is located on the 3D model or UI interface of the currently displayed page, execute S710.

[0085] S706: Start the timer to obtain the time the focus stays on the sky box.

[0086] S707: Determine whether the parking time exceeds a preset time threshold, if so, execute S708, if not, return to S704.

[0087] S708: Perform image recognition on the target area where the focus is located on the sky box.

[0088] S709: Determine whether the identified target object matches an object in a preset interaction library. If so, execute S710; if not, end.

[0089] S710: Respond to the target user's interactive operation according to the target model or UI option selected by focus.

[0090] S711: Perform 2D interaction.

[0091] In an embodiment of the present application, an image recognition function is added to the skybox. In this way, when the focus is on the UI interface that the target user is paying attention to or on the skybox outside the 3D model that has an action response, image recognition is triggered, the target object within the target area where the focus is located is identified, and a triggerable interaction prompt pop-up window is dynamically added to the identified target object to highlight the background part of interest in the visible field of view. The target user triggers the interactive behavior on the selected target object by operating the interactive prompt pop-up window, and obtains the response result of the interactive behavior generated on the target object, thereby improving the scalability of interaction in 3D applications and further improving the user experience.

[0092] Based on the same technical concept, an embodiment of the present application provides a head-mounted display device, which can be a virtual reality device or an augmented reality device. The head-mounted display device can implement the steps of the above-mentioned virtual interaction method based on image recognition and can achieve the same technical effect.

[0093] See also Figure 8 , the head mounted display device comprises a processor 801, a memory 802 and a display 803, wherein the display 803, the memory 802 and the processor 801 are connected via a bus 804;

[0094] The memory 802 stores a computer program, and the processor 801 performs the following operations according to the computer program:

[0095] Start the target 3D application and turn on the smart interaction switch;

[0096] Obtaining the state of the focus on the current display page displayed by the display 803, wherein the current display page includes a sky box, a 3D model and a UI interface;

[0097] When the focus is on the skybox, do the following:

[0098] Start the timer;

[0099] If the dwell time of the focus exceeds a preset time threshold, image recognition is performed on the target area on the sky box;

[0100] Responding to the target user's interactive operation based on the identified target object;

[0101] When the focus is located on the 3D model or the UI interface, the interactive operation of the target user is responded to according to the target model or UI option selected by the focus.

[0102] Optionally, the processor 801 performs image recognition on the target area on the sky box, and the specific operation is:

[0103] Determine a target area on the sky box with the focus as the center and a preset number of pixels as the radius;

[0104] The object features in the target area are extracted, the object features are matched with the features of candidate objects in a preset interaction library, and the category of the target object is identified.

[0105] Optionally, the processor 801 responds to the interactive operation of the target user according to the identified target object, and the specific operation is:

[0106] According to the category of the target object, an interactive prompt pop-up window is provided near the target object;

[0107] If no operation of the target user triggering the interaction prompt pop-up window is received within a preset time period, hiding the interaction prompt pop-up window;

[0108] If the target user's operation to trigger the interactive prompt pop-up window is received within a preset time period, the process jumps to a new 3D scene corresponding to the target object.

[0109] Optionally, the processor 801 jumps to a new 3D scene corresponding to the target object, and the specific operations are:

[0110] Loading a new 3D scene pre-set for the target object from the local system and displaying it;

[0111] A new 3D scene corresponding to the target object is downloaded from the application platform and displayed.

[0112] Optionally, the processor 801 further executes:

[0113] When the intelligent interaction switch is turned off, the interactive operation of the target user is responded to according to the target model or UI option selected in focus on the currently displayed page.

[0114] in, Figure 8The memory 802 in the embodiment may be a volatile memory, such as a random-access memory (RAM); the memory 802 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 802 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories; the processor 801 may include one or more central processing units (CPU) or a digital processing unit, etc.

[0115] It should be noted that Figure 8 This is just an example, and provides the necessary hardware for a head mounted display device to execute the steps of a virtual interaction method based on image recognition provided in an embodiment of the present application. If not shown, the head mounted display device may also include the hardware of conventional VR / AR devices such as speakers, microphones, IMU, communication interfaces, and power supplies.

[0116] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete a virtual interaction method based on image recognition in the aforementioned embodiment.

[0117] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a virtual interaction method based on image recognition in the aforementioned embodiment.

[0118] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0119] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram, and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart or multiple flows and / or one box or multiple boxes in the block diagram.

[0120] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0122] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A virtual interaction method based on image recognition, characterized in that: Applied to head-mounted display devices, including: Start the target 3D application and turn on the smart interaction switch; Get the focus status of the current display page, where the current display page includes the skybox, 3D model and UI interface; When the focus is on the skybox, do the following: Start the timer; If the dwell time of the focus exceeds a preset time threshold, image recognition is performed on the target area on the sky box; Responding to the target user's interactive operation based on the identified target object; When the focus is located on the 3D model or the UI interface, the interactive operation of the target user is responded to according to the target model or UI option selected by the focus.

2. The method according to claim 1, characterized in that The performing image recognition on the target area on the sky box includes: Determine a target area on the sky box with the focus as the center and a preset number of pixels as the radius; The object features in the target area are extracted, the object features are matched with the features of candidate objects in a preset interaction library, and the category of the target object is identified.

3. The method according to claim 1, characterized in that: The step of responding to the target user's interactive operation according to the identified target object includes: According to the category of the target object, an interactive prompt pop-up window is provided near the target object; If no operation of the target user on the interaction prompt pop-up window is triggered within a preset time period, hiding the interaction prompt pop-up window; If the target user's operation to trigger the interactive prompt pop-up window is received within a preset time period, the process jumps to a new 3D scene corresponding to the target object.

4. The method according to claim 3, characterized in that The jumping to the new 3D scene corresponding to the target object includes: Loading a new 3D scene pre-set for the target object from the local system and displaying it; A new 3D scene corresponding to the target object is downloaded from the application platform and displayed.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: When the intelligent interaction switch is turned off, the interactive operation of the target user is responded to according to the target model or UI option selected in focus on the currently displayed page.

6. A head mounted display device, characterized in that: It includes a processor, a memory and a display, wherein the display, the memory and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Start the target 3D application and turn on the smart interaction switch; Obtaining a focus state on a current display page displayed on the display, wherein the current display page includes a skybox, a 3D model, and a UI interface; When the focus is on the skybox, do the following: Start the timer; If the dwell time of the focus exceeds a preset time threshold, image recognition is performed on the target area on the sky box; Responding to the target user's interactive operation based on the identified target object; When the focus is located on the 3D model or the UI interface, the interactive operation of the target user is responded to according to the target model or UI option selected by the focus.

7. The head mounted display device according to claim 6, wherein: The processor performs image recognition on the target area on the sky box, and the specific operation is: Determine a target area on the sky box with the focus as the center and a preset number of pixels as the radius; The object features in the target area are extracted, the object features are matched with the features of candidate objects in a preset interaction library, and the category of the target object is identified.

8. The head mounted display device according to claim 6, wherein: The processor responds to the target user's interactive operation according to the identified target object, and the specific operation is: According to the category of the target object, an interactive prompt pop-up window is provided near the target object; If no operation of the target user on the interaction prompt pop-up window is triggered within a preset time period, hiding the interaction prompt pop-up window; If the target user's operation to trigger the interactive prompt pop-up window is received within a preset time period, the process jumps to a new 3D scene corresponding to the target object.

9. The head mounted display device according to claim 8, wherein: The processor jumps to the new 3D scene corresponding to the target object, and the specific operations are: Loading a new 3D scene pre-set for the target object from the local system and displaying it; A new 3D scene corresponding to the target object is downloaded from the application platform and displayed.

10. The head mounted display device according to any one of claims 6 to 9, characterized in that: The processor also executes: When the intelligent interaction switch is turned off, the interactive operation of the target user is responded to according to the target model or UI option selected in focus on the currently displayed page.

Images