A 9D interactive method, system, electronic device and storage medium

Abstract

The embodiment of the present invention discloses a 9D interaction method, system, electronic device and storage medium, wherein the method includes: obtaining a naked eye 3D scene through projection of a projection device; tracking the gaze point of a target person in real time to determine the target object corresponding to the gaze point; using the target object as a query condition to query and obtain at least one trigger action associated with the target object; obtaining the video data of the target person in real time through a camera; performing action detection based on the video data, and under the condition of detecting the trigger action: obtaining the pointing direction of the four fingers of the target person; determining four position points in the naked eye 3D scene based on the pointing direction, and moving the target object based on the four position points and the gaze point. The implementation of the embodiment of the present application improves the interaction efficiency of the 9D interaction scene while improving the accuracy of object movement.

Description

Technical Field

[0001] The present invention relates to the field of digital intelligence services, specifically to the field of digital intelligence services for three-dimensional models, and in particular to a 9D interaction method, system, electronic device and storage medium. Background Art

[0002] Although the current immersive cinema technology has brought a new immersive experience to users, there are still many challenges in the interaction between people and devices. In the current environment, the accuracy, response speed and ease of use of user interaction methods such as gesture recognition and voice control need to be improved. In addition, in the current immersive scene, users only watch content passively, and it is impossible to flexibly adjust the display content in the immersive scene based on user needs, resulting in a poor user experience. Summary of the invention

[0003] The embodiments of the present invention provide a 9D interaction method, system, electronic device and storage medium, which can combine character gestures and gaze points for interaction, thereby improving interaction efficiency and interaction accuracy.

[0004] On the one hand, an embodiment of the present invention provides a 9D interaction method, which is applied to a 9D interaction system in a 9D cinema, wherein the 9D interaction system includes a camera, a projection device, a stereo audio device, a fragrance propagation device, an environment control device, a manned device, and a blowing device, wherein the manned device includes a moving device, a rotating device, and a vibrating device, and the method includes:

[0005] The naked eye 3D scene projected by the projection device;

[0006] Track the gaze point of the target person in real time and determine the target object corresponding to the gaze point;

[0007] Taking the target object as a query condition, querying and obtaining at least one trigger action associated with the target object;

[0008] Acquire video data of the target person in real time through the camera;

[0009] Performing action detection according to the video data, and upon detecting the triggering action:

[0010] Obtain the pointing directions of the four fingers of the target person;

[0011] Determine four position points in the naked eye 3D scene based on the pointing direction, wherein the four position points are respectively located on a straight line where the four fingers are located;

[0012] Determine whether the gaze point and the four position points are located at the target object. If so, establish a spatial rectangular coordinate system O-xyz with the position of the target person as the origin, determine a first plane perpendicular to the xOy plane based on the moving positions of the four position points, and the four position points and the fingertips of the four fingers are located in the first plane;

[0013] Displaying an intersection line between the first plane and the object in the naked eye 3D scene;

[0014] Detecting the position of the gaze point in real time, and when it is detected that the gaze point is located at an intersection point, moving the target object to the intersection point, wherein the intersection point is a point on the intersection line;

[0015] When it is detected that the angles between the four fingers are greater than a preset angle threshold, the interaction process is terminated.

[0016] Optional, 9D refers to naked eye 3D, vibration, forward and backward movement, rotation, smell, wind, rain and snow;

[0017] Optionally, the environmental control device can create rain and snow, and combined with the blowing device, a natural scene of rain and snow drifting can be constructed.

[0018] Optionally, the naked eye 3D scene may be a scene of moving objects, such as a house construction scene, a building block stacking scene, etc.

[0019] Optionally, in a naked-eye 3D scene, different operations can be performed on different objects, and each operation can correspond to different trigger actions. For example, the object can be a bucket, and there can be two operations for the bucket, one of which is to move the bucket, and the other is to pour the water out of the bucket. The trigger action for moving the bucket can be to raise the arm and make a fist, and the trigger action for pouring the water out of the bucket can be to raise the arm with the palm facing outward.

[0020] Optionally, the four fingers are the index finger, the middle finger, the ring finger and the little finger; the pointing direction of the four fingers is specifically the direction in which the distal segments of the four fingers are pointing, that is, the pointing direction of the part where the nail is located.

[0021] Optionally, the 9D interactive system can connect the four projection points in sequence, and fit the line connecting the four points into a straight line when the turning angle is less than a preset value; similarly, the 9D interactive system can fit the fingertips of the four fingers into straight lines, and obtain the first plane based on the fitting of the two straight lines.

[0022] Optionally, in the rectangular coordinate system O-xyz, the xOy plane is a plane parallel to the horizontal plane.

[0023] Optionally, the intersection of the first plane and the object in the naked eye 3D scene forms an intersection line, and when the gaze point is located at a certain intersection point on the intersection line, the position of the intersection point is determined as the position to which the object needs to be moved, and the object is moved to the position of the intersection point. Specifically, the 9D interactive system can display the intersection line formed by the intersection point of the first plane and the object in the naked eye 3D scene in real time, so that the user can determine the degree of finger movement.

[0024] Optionally, the preset angle threshold between the four fingers may be 15°, and when the angle between the four fingers is greater than 15°, the above interaction process is terminated; the angle between the four fingers may be the angle between the straight lines where the four fingers are located.

[0025] It can be seen that in the embodiment of the present application, the user can perform double verification based on the pointing position and gaze point of the finger, and move the object to the corresponding position based on the verification result, thereby improving the interaction efficiency of the 9D interactive scene while improving the accuracy of object movement.

[0026] In a possible implementation manner, the performing action detection according to the video data, under the condition that the triggering action is detected, includes:

[0027] Decoding the video data, and extracting a picture sequence therefrom at intervals of a preset length, wherein the first frame of the picture sequence is a and the length is n;

[0028] Input the picture sequence of length n into the lightweight action detection model, wherein the action detection model includes a first convolution module, a pooling layer, a second convolution module, a third convolution module, a fourth convolution module, and a fifth convolution module; the convolution kernel of the first convolution module is 3*5*5; the second convolution module, the third convolution module, the fourth convolution module, and the fifth convolution module each include a resnet residual block;

[0029] When the lightweight action detection model detects a predetermined action, a picture sequence with a first frame of a and a length of m is input into the action classification model for action detection, and the type of the final triggered action is output; wherein m is greater than n, and the picture sequence with a length of m includes a picture sequence with a length of n in which the predetermined action is detected.

[0030] It should be noted that the preset length can be set based on actual needs. For example, the preset length can be 30 frames, 50 frames, or 100 frames.

[0031] It should be noted that in 9D interaction, the data processing volume of acquiring video data for a long time is large, and directly accurately detecting the trigger action requires a large amount of computing power. Therefore, this solution sets up a lightweight action detection model to first detect the predetermined action, and then use the action classification model to accurately detect the trigger action type after detecting the predetermined action, so as to reduce the data processing volume.

[0032] Optionally, the trigger action may be shaking the right index finger, extending the right index finger and middle finger at the same time, extending the five fingers of the right hand, interlocking the left and right thumbs, etc. The designer may design a number of gesture actions. Meanwhile, the predetermined action may be raising the hand, raising the elbow, etc.

[0033] It should be noted that the number of input image frames of the action detection model is relatively small, for example, it is set to 3, 4 or 5 frames, etc., and the processing speed is accelerated based on the context information of the action. After the predetermined action is detected, a picture sequence of length m is input into the action classification model for action detection, where m is greater than n and the picture sequence of length m contains a picture sequence of length n in which the predetermined action is detected. The purpose of this setting is to take a longer picture sequence and put it into the action classification model to obtain a more accurate detection result after obtaining a signal that a trigger action is about to occur.

[0034] When the action detection model detects a predetermined action, a picture sequence with a first frame a and a length m is input into the action classification model for action detection, and the type of the final triggering action output includes:

[0035] Perform detection based on cameras set at different angles to obtain depth image sequences;

[0036] Input the picture sequence with the first frame a and the length m into the first branch of the action classification model to perform multi-scale feature extraction, and obtain the first picture feature, the second picture feature, the third picture feature, the fourth picture feature and the fifth picture feature from large to small sizes respectively;

[0037] Inputting the depth image sequence into the first branch of the action classification model to perform multi-scale feature extraction, and obtaining a first depth feature, a second depth feature, a third depth feature, a fourth depth feature and a fifth depth feature in five sizes from large to small respectively;

[0038] The first image feature and the first depth feature are fused to obtain a first fused feature; the second image feature and the second depth feature are fused to obtain a second fused feature; the third image feature and the third depth feature are fused to obtain a third fused feature; the fourth image feature and the fourth depth feature are fused to obtain a fourth fused feature; the fifth image feature and the fifth depth feature are fused to obtain a fifth fused feature;

[0039] The depth features, image features and fusion features of the same scale are multiplied by different weights respectively, and input into the softmax layer to output the type of the trigger action.

[0040] For example, the triggering action is to raise the arm and make a fist, and the predetermined action may be to raise the arm. When the arm is raised, a corresponding image sequence of length m is obtained and input into the action classification model.

[0041] It should be noted that when classifying the types of trigger actions, due to the delicacy of gesture actions, in order to obtain more accurate classification results, a depth image sequence is additionally acquired in the embodiment of the present application, and the depth information is combined with the information in the image sequence, which can more accurately distinguish the relatively similar left and right hand features, and the features of different fingers on the same hand.

[0042] In one possible implementation, the training process of the action classification model includes: using a camera to capture action images and their corresponding depth images in different light and backgrounds; performing contrast enhancement operations on the action images and their corresponding depth images; using random flipping and matrix processing of the data set to expand the data set; and using the expanded data set to train the action classification model.

[0043] In a possible implementation, the method also includes: detecting the position of each body part of the target person through the camera; obtaining the position of the target air outlet of the naked-eye 3D scene, the target air outlet being any one of the following: a window, a door, or a cave entrance; determining the target plane where the target air outlet is located; projecting the position of each body part of the target person onto the target plane to obtain a projection area; determining the overlapping part of the projection area and the first area in the target plane; the first area is the area occupied by the target air outlet; determining the target body part of the target person based on the overlapping part; and starting the blowing device to blow air to the target device.

[0044] In a possible implementation, the blowing device is composed of invisible blowing holes, and the invisible blowing holes are evenly arranged on the surrounding walls, the ground and the ceiling.

[0045] In a possible implementation, before using the target object as a query condition to query and obtain at least one trigger action associated with the target object, the method further includes: demonstrating a preset type of scene through the naked-eye 3D scene; receiving an action entry instruction input by a user; acquiring the user's action through the camera, and storing the first action when it is detected that the user maintains the first action for a preset time; and using the first action as a trigger action for a corresponding type of scene.

[0046] In one possible implementation, the passenger device can be associated with any of the following scenes: a car scene, a plane scene, or a ship scene; the 9D interactive system controls the forward, backward, left, right, rotation, and vibration of the passenger device through specific scene information in the naked-eye 3D scene to simulate the car scene, plane scene, or ship scene.

[0047] On the other hand, an embodiment of the present invention provides a 9D interactive system, including:

[0048] The 9D interactive system is applied to 9D cinemas, and the 9D interactive system includes a camera, a projection device, a stereo audio device, a fragrance dissemination device, an environmental control device, a manned device, and a blowing device. The manned device includes a moving device, a rotating device, and a vibrating device. The 9D interactive system is used to perform the following method steps: obtaining a naked-eye 3D scene projected by the projection device; tracking the gaze point of a target person in real time to determine the target object corresponding to the gaze point; using the target object as a query condition to query and obtain at least one trigger action associated with the target object; obtaining video data of the target person in real time through the camera; performing action detection based on the video data, and under the condition that the trigger action is detected: obtaining the pointing direction of four fingers of the target person; based on the pointing direction Determine four position points in the naked-eye 3D scene, and the four position points are respectively located on the straight line where the four fingers are located; judge whether the gaze point and the four position points are located at the target object, and if so, establish a spatial rectangular coordinate system O-xyz with the position of the target person as the origin, and determine a first plane perpendicular to the xOy plane based on the moving positions of the four position points, and the four position points and the fingertips of the four fingers are located in the first plane; display the intersection line of the first plane and the object in the naked-eye 3D scene; detect the position of the gaze point in real time, and when it is detected that the gaze point is located on the intersection, move the target object to the intersection, and the intersection is a point on the intersection line; when it is detected that the angle between the four fingers is greater than a preset angle threshold, end the interaction process.

[0049] On the other hand, an embodiment of the present invention provides an electronic device, comprising: a processor, suitable for implementing one or more instructions; and a computer storage medium, wherein the computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by the processor and executing the above-mentioned 9D interaction method.

[0050] On the other hand, an embodiment of the present invention provides a computer storage medium, characterized in that the computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the above-mentioned 9D interaction method.

[0051] It can be seen that the user can perform double verification based on the finger's pointing position and gaze point, and move the object to the corresponding position based on the verification result, thereby improving the interaction efficiency of the 9D interactive scene while improving the accuracy of object movement.

[0052] Furthermore, when detecting trigger actions, the solution of the present application adopts a process of first detecting the action and then accurately classifying it, wherein the detection of the action uses a smaller number of image sequence frames than the accurate classification, so as to improve the accuracy of trigger action detection while ensuring the processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0054] Figure 1 is a schematic diagram of a 9D interactive scene provided by an embodiment of the present invention;

[0055] Figure 2A is a flowchart of a 9D interaction method provided by an embodiment of the present invention;

[0056] Figure 2B is a schematic diagram of a network model in a 9D interaction method provided by an embodiment of the present invention;

[0057] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0059] like Figure 1 As shown, Figure 1 is a schematic diagram of a 9D interaction scene provided in an embodiment of the present application. A 9D interaction method provided in an embodiment of the present application can be used in Figure 1A schematic diagram of a 9D interactive scene is shown; wherein, a target person is included, and the angle between the four fingers of the target person is less than a preset angle threshold, specifically, the preset angle threshold may be 15°; the corresponding four points A1, A2, A3 and A4 may be determined based on the directions pointed by the four fingers; in the 9D interactive scene, first, when the gaze point of the target person and the four points A1, A2, A3 and A4 select the target object, then the target person may control the pointing direction of the four fingers, and at the same time, the 9D interactive system may construct a first plane based on the fingertips of the four fingers and the four points A1, A2, A3 and A4, wherein the first plane is perpendicular to the horizontal plane; the first plane is Figure 1 The plane formed by the dotted line in the middle; the first plane intersects with the object / plane in the projected scene in the 9D interaction, and the four points A1, A2, A3 and A4 are located on the intersection of the two planes. When the target person's line of sight falls on the intersection, that is, the gaze point is located at point B1 on the intersection, then point B1 is determined as the moving position point, and the target object is moved to point B1. When the angle between the four fingers is greater than the preset angle threshold of 15°, the position movement process for the target object ends.

[0060] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order.

[0061] Based on the above description, an embodiment of the present invention provides a 9D interaction method, which can be executed by an electronic device. Figure 2A The 9D interactive method is applied to a 9D interactive system in a 9D cinema. The 9D interactive system includes a camera, a projection device, a stereo audio device, a fragrance diffusion device, an environmental control device, a manned device, and a blowing device. The manned device includes a moving device, a rotating device, and a vibrating device. The method may include the following steps S201-S206:

[0062] S201, obtaining a naked-eye 3D scene through projection of the projection device;

[0063] Among them, 9D refers to naked eye 3D, vibration, front and back left and right movement, rotation, smell, wind, rain and snow; environmental control equipment can create rain and snow, and combined with blowing equipment, it can construct a natural rain and snow drifting scene; naked eye 3D scene can be a scene of moving objects, such as house construction scene, building block stacking scene, etc.

[0064] S202, tracking the gaze point of the target person in real time, and determining the target object corresponding to the gaze point;

[0065] S203, using the target object as a query condition, querying and obtaining at least one triggering action associated with the target object;

[0066] S204, acquiring video data of the target person in real time through the camera;

[0067] S205, performing action detection according to the video data, and upon detecting the triggering action:

[0068] Obtain the pointing directions of the four fingers of the target person;

[0069] Among them, the four fingers are the index finger, middle finger, ring finger and little finger; the pointing direction of the four fingers is specifically the direction of the distal segments of the four fingers, that is, the pointing direction of the part where the nail / fingertip is located.

[0070] Wherein, step S205 includes steps S2051-S2053:

[0071] Step S2051, decoding the video data, and extracting a picture sequence therefrom at intervals of a preset length, wherein the first frame of the picture sequence is a and the length is n;

[0072] Step S2052: input the picture sequence of length n into a lightweight action detection model, wherein the action detection model includes a first convolution module, a pooling layer, a second convolution module, a third convolution module, a fourth convolution module, and a fifth convolution module; the convolution kernel of the first convolution module is 3*5*5; the second convolution module, the third convolution module, the fourth convolution module, and the fifth convolution module each include a resnet residual block;

[0073] Step S2053: when the lightweight action detection model detects a predetermined action, a picture sequence with a first frame a and a length of m is input into the action classification model for action detection, and the type of the final triggering action is output; wherein m is greater than n, and the picture sequence with a length of m includes a picture sequence with a length of n in which the predetermined action is detected;

[0074] It should be noted that the preset length can be set based on actual needs. For example, the preset length can be 30 frames, 50 frames or 100 frames, etc., and the preset length can be based on different projection scenes of the 9D interactive system.

[0075] For example, see Figure 2B , Figure 2BIt is a schematic diagram of a network model in a 9D interaction method provided by an embodiment of the present invention; wherein, the 9D interaction system extracts a picture sequence every 8 frames, and n can be 3 frames; that is, 3 frames of images are extracted every 8 frames in the video image sequence and input into the action detection model; when the action detection model detects a predetermined action, the 8 frames of images corresponding to the 3-frame image sequence are input into the action classification model, so as to further accurately identify the type of triggering action.

[0076] It should be noted that in 9D interaction, the data processing volume of acquiring video data for a long time is large, and directly accurately detecting the trigger action requires a large amount of computing power. Therefore, this solution sets up a lightweight action detection model to first detect the predetermined action, and then use the action classification model to accurately detect the trigger action type after detecting the predetermined action, so as to reduce the data processing volume.

[0077] In a possible implementation, the trigger action may be shaking the right index finger, extending the right index finger and middle finger at the same time, extending the five fingers of the right hand, interlocking the left and right thumbs, etc. The designer may design a number of gesture actions. Meanwhile, the predetermined action may be raising the hand, raising the elbow, etc.

[0078] It should be noted that the number of input image frames of the action detection model is relatively small, for example, it is set to 3, 4 or 5 frames, etc., and the processing speed is accelerated based on the context information of the action. After the predetermined action is detected, a picture sequence of length m is input into the action classification model for action detection, where m is greater than n and the picture sequence of length m contains a picture sequence of length n in which the predetermined action is detected. The purpose of this setting is to take a longer picture sequence and put it into the action classification model to obtain a more accurate detection result after obtaining a signal that a trigger action is about to occur.

[0079] Wherein, step S2053 also includes steps S20531-S20535:

[0080] Step 20531: Perform detection according to cameras set at different angles to obtain a depth image sequence;

[0081] Step 20532: Input the picture sequence with the first frame a and the length m into the first branch of the action classification model to perform multi-scale feature extraction, and obtain five first picture features, second picture features, third picture features, fourth picture features and fifth picture features in descending order of size;

[0082] Step 20533: Input the depth image sequence into the first branch of the action classification model to perform multi-scale feature extraction, and obtain a first depth feature, a second depth feature, a third depth feature, a fourth depth feature, and a fifth depth feature of five sizes from large to small respectively;

[0083] Step 20534: fuse the first image feature and the first depth feature to obtain a first fused feature; fuse the second image feature and the second depth feature to obtain a second fused feature; fuse the third image feature and the third depth feature to obtain a third fused feature; fuse the fourth image feature and the fourth depth feature to obtain a fourth fused feature; fuse the fifth image feature and the fifth depth feature to obtain a fifth fused feature;

[0084] Step 20535: multiply the depth features, image features and fusion features of the same scale by different weights respectively, input them into the softmax layer, and output the type of the trigger action.

[0085] It should be noted that when classifying the types of trigger actions, since relatively subtle gestures are used to control multiple operations, the differences between different gestures are relatively small. In order to obtain more accurate classification results, a depth image sequence is additionally obtained in the embodiment of the present application. The depth information is combined with the information in the image sequence to more accurately distinguish between relatively similar left and right hand features, different fingers on the same hand, etc.

[0086] S206, determining four position points in the naked eye 3D scene based on the pointing direction, and moving the target object based on the four position points and the gaze point;

[0087] Specifically, the four position points are respectively located on the straight lines where the four fingers are located;

[0088] Determine whether the gaze point and the four position points are located at the target object. If so, establish a spatial rectangular coordinate system O-xyz with the position of the target person as the origin, determine a first plane perpendicular to the xOy plane based on the moving positions of the four position points, and the four position points and the fingertips of the four fingers are located in the first plane; wherein, in the rectangular coordinate system O-xyz, the xOy plane is a plane parallel to the horizontal plane;

[0089] Displaying an intersection line between the first plane and the object in the naked eye 3D scene;

[0090] The position of the gaze point is detected in real time. When it is detected that the gaze point is located at an intersection, the target object is moved to the intersection, and the intersection is a point on the intersection line; wherein the intersection of the first plane and the object in the naked-eye 3D scene constitutes an intersection line, and when the gaze point is located at a certain intersection on the intersection line, the position of the intersection is determined as the position to which the object needs to be moved, and the object is moved to the position where the intersection is located.

[0091] When it is detected that the angle between the four fingers is greater than a preset angle threshold, the interaction process is terminated, wherein the preset angle threshold between the four fingers can be 15°. When the angle between the four fingers is greater than 15°, the above interaction process is terminated; the angle between the four fingers can be the angle between the straight lines where the four fingers are located.

[0092] In the naked-eye 3D scene, different operations can be performed on different objects, and each operation can correspond to different trigger actions. For example, the object can be a bucket, and there can be two operations for the bucket, one of which is to move the bucket, and the other is to pour the water in the bucket. The trigger action for moving the bucket can be to raise the arm and make a fist, and the trigger action for pouring the water in the bucket can be to raise the arm with the palm facing outward.

[0093] In one possible implementation, the training process of the action classification model in step S2053 includes: using a camera to capture action images and their corresponding depth images in different light and backgrounds; performing contrast enhancement operations on the action images and their corresponding depth images; using random flipping and matrix processing of the data set to expand the data set; and using the expanded data set to train the action classification model.

[0094] In a possible implementation, the method also includes: detecting the position of each body part of the target person through the camera; obtaining the position of the target air outlet of the naked-eye 3D scene, the target air outlet being any one of the following: a window, a door, or a cave entrance; determining the target plane where the target air outlet is located; projecting the position of each body part of the target person onto the target plane to obtain a projection area; determining the overlapping part of the projection area and the first area in the target plane; the first area is the area occupied by the target air outlet; determining the target body part of the target person based on the overlapping part; and starting the blowing device to blow air to the target device.

[0095] In a possible implementation, the blowing device is composed of invisible blowing holes, and the invisible blowing holes are evenly arranged on the surrounding walls, the ground and the ceiling.

[0096] In a possible implementation, before using the target object as a query condition to query and obtain at least one trigger action associated with the target object, the method further includes: demonstrating a preset type of scene through the naked-eye 3D scene; receiving an action entry instruction input by a user; acquiring the user's action through the camera, and storing the first action when it is detected that the user maintains the first action for a preset time; and using the first action as a trigger action for a corresponding type of scene.

[0097] In one possible implementation, the manned device can be associated with any of the following scenes: a car scene, a plane scene, or a ship scene; the 9D interactive system controls the manned device to move forward, backward, left, right, rotate, and vibrate through specific scene information in the naked-eye 3D scene to simulate the car scene, plane scene, or ship scene.

[0098] Based on the description of the generation method embodiment described in the above text, the embodiment of the present invention further discloses a 9D interactive system, which is applied to a 9D cinema. The 9D interactive system includes a camera, a projection device, a stereo audio device, a fragrance dissemination device, an environmental control device, a manned device, and a blowing device. The manned device includes a moving device, a rotating device, and a vibrating device. The 9D interactive system is used to perform the following method steps: a naked-eye 3D scene projected by the projection device; real-time tracking of the gaze point of the target person to determine the target object corresponding to the gaze point; using the target object as a query condition, querying to obtain at least one trigger action associated with the target object; acquiring video data of the target person in real time through the camera; performing action detection based on the video data, and under the condition that the trigger action is detected: acquiring the target The pointing direction of four fingers of a person; based on the pointing direction, four position points in the naked-eye 3D scene are determined, and the four position points are respectively located on the straight line where the four fingers are located; it is judged whether the gaze point and the four position points are located at the target object, and if so, a spatial rectangular coordinate system O-xyz is established with the position of the target person as the origin, and a first plane perpendicular to the xOy plane is determined based on the moving positions of the four position points, and the four position points and the fingertips of the four fingers are located in the first plane; the intersection line of the first plane and the object in the naked-eye 3D scene is displayed; the position of the gaze point is detected in real time, and when it is detected that the gaze point is located at the intersection point, the target object is moved to the intersection point, and the intersection point is a point on the intersection line; when it is detected that the angle between the four fingers is greater than the preset angle threshold, the interaction process is ended.

[0099] The embodiment of the present invention also provides a computer storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer storage medium provides a storage space, which stores the operating system of the electronic device. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0100] Figure 3 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The system includes: at least one processor 301, such as a central processing unit (CPU), at least one memory 302, and at least one bus 303.

[0101] The memory 302 may store program instructions, and the processor 301 may be used to call the program instructions to execute a 9D interaction method.

[0102] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically-erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), a solid state disk (SSD) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0103] It should be noted that all steps of the embodiments of the present application are performed under legal and compliant conditions, that is, all steps of the embodiments of the present application are performed with authorization.

[0104] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A 9D interaction method, characterized in that: A 9D interactive system used in a 9D cinema, the 9D interactive system comprising a camera, a projection device, a stereo audio device, a fragrance diffusion device, an environmental control device, a manned device, and a blowing device, the manned device comprising a moving device, a rotating device, and a vibrating device, the method comprising: Obtaining a naked-eye 3D scene through projection of the projection device; Track the gaze point of the target person in real time and determine the target object corresponding to the gaze point; Taking the target object as a query condition, querying and obtaining at least one trigger action associated with the target object; Acquire video data of the target person in real time through the camera; Performing action detection according to the video data, and upon detecting the triggering action: Obtaining the pointing directions of the four fingers of the target person, where the angle between the four fingers is less than a preset angle threshold; Determine four position points in the naked eye 3D scene based on the pointing direction, wherein the four position points are respectively located on a straight line where the four fingers are located; Determine whether the gaze point and the four position points are located at the target object. If so, establish a spatial rectangular coordinate system O-xyz with the position of the target person as the origin, determine a first plane perpendicular to the xOy plane based on the moving positions of the four position points, and the four position points and the fingertips of the four fingers are located in the first plane; Displaying an intersection line between the first plane and the object in the naked eye 3D scene; Detecting the position of the gaze point in real time, and when it is detected that the gaze point is located at an intersection point, moving the target object to the intersection point, wherein the intersection point is a point on the intersection line; When it is detected that the angles between the four fingers are greater than the preset angle threshold, the interaction process is terminated.

2. The 9D interaction method according to claim 1, characterized in that: The performing action detection according to the video data, under the condition that the trigger action is detected, includes: Decoding the video data, and extracting a picture sequence therefrom at intervals of a preset length, wherein the first frame of the picture sequence is a and the length is n; Input the picture sequence of length n into the lightweight action detection model, wherein the lightweight action detection model includes a first convolution module, a pooling layer, a second convolution module, a third convolution module, a fourth convolution module, and a fifth convolution module; the convolution kernel of the first convolution module is 3*5*5; the second convolution module, the third convolution module, the fourth convolution module, and the fifth convolution module each include a resnet residual block; When the lightweight action detection model detects a predetermined action, a picture sequence with a first frame a and a length of m is input into the action classification model for action detection, and the type of the final triggering action is output; wherein m is greater than n, and the picture sequence with a length of m includes a picture sequence with a length of n in which the predetermined action is detected; When the lightweight action detection model detects a predetermined action, a picture sequence with a first frame a and a length m is input into the action classification model for action detection, and the type of the final triggering action output includes: Perform detection based on cameras set at different angles to obtain depth image sequences; Input the picture sequence with the first frame a and the length m into the first branch of the action classification model to perform multi-scale feature extraction, and obtain the first picture feature, the second picture feature, the third picture feature, the fourth picture feature and the fifth picture feature from large to small sizes respectively; Inputting the depth image sequence into the first branch of the action classification model to perform multi-scale feature extraction, and obtaining a first depth feature, a second depth feature, a third depth feature, a fourth depth feature and a fifth depth feature in five sizes from large to small respectively; The first image feature and the first depth feature are fused to obtain a first fused feature; the second image feature and the second depth feature are fused to obtain a second fused feature; the third image feature and the third depth feature are fused to obtain a third fused feature; the fourth image feature and the fourth depth feature are fused to obtain a fourth fused feature; the fifth image feature and the fifth depth feature are fused to obtain a fifth fused feature; The depth features, image features and fusion features of the same scale are multiplied by different weights respectively, and input into the softmax layer to output the type of the trigger action.

3. The 9D interaction method according to claim 2, characterized in that: The training process of the action classification model includes: Use the camera to collect action images and their corresponding depth images under different light and background conditions; Performing contrast enhancement operation on the action image and its corresponding depth image; Use random flipping,matrix to process the dataset to augment the dataset; Use the augmented dataset to train an action classification model.

4. The 9D interaction method according to claim 1, characterized in that: The method further comprises: Detecting the positions of various parts of the target person's body by using the camera; Obtaining the position of a target air outlet of the naked eye 3D scene, wherein the target air outlet is any one of the following: a window, a door, or a cave entrance; Determine a target plane where the target tuyere is located; Projecting the positions of various parts of the target person's body onto the target plane to obtain a projection area; Determine an overlapping portion of the projection area and a first area in the target plane; the first area is an area occupied by the target tuyere; Determining a target body part of the target person based on the overlapping portion; The blowing device is started to blow air to the target device.

5. According to the 9D interaction method of claim 4, the blowing device is composed of invisible blowing holes, and the invisible blowing holes are evenly arranged in the surrounding walls, the ground and the ceiling.

6. The 9D interaction method according to claim 5, characterized in that: Before querying and obtaining at least one triggering action associated with the target object using the target object as a query condition, the method further includes: Demonstrating a preset type of scene through the naked eye 3D scene; Receive action entry instructions input by the user; Acquiring a user's action through the camera, and storing the first action when it is detected that the user maintains a first action for a preset time; The first action is used as a trigger action for a scene of the corresponding type.

7. The method according to claim 6, characterized in that The passenger device is associated with any of the following scenarios: a car-riding scenario, a plane-riding scenario, or a ship-riding scenario; The 9D interactive system controls the forward, backward, left, right, rotation and vibration of the manned device through the specific scene information in the naked-eye 3D scene to simulate the scene of riding in a car, a plane or a boat.

8. A 9D interactive system, characterized in that: include: The 9D interactive system is applied to a 9D cinema. The 9D interactive system includes a camera, a projection device, a stereo audio device, a fragrance diffusion device, an environmental control device, a manned device, and a blowing device. The manned device includes a moving device, a rotating device, and a vibrating device. The 9D interactive system is used to perform the following method steps: Obtaining a naked-eye 3D scene through projection of the projection device; Track the gaze point of the target person in real time and determine the target object corresponding to the gaze point; Taking the target object as a query condition, querying and obtaining at least one trigger action associated with the target object; Acquire video data of the target person through the camera; Performing action detection according to the video data, and upon detecting the triggering action: Obtain the pointing directions of the four fingers of the target person; Determine four position points in the naked eye 3D scene based on the pointing direction, wherein the four position points are respectively located on a straight line where the four fingers are located; Determine whether the gaze point and the four position points are located at the target object. If so, establish a spatial rectangular coordinate system O-xyz with the position of the target person as the origin, determine a first plane perpendicular to the xOy plane based on the moving positions of the four position points, and the four position points and the fingertips of the four fingers are located in the first plane; Displaying an intersection line between the first plane and the object in the naked eye 3D scene; Detecting the position of the gaze point in real time, and when it is detected that the gaze point is located at an intersection point, moving the target object to the intersection point, wherein the intersection point is a point on the intersection line; When it is detected that the angles between the four fingers are greater than a preset angle threshold, the interaction process is terminated.

9. An electronic device, characterized in that: include: a processor adapted to implement one or more instructions; and, A computer storage medium storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing a 9D interaction method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that: The computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing a 9D interaction method as described in any one of claims 1-7.

Images