A remote wireless virtual interaction method and system based on highly immersive AR technology

Abstract

The present invention relates to the field of image data processing, and in particular to a remote wireless virtual interaction method and system based on high immersion AR technology, the method comprising: obtaining a target image operated by a target user in a real space; determining the position change parameter of the current joint pixel according to the position information of the current joint pixel and other joint pixel points at different time points in the joint; classifying the joint pixel points according to the position information of each joint pixel at different time points in different target images of the target user, and obtaining multiple clusters; determining the importance of the current joint pixel points according to the standard deviation of the position change parameter and the position change parameter of the current joint pixel points; if the importance is greater than a threshold value, transmitting the position information of the current joint pixel points at different time points to the target device. In this way, the present invention improves the data transmission speed and reduces the delay between the virtual space and the real space.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a remote wireless virtual interaction method and system based on highly immersive AR technology. Background Art

[0002] Augmented Reality (AR) is a technology that superimposes virtual information onto the real world, creating the effect of virtual and real sharing the same space in front of the user. AR technology is not limited to the visual level, but can also include the enhancement of multiple sensory experiences such as hearing and touch. Its purpose is to allow users to sense the spatial and temporal connection and fusion of virtual and real spaces, so as to enhance users' perception and cognition of the real environment. By superimposing virtual chemical equipment onto the real environment through AR technology, students can simulate the operation of equipment in a real laboratory environment. AR technology can display the operation steps and processes in real time, guide students to correctly operate the equipment and experimental steps, reduce errors in actual operations, and help improve familiarity with the use of operating equipment and operating skills. At the same time, the chemical reaction process, material flow and changes can be intuitively displayed in the AR environment, so that students can intuitively understand complex chemical processes and principles.

[0003] In some scenarios, in chemical engineering teaching and training, a three-dimensional (3D) virtual simulation experiment platform is established to provide students with an immersive learning experience, which can make learning more vivid and interactive. However, in the process of multi-person interactive data transmission, due to the cumbersome and complex amount of data, data transmission is slow, resulting in a large delay between the virtual space and the actual space, and even unable to intuitively feel the chemical reaction phenomenon and experimental results, the user's experience of simulation experiments is poor. Summary of the invention

[0004] In order to solve the technical problem of large delay between virtual space and real space caused by slow data transmission, the purpose of the present invention is to provide a remote wireless virtual interaction method and system based on high immersion AR technology. The technical solution adopted is as follows:

[0005] In a first aspect, an embodiment of the present application provides a remote wireless virtual interaction method based on high-immersion AR technology, including: obtaining a target image of a target user operating in a real space, the target image including position information of joint pixel points of each joint of the target user's limbs at different time points; determining a position change parameter of a current joint pixel point according to the position information of a current joint pixel point and other joint pixel points in the joint at different time points; classifying the joint pixels according to the position information of each joint pixel point at different time points in different target images of the target user to obtain multiple clusters; determining the importance of the current joint pixel point according to the standard deviation of the position change parameters of the joint pixel points in the cluster where the current joint pixel point is located and the position change parameters of the current joint pixel point; if the importance is greater than a threshold, transmitting the position information of the current joint pixel points at different time points to the target device; if the importance is not greater than the threshold, caching the position information of the current joint pixel points at different time points locally.

[0006] Optionally, determining the position change parameters of the current joint pixel point based on the position information of the current joint pixel point and other joint pixel points in the joint at different time points includes: determining the position change amplitude of the current joint pixel point based on the position information of the current joint pixel point of the joint at different time points, the position change amplitude of the current joint pixel point characterizing the degree of movement change of the current joint pixel point; connecting the position information of the current joint pixel point at different time points into a broken line to obtain a target joint broken line, connecting the position information of other joint pixel points belonging to the same joint as the current joint pixel point at different time points into a broken line to obtain a reference joint broken line; determining the position change parameters of the current joint pixel point based on the similarity and position change amplitude between the target joint broken line and the reference joint broken line.

[0007] Optionally, determining the position change amplitude of the current joint pixel point based on the position information of the current joint pixel point of the joint at different time points includes: determining the Euclidean distance between the position information of the current joint pixel point of the joint at adjacent time points, and the position change angle of the current joint pixel point at each time point; determining the first product between the Euclidean distance and the position change angle, and the first difference between the number of time points and a preset value; determining a first superposition value of the first ratio of the absolute value of the first product and the first difference, and normalizing the first superposition value to obtain the position change amplitude of the current joint pixel point.

[0008] Optionally, determining the position change parameters of the current joint pixel point based on the similarity and position change amplitude between the target joint fold line and the reference joint fold line includes: calculating the DTW distance between the position information of the corresponding joint pixel points in the target joint fold line and the reference joint fold line; superimposing the DTW distances of each joint pixel point in the target joint fold line and the reference joint fold line to obtain a second superimposed value; performing negative correlation processing on the second superimposed value to obtain similarity; normalizing the similarity and multiplying it by the position change amplitude to obtain the position change parameters of the current joint pixel point.

[0009] Optionally, the joint pixels are classified according to the position information of each joint pixel point in different target images of the target users at different time points to obtain multiple clusters, including: determining the change angle of each joint pixel point in different target images of each target user according to the position information of each joint pixel point in different target images of all target users at different time points; and classifying the joint pixels according to the change angle of each joint pixel point to obtain multiple clusters.

[0010] Optionally, based on the position information of each joint pixel point in different target images of all target users at different time points, determining the change angle of each joint pixel point in different target images of each target user includes: connecting the position information of each joint pixel point in each target image at adjacent time points to obtain a connecting straight line, and determining the angle between the connecting straight line and the horizontal axis as the position change angle of each joint pixel point at the previous time point of the adjacent time; superimposing and averaging the position change angles of each previous time point in each target image to obtain the change angle of each joint pixel point.

[0011] Optionally, determining the importance of the current joint pixel point based on the standard deviation of the position change parameters of the joint pixels in the cluster where the current joint pixel point is located and the position change parameters of the current joint pixel point includes: calculating a second ratio of the number of joint pixels in the cluster to which the target user belongs to the number of target users; determining the maximum value from the position change parameters of each joint pixel point; and determining the product of the inverse of the sum of the standard deviation and a preset value, the maximum value, and the second ratio as the importance of the current joint pixel point.

[0012] Optionally, transmitting the position information of the current joint pixel points at different time points to the target device includes: dividing the position information of the current joint pixel points at different time points into multiple data blocks; and losslessly compressing each data block and transmitting it to the target device.

[0013] In the second aspect, an embodiment of the present application provides a remote wireless virtual interaction system based on high-immersion AR technology, including: an acquisition module, used to acquire a target image of a target user operating in a real space, the target image including position information of joint pixels of each joint of the target user's limbs at different time points; a determination module, used to determine the position change parameter of the current joint pixel point according to the position information of the current joint pixel point and other joint pixels in the joint at different time points; a classification module, used to classify the joint pixels according to the position information of the current joint pixel points at different time points in different target images of the target user to obtain multiple clusters; the determination module is also used to determine the importance of the current joint pixel point according to the standard deviation of the position change parameters of the joint pixels in the cluster where the current joint pixel point is located and the position change parameters of the current joint pixel point; a transmission module, used to transmit the position information of the current joint pixel points at different time points to the target device if the importance is greater than a threshold; a cache module, used to cache the position information of the current joint pixel points at different time points locally if the importance is not greater than a threshold.

[0014] In the third aspect, an embodiment of the present application provides a remote wireless virtual interaction system based on high-immersion AR technology, which includes: a processor and a memory; wherein the memory is used to store computer programs that can be run on the processor; the processor is used to execute the program stored in the memory to implement the steps of the remote wireless virtual interaction method based on high-immersion AR technology as mentioned in the first aspect.

[0015] The present invention has the following beneficial effects: firstly, a target image in which a target user performs an operation in an actual space is obtained, wherein the target image includes position information of joint pixels of each joint of a limb of the target user at different time points; then, according to the position information of the current joint pixel in the joint and other joint pixels at different time points, a position change parameter of the current joint pixel is determined; then, according to the position information of the current joint pixel at different time points in different target images of the target user, the joint pixels are classified to obtain a plurality of clusters; finally, according to the standard deviation of the position change parameter of the joint pixel in the cluster where the current joint pixel is located and the position change parameter of the current joint pixel, the importance of the current joint pixel is determined; if the importance is greater than a threshold value, the position information of the current joint pixel at different time points is transmitted to a target device; if the importance is not greater than a threshold value, the position information of the current joint pixel at different time points is cached locally.

[0016] In this way, the embodiment of the present invention determines the importance of joint pixels through the position information of each joint pixel in the target image of the target user at different time points, and transmits the position information of joint pixels whose importance is greater than the threshold to the target device, while the position information whose importance is lower than the threshold is directly cached locally instead of being transmitted to the target device. The amount of data transmitted to the target device is reduced, the data transmission speed is increased, the delay between the virtual space and the actual space is reduced, and the user's experience of simulation experiments is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of a remote wireless virtual interaction method based on highly immersive AR technology provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of different joints of a human body provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of the module composition of a remote wireless virtual interaction system based on highly immersive AR technology provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the structure of a remote wireless virtual interaction system based on highly immersive AR technology provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a remote wireless virtual interaction method and system based on high immersion AR technology proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0023] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0024] The following is a detailed description of a remote wireless virtual interaction method and system based on highly immersive AR technology provided by the present invention in conjunction with the accompanying drawings.

[0025] Embodiment 1:

[0026] See also Figure 1 , which shows a flow chart of a remote wireless virtual interaction method based on high immersion AR technology provided by an embodiment of the present invention, including:

[0027] S101, obtaining a target image of a target user performing an operation in an actual space.

[0028] The target image includes position information of joint pixel points of each joint of the target user's limbs at different time points.

[0029] Specifically, the actual space in the embodiment of the present invention can be an empty classroom. Multiple infrared motion capture devices and operating devices are installed in the classroom, which can record the user's limb motion data and the user's environmental information in real time. In addition, several AR devices are also set up in the classroom to present a virtual laboratory to the user. After the infrared motion capture device captures the user's limb motion data, it is uploaded to the data center. The data center maps the motion data in the virtual laboratory environment. The user experiences the virtual laboratory scene by wearing an AR device, and controls the virtual laboratory environment through the operating device worn, forming corresponding experimental changes, and then feeding back the corresponding changes to the AR device, thereby achieving the purpose of simulated operation.

[0030] Furthermore, the target image may be the motion data of each joint of the user's limbs during the movement process, which includes the position information of the joint pixel points in each joint of the user's limbs at different time points. The target image may be a frame of pictures, each of which includes the position information of the joint pixel points in each joint of the user's limbs at a certain time point. Figure 2 As shown, according to the functions of joints in different parts of the human body and the movement characteristics during exercise, the human body is divided into the following types of joints. In the target image, at least one joint image of the human body is included, and each joint image includes multiple joint pixels. The joint pixels belonging to the same joint belong to the same category of joint pixels.

[0031] Furthermore, the position information may be the position coordinates of the joint pixel points in each joint in the image. When determining the position coordinates, a coordinate system is established with the first pixel point from bottom to top in the target image as the coordinate origin, and a marker point optical motion capture system is used to record the motion changes of several joints of the user, and the position coordinates of each joint when the motions of several joints of the user change. The position coordinates of each joint recorded are formed into a data sequence in the order of collection time, which is recorded as the position time sequence of the joint.

[0032] S102, determining a position change parameter of a current joint pixel point according to position information of a current joint pixel point and other joint pixel points in the joint at different time points.

[0033] Specifically, the position change parameter can reflect the degree of change of the position information of the joint pixel points. When the user remains still, the position information in the collected joint position sequence is the same, or fluctuates within a small range. When the user's movements change, the position information in the collected joint position sequence changes significantly, and the degree of change is much greater than that in the static state. Therefore, the degree of change of the joint movement is determined by first calculating the change amplitude of the position information in the position sequence of the same joint. Furthermore, when the user's movements change, it is not a single joint that changes, but a chain reaction, which is often accompanied by changes in joint pixel points of the same category or other joint pixel points connected to the joint pixel points. Therefore, by determining the similarity of the position sequence of each joint with the changes of other joint pixel points in the same category, combined with the change amplitude of the joint, the position change parameter of the joint pixel point is obtained.

[0034] Furthermore, when determining the position change parameters of the current joint pixel points, firstly, the position change amplitude of the current joint pixel points is determined according to the position information of the current joint pixel points of the joint at different time points, and the position change amplitude of the current joint pixel points represents the degree of movement change of the current joint pixel points; then, the position information of the current joint pixel points at different time points is connected into a broken line to obtain a target joint broken line, and the position information of other joint pixel points belonging to the same joint as the current joint pixel point at different time points is connected into a broken line to obtain a reference joint broken line; finally, the position change parameters of the current joint pixel points are determined according to the similarity and position change amplitude between the target joint broken line and the reference joint broken line.

[0035] Among them, when determining the position change amplitude of the current joint pixel point, first determine the Euclidean distance between the position information of the current joint pixel point of the joint at adjacent time points, and the position change angle of the current joint pixel point at each time point; then determine the first product between the Euclidean distance and the position change angle, and the first difference between the number of time points and the preset value; then determine the first superposition value of the first ratio of the absolute value of the first product and the first difference, and normalize the first superposition value to obtain the current position change amplitude of the joint pixel point.

[0036] Specifically, the position change angle of the current joint pixel point at each time point is determined as follows: on the target image of the same motion capture camera, the position coordinates of the i-th joint pixel point at the k-th time are connected with the position coordinates at the k+1-th time, and the angle between the connecting straight line and the direction parallel to the x-axis is recorded as the position change angle of the joint pixel point at the k-th time. Further, the calculation method of the Euclidean distance can refer to the prior art, and the embodiments of the present invention will not be repeated here.

[0037] Furthermore, the preset value can be set to 1, and the current position change amplitude of the joint pixel point can be expressed by the following formula:

[0038]

[0039] In the above formula, U i,f Indicates the position change amplitude of the position information of the i-th joint pixel at different time points in the f-th target image. f,i,k Represents the Euclidean distance between the position information of the i-th joint pixel point at the k-th time point and the position information of the k+1-th time point on the f-th target image. f,i,k It represents the position change angle of the i-th joint pixel at the k-th moment in the f-th target image. n represents the number of time points in a target image.

[0040] Furthermore, the larger the Euclidean distance between the position information of the joint pixels, the larger the position change angle, which means that the position change amplitude of the same joint pixel at different time points is larger, that is, the more obvious the position difference of the same joint pixel at different time points, the greater the degree of movement change of the joint pixel.

[0041] Furthermore, a spatial three-dimensional coordinate system is established with the shooting time of the target image as the horizontal coordinate and the position information of the joint pixel points as the vertical coordinate. The position information of the current joint pixel points in the target image shot at several historical moments is marked in the coordinates, and the position information of the current joint pixel points is connected to form a polyline, which is recorded as the target joint polyline. Similarly, the polylines of all other joint pixel points belonging to the same category as the current joint pixel point are obtained and recorded as reference joint polylines.

[0042] Furthermore, when determining the position change parameters of the current joint pixel points, as an optional embodiment of the present invention, the DTW distance between the position information of the corresponding joint pixel points in the target joint fold line and the reference joint fold line is first calculated; then the DTW distances of each joint pixel point in the target joint fold line and the reference joint fold line are superimposed to obtain a second superimposed value; the second superimposed value is then negatively correlated to obtain similarity; finally, the similarity is normalized and multiplied by the position change amplitude to obtain the position change parameters of the current joint pixel points.

[0043] Specifically, the calculation method of the DTW distance can refer to the prior art, and the embodiment of the present invention will not be repeated here. The similarity can be calculated using the following formula:

[0044]

[0045] In the above formula, Y f,i Y represents the similarity between the i-th joint pixel and other joint pixels in the same category in the target image captured by the f-th motion capture camera. k Indicates the position information of the kth joint pixel point in the target joint polyline. x,k Indicates the position information of the kth joint pixel point in the xth reference joint polyline. DTW(Y k ,Y x,k ) represents the DTW distance between the target joint polyline and the reference joint polyline. m represents the number of joint pixels in the same category. exp is the natural exponential function, which is used to Negative correlation processing.

[0046] Furthermore, the DTW distance is used to express the similarity between joint pixels of the same category. The smaller the difference in position changes between different joint pixels, the smaller the calculated DTW distance is, which means that the changes in joint pixels of the same category are more consistent during the movement change.

[0047] Furthermore, the position change parameter of the current joint pixel point can be calculated using the following formula:

[0048] U ′ i,f =norm(Y f,i )×U i,f

[0049] In the above formula, U ′ i,f Y represents the position change parameter of the i-th joint pixel point in the target image of the f-th motion capture camera. f,iU represents the similarity between the i-th joint pixel and other joint pixels in the same category in the target image captured by the f-th motion capture camera. i,f represents the position change amplitude of the position information of the i-th joint pixel at different time points in the f-th target image. norm represents the normalization function, which is used to f,i Normalize.

[0050] Among them, the greater the amplitude of the position change of the joint pixel points, the greater the similarity of the joint pixels of the same category, which means that the movement change of the joint pixel points is more obvious. The position information of the captured joint pixels at different time points needs to be transmitted emphatically.

[0051] S103, classifying the joint pixels according to the position information of the current joint pixels in different target images of the target user at different time points to obtain a plurality of clusters.

[0052] Specifically, during actual teaching, multiple students in the classroom simulate experimental operations at the same time. According to the regularity of the changes in the movements of different students, a clustering algorithm is used to classify them, obtain the movement change information belonging to the experimental operation, and then obtain the important parameters of each joint pixel point.

[0053] Furthermore, when classifying joint pixel points, first, according to the position information of each joint pixel point in different target images of all target users at different time points, the change angle of each joint pixel point in different target images of each target user is determined; then, the joint pixel points are classified according to the change angle of each joint pixel point to obtain multiple clusters.

[0054] Specifically, the change angle of the current joint pixel point refers to the overall change angle of the current joint pixel point in the time period at different time points. When determining the change angle of the current joint pixel point, firstly, the position information of each joint pixel point in each target image at adjacent time points is connected to obtain a connecting straight line, and the angle between the connecting straight line and the horizontal axis is determined as the position change angle of each joint pixel point at the previous time point at the adjacent time; then, the position change angles of each previous time point in each target image are superimposed and averaged to obtain the change angle of each joint pixel point.

[0055] Among them, the position change angle of the joint pixel point at each moment is determined in the same way as the position change angle in the above embodiment, that is, the determination method is as follows: on the target image of the same motion capture camera, the position coordinates of the i-th joint pixel point at the k-th moment are connected with the position coordinates at the k+1-th moment, and the angle between the connecting straight line and the direction parallel to the x-axis is recorded as the position change angle of the joint pixel point at the k-th moment.

[0056] Furthermore, the change angle of each joint pixel point can be calculated using the following formula:

[0057]

[0058] In the above formula, δ p,i θ represents the change angle of the pixel point of the i-th joint of the p-th user in the target image captured by all motion capture cameras. p,f,i It represents the position change angle of the pixel point of the i-th joint of the p-th user in the target image captured by the f-th motion capture camera at the previous moment. N represents the number of motion capture cameras set, that is, the number of target images obtained at a certain moment.

[0059] Furthermore, when classifying the pixels of each joint, the K-means clustering algorithm can be used to classify the user's joint pixels according to the change angle, that is, to classify the user's action changes. Since the degree of change of the user's action is consistent when performing the same experimental operation, the change angle of the same joint pixel is also similar, but some users have action changes that are irrelevant to the experimental operation, and the number of users with consistent action changes is much greater than that of users with irrelevant experimental operations. Therefore, two cluster centers are set, and the change angle of the user's i-th joint pixel point is used as the cluster distance, and it is divided into the class corresponding to the cluster center with the smallest distance. For each cluster center k, the center of the class is recalculated. Repeat the above process until all classes no longer change. At this point, two clusters are obtained for the change angle of the same joint pixel point. The change angle of the pixel joint point in the cluster with a larger amount of data is the change angle that affects the experimental operation, and the user's action changes need to be transmitted intensively.

[0060] S104, determining the importance of the current joint pixel point according to the standard deviation of the position change parameter of the joint pixel points in the cluster where the current joint pixel point is located and the position change parameter of the current joint pixel point.

[0061] Specifically, the position change parameters of each joint pixel in the cluster where the current joint pixel is located are weighted to determine the importance of the current joint pixel. Then, the captured joint pixel position information is classified and transmitted according to the importance, which not only speeds up data transmission but also helps to accurately reconstruct real actions in virtual space and improve the accuracy of reproduced actions.

[0062] Furthermore, when determining the importance of the current joint pixel point, as an optional embodiment of the present invention, firstly, the second ratio of the number of joint pixels in the cluster to which the target user belongs to the number of target users is calculated; then the maximum value is determined from the position change parameters of each joint pixel point; finally, the product of the reciprocal of the sum of the standard deviation and the preset value, the maximum value and the second ratio is determined as the importance of the current joint pixel point.

[0063] Specifically, the preset value can be selected according to the actual situation. In the embodiment of the present invention, the value is 1. The importance of the current joint pixel point can be expressed by the following formula:

[0064]

[0065] In the above formula, I i Indicates the importance of the i-th joint pixel. N 簇 Represents the number of joint pixels in the cluster to which the p-th user belongs. N 总 Indicates the number of all target users. U ′ i,f represents the position change parameter of the pixel point of the i-th joint on the target image of the f-th motion capture camera. N represents the number of motion capture cameras set, that is, the number of target images obtained at a certain moment. i Represents the standard deviation of the position change parameter of the i-th joint pixel point in the cluster to which the p-th user belongs.

[0066] Furthermore, in the target images captured by the motion capture cameras at different angles, the degree of change of the user's motion is not exactly the same, and due to the existence of limb occlusion and other situations, there is no change in the joint pixel points caused by the motion change in the captured target images. Therefore, the maximum value of the joint pixel point position change parameter caused by the user's motion change in the target images captured by the motion capture cameras is selected. The larger the value of the position change parameter, the greater the amplitude of the motion change of the joint pixel point, and the joint pixel point needs to be paid special attention to. It means that the position change parameters of each joint pixel in the cluster are weighted. When the standard deviation of the cluster is smaller, it means that the dispersion of the position change parameters in the cluster is smaller, and the movement change amplitude of the joint pixel is more uniform, then the joint pixel needs to be paid special attention.

[0067] S105, if the importance is greater than the threshold, the position information of the current joint pixel points at different time points is transmitted to the target device; if the importance is not greater than the threshold, the position information of the current joint pixel points at different time points is cached locally.

[0068] Specifically, the threshold value can be determined according to the actual situation, and the threshold value in the embodiment of the present invention is 0.6. If the importance is greater than 0.6, it means that the current joint pixel point needs to be focused on, and the position information of the current joint pixel point at different time points can be transmitted to the target device. When transmitting the position information to the target device, the position information of the current joint pixel point at different time points can be divided into multiple data blocks; then each data block is losslessly compressed and transmitted to the target device. Among them, the lossless compression method can adopt the DEFLATE compression algorithm. The target device can be an AR device, and the process of transmitting the data block to the target device after lossless compression is as follows: using the transmission control protocol (Transmission Control Protocol, TCP) to transmit the compressed data to the data processing platform, and then the data processing platform uses the fifth generation mobile communication technology (Fifth Generation Mobile Networks, 5G) communication signal to transmit to the virtual space, forming corresponding experimental changes, and feeding back to the AR device worn by the user to complete the virtual experiment. Among them, the AR device can be AR glasses, etc. If the importance is not greater than the threshold, the position information of the current joint pixel point at different time points is cached locally. Or directly delete the position information of the current joint pixel point at different time points.

[0069] The embodiment of the present invention determines the importance of joint pixels through the position information of each joint pixel in the target image of the target user at different time points, and transmits the position information of joint pixels whose importance is greater than a threshold to the target device, while the position information whose importance is lower than the threshold is directly cached locally instead of being transmitted to the target device. The amount of data transmitted to the target device is reduced, the data transmission speed is increased, the delay between the virtual space and the real space is reduced, and the user's experience of simulation experiments is improved.

[0070] Embodiment 2:

[0071] Corresponding to the remote wireless virtual interaction method based on the highly immersive AR technology provided in the above embodiment, based on the same technical concept, the embodiment of the present application also provides a remote wireless virtual interaction system based on the highly immersive AR technology. Figure 3 A schematic diagram of the module composition of a remote wireless virtual interaction system based on highly immersive AR technology provided in an embodiment of the present application. The remote wireless virtual interaction system based on highly immersive AR technology is used to perform Figure 1 The remote wireless virtual interaction method based on highly immersive AR technology described in the paper is as follows: Figure 3As shown, the remote wireless virtual interaction system 300 based on high immersion AR technology includes: an acquisition module 301, which is used to acquire a target image in which a target user operates in a real space, and the target image includes the position information of the joint pixels of each joint of the target user's limbs at different time points; a determination module 302, which is used to determine the position change parameter of the current joint pixel point according to the position information of the current joint pixel point and other joint pixels in the joint at different time points; a classification module 303, which is used to classify the joint pixels according to the position information of the current joint pixel points at different time points in different target images of the target user to obtain multiple clusters; the determination module 302 is also used to determine the importance of the current joint pixel point according to the standard deviation of the position change parameter of the joint pixel point in the cluster where the current joint pixel point is located and the position change parameter of the current joint pixel point; a transmission module 304, which is used to transmit the position information of the current joint pixel point at different time points to the target device if the importance is greater than a threshold value; and a cache module 305, which is used to cache the position information of the current joint pixel point at different time points locally if the importance is not greater than a threshold value.

[0072] The embodiment of the present invention determines the importance of joint pixels through the position information of each joint pixel in the target image of the target user at different time points, and transmits the position information of joint pixels whose importance is greater than a threshold to the target device, while the position information whose importance is lower than the threshold is directly cached locally instead of being transmitted to the target device. The amount of data transmitted to the target device is reduced, the data transmission speed is increased, the delay between the virtual space and the real space is reduced, and the user's experience of simulation experiments is improved.

[0073] Optionally, the determination module 302 is also used to determine the position change amplitude of the current joint pixel point based on the position information of the current joint pixel point of the joint at different time points, the position change amplitude of the joint pixel point characterizing the degree of movement change of the current joint pixel point; connecting the position information of the current joint pixel point at different time points into a broken line to obtain a target joint broken line, connecting the position information of other joint pixel points belonging to the same joint as the current joint pixel point at different time points into a broken line to obtain a reference joint broken line; determining the position change parameters of the current joint pixel point based on the similarity and position change amplitude between the target joint broken line and the reference joint broken line.

[0074] Optionally, the determination module 302 is also used to determine the Euclidean distance between the position information of the current joint pixel points of the joint at adjacent time points, and the position change angle of the current joint pixel points at each time point; determine the first product between the Euclidean distance and the position change angle, and the first difference between the number of time points and a preset value; determine the first superposition value of the first ratio of the absolute value of the first product and the first difference, and normalize the first superposition value to obtain the position change amplitude of the current joint pixel point.

[0075] Optionally, the determination module 302 is also used to calculate the DTW distance between the position information of the corresponding joint pixel points in the target joint fold line and the reference joint fold line; superimpose the DTW distances of each joint pixel point in the target joint fold line and the reference joint fold line to obtain a second superposition value; perform negative correlation processing on the second superposition value to obtain similarity; normalize the similarity and multiply it by the position change amplitude to obtain the position change parameter of the current joint pixel point.

[0076] Optionally, the classification module 303 is also used to determine the change angle of each joint pixel point in the different target images of each target user according to the position information of each joint pixel point in the different target images of all target users at different time points; and classify the joint pixel points according to the change angle of each joint pixel point to obtain multiple clusters.

[0077] Optionally, the classification module 303 is also used to connect the position information of each joint pixel point in each target image at adjacent moments to obtain a connecting straight line, and determine the angle between the connecting straight line and the horizontal axis as the position change angle of each joint pixel point at the previous moment in the adjacent moment; the position change angles of each previous moment in each target image are superimposed and averaged to obtain the change angle of each joint pixel point.

[0078] Optionally, the classification module 303 is also used to calculate a second ratio of the number of joint pixels in the cluster to which the target user belongs to the number of target users; determine the maximum value from the position change parameters of each joint pixel; and determine the product of the reciprocal of the sum of the standard deviation and a preset value, the maximum value, and the second ratio as the importance of the current joint pixel.

[0079] Optionally, the transmission module 304 is further used to divide the position information of the current joint pixel point at different time points into multiple data blocks; and transmit each data block to the target device after lossless compression.

[0080] The remote wireless virtual interaction system based on high immersion AR technology provided in the embodiment of the present application can implement the various processes in the embodiments corresponding to the above-mentioned remote wireless virtual interaction method based on high immersion AR technology, and has the same or similar beneficial effects. To avoid repetition, it will not be repeated here.

[0081] Embodiment three:

[0082] Corresponding to the remote wireless virtual interaction method based on the highly immersive AR technology provided in the above embodiment, based on the same technical concept, the embodiment of the present application also provides a remote wireless virtual interaction system based on the highly immersive AR technology, which is used to execute the remote wireless virtual interaction method based on the highly immersive AR technology. Figure 4 A schematic diagram of a remote wireless virtual interaction system based on highly immersive AR technology is provided to implement various embodiments of the present application. Figure 4 The remote wireless virtual interaction system based on high immersion AR technology may have relatively large differences due to different configurations or performances, and may include one or more processors 401 and memory 402, the memory 402 is used to store computer programs that can be run on the processor 401, and the processor 401 is used to execute the program stored in the memory 402 to achieve the above Figure 1 The memory 402 may be a short-term storage or a persistent storage. The application stored in the memory 402 may include one or more modules (not shown in the figure), each of which may include a series of computer executable instructions in a remote wireless virtual interaction system based on high immersion AR technology.

[0083] Furthermore, the processor 401 can be configured to communicate with the memory 402 to execute a series of computer executable instructions in the memory 402 on the remote wireless virtual interaction system based on the high immersion AR technology. The remote wireless virtual interaction system based on the high immersion AR technology can also include one or more power supplies 403, one or more wired or wireless network interfaces 404, one or more input and output interfaces 405, and one or more keyboards 406.

[0084] Specifically in this embodiment, the remote wireless virtual interaction system based on high immersion AR technology includes a processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other through the bus; the memory is used to store computer programs; the processor is used to execute the programs stored in the memory to achieve the above Figure 1 The various steps in the method embodiment are similar to those in the method embodiment, and have the beneficial effects of the above method embodiments. To avoid repetition, the embodiments of the present application will not be described again here.

[0085] It should be noted that the remote wireless virtual interaction system based on high immersion AR technology provided in the embodiment of the present application and the remote wireless virtual interaction method based on high immersion AR technology provided in the embodiment of the present application are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned remote wireless virtual interaction method based on high immersion AR technology, and has the same or similar beneficial effects, and the repetitions will not be repeated.

[0086] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A remote wireless virtual interaction method based on highly immersive AR technology, characterized in that: The remote wireless virtual interaction method based on high immersion AR technology includes: Acquire a target image of a target user operating in a real space, wherein the target image includes position information of joint pixel points of each joint of a limb of the target user at different time points; Determine the position change parameter of the current joint pixel point according to the position information of the current joint pixel point and other joint pixel points in the joint at different time points; Classifying the joint pixels according to the position information of each joint pixel in different target images of the target user at different time points to obtain a plurality of clusters; Determining the importance of the current joint pixel point according to the standard deviation of the position change parameter of the joint pixel point in the cluster where the current joint pixel point is located and the position change parameter of the current joint pixel point; If the importance is greater than a threshold, the position information of the current joint pixel point at different time points is transmitted to the target device; If the importance is not greater than the threshold, the position information of the current joint pixel point at different time points is cached locally; Determining the position change parameter of the current joint pixel point according to the position information of the current joint pixel point and other joint pixel points at different time points in the joint includes: Determine the position change amplitude of the current joint pixel point according to the position information of the current joint pixel point of the joint at different time points, wherein the position change amplitude of the current joint pixel point represents the degree of movement change of the current joint pixel point; Connecting the position information of the current joint pixel points at different time points into a broken line to obtain a target joint broken line, and connecting the position information of other joint pixel points belonging to the same joint as the current joint pixel point at different time points into a broken line to obtain a reference joint broken line; Determining a position change parameter of the current joint pixel point according to the similarity between the target joint fold line and the reference joint fold line and the position change amplitude; Determining the position change parameter of the current joint pixel point according to the similarity between the target joint fold line and the reference joint fold line and the position change amplitude includes: Calculating the DTW distance between the position information of the corresponding joint pixel points in the target joint fold line and the reference joint fold line; Superimposing the DTW distances of the joint pixels in the target joint fold line and the reference joint fold line to obtain a second superposition value; Performing negative correlation processing on the second superposition value to obtain the similarity; Normalizing the similarity and multiplying it by the position change amplitude to obtain the position change parameter of the current joint pixel point; Determining the importance of the current joint pixel point according to the standard deviation of the position change parameter of the joint pixel point in the cluster where the current joint pixel point is located and the position change parameter of the current joint pixel point comprises: Calculating a second ratio of the number of joint pixel points in the cluster to which the target user belongs to the number of the target users; Determine a maximum value from the position change parameters of each of the joint pixel points; The product of the reciprocal of the sum of the standard deviation and the preset value, the maximum value and the second ratio is determined as the importance of the current joint pixel point.

2. The remote wireless virtual interaction method based on high immersion AR technology according to claim 1 is characterized in that: Determining the position change amplitude of the current joint pixel point according to the position information of the current joint pixel point of the joint at different time points includes: Determine the Euclidean distance between the position information of the current joint pixel points of the joint at adjacent time points, and the position change angle of the current joint pixel points at each of the time points; Determine a first product of the Euclidean distance and the position change angle, and a first difference between the number of the time points and a preset value; A first superposition value of a first ratio of the absolute value of the first product to the first difference is determined, and the first superposition value is normalized to obtain a position change amplitude of the current joint pixel point.

3. The remote wireless virtual interaction method based on high immersion AR technology according to claim 1 is characterized in that: The step of classifying the joint pixels according to the position information of each joint pixel in different target images of the target user at different time points to obtain a plurality of clusters includes: Determine the change angle of each joint pixel point in the different target images of each target user according to the position information of each joint pixel point in the different target images of all target users at different time points; The joint pixel points are classified according to the change angle of each joint pixel point to obtain a plurality of clusters.

4. The remote wireless virtual interaction method based on high immersion AR technology according to claim 3 is characterized in that: Determining the change angle of each joint pixel point in different target images of each target user according to the position information of each joint pixel point in different target images of all target users at different time points includes: Connecting the position information of each joint pixel point in each target image at adjacent moments to obtain a connecting straight line, and determining the angle between the connecting straight line and the horizontal axis as the position change angle of each joint pixel point at a previous moment of the adjacent moments; The position change angles at each previous time point in each target image are superimposed and averaged to obtain the change angle of each joint pixel point.

5. The remote wireless virtual interaction method based on high immersion AR technology according to claim 1 is characterized in that: The transmitting the position information of the current joint pixel points at different time points to the target device comprises: Dividing the position information of the current joint pixel points at different time points into a plurality of data blocks; Each of the data blocks is losslessly compressed and then transmitted to the target device.

6. A remote wireless virtual interaction system based on highly immersive AR technology, characterized in that: include: An acquisition module, used to acquire a target image of a target user operating in a real space, wherein the target image includes position information of joint pixel points of each joint of a limb of the target user at different time points; A determination module, for determining the position change parameter of the current joint pixel point according to the position information of the current joint pixel point and other joint pixel points at different time points in the joint, comprising: Determine the position change amplitude of the current joint pixel point according to the position information of the current joint pixel point at different time points, the position change amplitude of the current joint pixel point characterizes the degree of movement change of the current joint pixel point; connect the position information of the current joint pixel point at different time points into a broken line to obtain a target joint broken line, connect the position information of other joint pixel points belonging to the same joint as the current joint pixel point at different time points into a broken line to obtain a reference joint broken line; determine the position change parameter of the current joint pixel point according to the similarity between the target joint broken line and the reference joint broken line and the position change amplitude; Determining the position change parameter of the current joint pixel point according to the similarity between the target joint fold line and the reference joint fold line and the position change amplitude includes: Calculating the DTW distance between the position information of the corresponding joint pixel points in the target joint fold line and the reference joint fold line; superimposing the DTW distances of the joint pixel points in the target joint fold line and the reference joint fold line to obtain a second superimposed value; performing negative correlation processing on the second superimposed value to obtain the similarity; normalizing the similarity and multiplying it by the position change amplitude to obtain the position change parameter of the current joint pixel point; A classification module, used for classifying the joint pixels according to the position information of the current joint pixels at different time points in different target images of the target user to obtain multiple clusters; The determination module is further used to determine the importance of the current joint pixel point according to the standard deviation of the position change parameter of the joint pixel point in the cluster where the current joint pixel point is located and the position change parameter of the current joint pixel point, including: Calculate a second ratio of the number of joint pixels in the cluster to which the target user belongs to the number of the target users; determine a maximum value from the position change parameters of each of the joint pixels; determine the product of the reciprocal of the sum of the standard deviation and a preset value, the maximum value, and the second ratio as the importance of the current joint pixel; A transmission module, used for transmitting the position information of the current joint pixel point at different time points to a target device if the importance is greater than a threshold; A cache module is used to cache the position information of the current joint pixel point at different time points locally if the importance is not greater than the threshold.

7. A remote wireless virtual interactive system based on highly immersive AR technology, characterized in that: The remote wireless virtual interaction system based on high immersion AR technology includes: a processor and a memory; wherein the memory is used to store computer programs that can be run on the processor; the processor is used to execute the program stored in the memory to implement the steps of the remote wireless virtual interaction method based on high immersion AR technology as described in any one of claims 1-5.

Images