A method and system for interactive AR red envelope

Abstract

The present invention relates to the field of social software methods, and in particular to a method and system for interactive AR red envelopes. It is characterized in that: the physical scene includes at least a first client id1 mobile phone and a second client id2 mobile phone, the second client id2 mobile phone initiates a red envelope request, the client id2 mobile phone obtains the spatial orientation parameter through its sensor and sets the spatial orientation parameter of the red envelope image to be sent, that is, the setting parameter, and uploads the setting parameter to the server, the server generates an animated red envelope image with the spatial orientation parameter according to the setting parameter and publishes the animated red envelope image; the visual effect of the person sending the red envelope holding the red envelope (red envelope logo image or red envelope image) at the end of his arm and throwing the red envelope image through a parabolic action is achieved.

Description

Technical Field

[0001] The present invention relates to the field of social software methods, and in particular to a method and system for interactive AR red envelopes. Background Art

[0002] Augmented Reality (AR) is a technology that calculates the position and angle of camera images in real time and adds corresponding images, videos, and 3D models. The goal of this technology is to overlay the virtual world on the real world on the screen and interact with it.

[0003] Currently, social software such as instant messaging QQ, WeChat, etc. have enabled the function of sending red envelopes as payment tools. In particular, the red envelope grabbing function provides fun for users.

[0004] Chinese patent application 2016105984877 An AR method and system discloses a method and system for AR red envelopes based on geographic location, but the red envelope initiator of this method cannot interact with the red envelope recipient.

[0005] The image recognition based on the present invention includes existing mature feature recognition, object contour recognition, color recognition, action recognition, face recognition, text recognition, etc., and adopts corresponding recognition strategies in different environments. In particular, the physical position of the mobile phone is indirectly determined and tracked through the recognition and tracking of the human arm and the end hand. Summary of the invention

[0006] The purpose of the present invention is to provide an AR red envelope method and system for overlaying a virtual world on the real world on a screen for interaction. The system can be based on an independent instant messaging IM platform, or through a third-party service (API), or embedded in an existing IM platform, such as QQ, WeChat, Momo, etc.

[0007] The mobile phone corresponding to the client id is recorded as client id mobile phone.

[0008] The first client is recorded as client id1, the second client is recorded as client id2, and so on.

[0009] s=∫∫a*(dt) 2 is the spatial motion trajectory of the mobile phone, which is obtained by quadratically integrating the acceleration vector a with respect to time t.

[0010] v=∫a*dt is the spatial motion speed of the mobile phone, which is obtained by integrating the acceleration vector a over time t.

[0011] The mobile phone motion parameters include motion trajectory s, motion speed v (instant speed), and acceleration a.

[0012] S is the trajectory of the image of the mobile phone (or replaced by a certain end of the arm) in the video over time, that is, the image pixel motion trajectory, that is, the image motion trajectory.

[0013] V=S′ is the displacement speed of the image of the mobile phone (or replaced by a certain end of the arm) in the video over time, that is, the image pixel speed, that is, the image speed (image movement speed).

[0014] A=S′′ is the acceleration of the displacement of the image of the mobile phone (or replaced by a certain end of the arm) in the video over time, that is, the image pixel acceleration, that is, the image acceleration (image motion acceleration).

[0015] The mobile phone motion parameters include image pixel motion trajectory S, image pixel velocity V, and image pixel acceleration A.

[0016] The technical solution of the present invention is:

[0017] An interactive AR red envelope method, comprising a client and a server, characterized in that:

[0018] The physical scene includes at least a first client id1 mobile phone and a second client id2 mobile phone. The second client id2 mobile phone initiates a red envelope request. The client id2 mobile phone obtains the spatial orientation parameters through its sensor and sets the spatial orientation parameters of the red envelope image to be sent, i.e., setting parameters, and uploads the setting parameters to the server. The server generates an animated red envelope image with the spatial orientation parameters according to the setting parameters and publishes the animated red envelope image.

[0019] After the client id1 mobile phone uses its camera to capture the scene containing the second client id2 mobile phone, it determines the end-of-arm motion of the client id2 mobile phone holder through recognition software and superimposes the red envelope logo image and the end of the arm on the client id1 mobile phone screen, while the red envelope logo image tracks the end-of-arm motion;

[0020] The mobile phone of client id2 determines the red envelope and sends an instruction, and the mobile phone of client id1 obtains the local space orientation parameter. If the local space orientation parameter is outside the red envelope image parameter, that is, the set parameter threshold, the red envelope image will not be displayed. If the local space orientation parameter is within the red envelope image parameter, that is, the set parameter threshold, the animated red envelope image is superimposed on the scene video display, triggering the animated red envelope image to obtain the red envelope, and the mobile phone of client id1 uploads the electronic certificate and the server executes the amount transfer operation.

[0021] The interactive AR red envelope method is also characterized in that: after the first client id1 mobile phone uses its camera to shoot a scene containing the second client id2 mobile phone, the recognition software obtains the action parameters of the second client id2 mobile phone and uploads them to the server. At the same time, the second client id2 mobile phone obtains its motion parameters through its sensor and uploads them to the server. The server compares the consistency of the action parameters and motion parameters to determine the mobile phone corresponding to the second client id2. After identification by the recognition software, the id2 mobile phone is tracked and the image is superimposed. The red envelope logo image is superimposed on the end of the arm of the client id2 mobile phone holder.

[0022] The interactive AR red envelope method is also characterized in that: when the client id2 mobile phone moves, its built-in sensor obtains its acceleration vector a, and then calculates the velocity vector v=∫a*dt, and further calculates the motion trajectory s=∫∫a*(dt) 2 The first client id1 mobile phone simultaneously obtains the video of the client id2 mobile phone in motion to obtain the image pixel motion trajectory S, and calculates the image pixel velocity V=S′ by taking the first-order derivative of S. Further, the image pixel acceleration A=S′′ is calculated by taking the second-order derivative of S. The coordinates of the client id2 mobile phone in the scene are determined by comparing the consistency of the motion trajectory s and the image pixel motion trajectory S, or the consistency of the instantaneous velocity v and the image pixel velocity V, or the consistency of the acceleration a and the image pixel acceleration A.

[0023] The interactive AR red envelope method is also characterized in that: the client id2 mobile phone determines that the red envelope sends an instruction in response to the maximum acceleration value or set value of its sensor.

[0024] The interactive AR red envelope method is also characterized in that: the client id2 mobile phone determines that the red envelope is issued in response to the maximum value or set value of the calculated image pixel acceleration A=S′′.

[0025] or,

[0026] The interactive AR red envelope method is also characterized in that: the client id2 mobile phone modulates the identification id2 in its flash light wave and sends it outward, the client id1 mobile phone obtains the flash light wave signal through its camera and demodulates it to obtain the identification id2, and the client id1 mobile phone determines the scene position of the client id2 mobile phone on its screen and reports it to the server.

[0027] The interactive AR red envelope method is also characterized in that the client id2 mobile phone modulates the identifier id2 in its flash light wave in an amplitude modulation manner and sends it outward.

[0028] Or, the client id2 mobile phone modulates the identifier id2 in a frequency modulation manner and sends it out in its flash light wave.

[0029] Or, the client id2 mobile phone will modulate the identifier id2 in the f1\f2 dual-frequency mode in its flash light wave and send it outward, f1 represents 1 and f2 represents 0.

[0030] Further, 30Hz≥f1>0, 30Hz≥f2>0.

[0031] Going further, f1=30Hz, f2=20Hz.

[0032] The method is applied to instant messaging QQ and WeChat.

[0033] An interactive AR red envelope system, including a server and a client, is characterized by:

[0034] The server side includes a red envelope image library, an animation image generation unit, a push unit, a geographic location assignment unit, a geographic location range assignment unit, an action assignment unit, and a text adding unit;

[0035] The server also includes: client id mobile phone coordinates, a client tracking unit, the client id mobile phone coordinates are scene coordinates of the client id and the mobile phone obtained by the server, the client tracking unit realizes the movement tracking of a mobile phone of a certain client id, and also includes a motion calculation unit, which is responsible for calculating motion parameters according to the sensor signal obtained from the client;

[0036] The client includes at least an animated red envelope image acquisition unit, a geographic location positioning unit, a position acquisition unit, and an image superposition unit;

[0037] The client also includes: a sensor unit, an image recognition unit, an image tracking unit, and an action parameter calculation unit. The sensor unit is responsible for acquiring the action signal of the mobile phone. The image recognition unit is responsible for identifying the human body and its arms and determining the end of the arm. The image tracking unit is responsible for tracking the determined end of the arm. The action parameter calculation unit is responsible for calculating the action parameters of the end of the arm.

[0038] The action parameter calculation unit of the client is responsible for calculating the action parameters of the end of the arm, and the action parameters are action trajectories. The motion calculation unit of the server is responsible for calculating the motion parameters according to the sensor signals obtained from the client, and the motion parameters are motion trajectories. The server also includes a trajectory matching unit, which is responsible for matching the action trajectory and the motion trajectory to identify the consistency of the trajectories and determine the mobile phone corresponding to the client ID.

[0039] or:

[0040] The action parameter calculation unit of the client is responsible for calculating the action parameter of the end of the arm, and the action parameter is the action speed. The motion calculation unit of the server is responsible for calculating the motion parameter according to the sensor signal obtained from the client, and the motion parameter is the motion speed. The server also includes a speed matching unit, which is responsible for matching the action speed and the motion speed to identify the consistency of the speed and determine the mobile phone corresponding to the client ID;

[0041] or:

[0042] The action parameter calculation unit of the client is responsible for calculating the action parameters of the end of the arm, and the action parameters are action acceleration. The motion calculation unit of the server is responsible for calculating the motion parameters based on the sensor signals obtained from the client, and the motion parameters are motion acceleration. The server also includes an acceleration matching unit, which is responsible for matching the action acceleration and the motion acceleration to identify the consistency of the acceleration and determine the mobile phone corresponding to the client ID.

[0043] An interactive AR red envelope system, including a server and a client, is characterized by:

[0044] The server side includes a red envelope image library, an animation image generation unit, a push unit, a geographic location assignment unit, a geographic location range assignment unit, an action assignment unit, and a text adding unit;

[0045] The server also includes: client id mobile phone coordinates, a client tracking unit, the client id mobile phone coordinates are scene coordinates of the one-to-one correspondence between the client id and the mobile phone obtained by the server, and the client tracking unit realizes the movement tracking of a mobile phone of a certain client id;

[0046] The client includes at least an animated red envelope image acquisition unit, a geographic location positioning unit, a position acquisition unit, and an image superposition unit;

[0047] The client also includes: an encoding unit, a modulation unit, and a pixel demodulation unit. The encoding unit is responsible for converting the client ID into a transmittable transmission code, the modulation unit is responsible for modulating the transmission code and loading it into the light wave of the mobile phone flash, and the pixel demodulation unit is responsible for demodulating the video image from the mobile phone camera and other mobile phone flash flashes.

[0048] The interactive AR red envelope system is characterized in that the system is connected to or embedded in the instant messaging QQ or WeChat.

[0049] The beneficial effects of the present invention are: providing a method and system for overlaying a virtual world on the real world on a screen and performing interactive red envelopes, achieving a visual effect in which a person sending a red envelope holds a red envelope (red envelope logo image or red envelope image) at the end of his arm and throws the red envelope image through a parabolic action. The system can be based on an independent instant messaging IM platform, or through a third-party service (API), or embedded in an existing IM platform, such as QQ, WeChat, Momo, etc., and can realize an application mode of publishing virtual advertising images through an AR red envelope sending system. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the principle of the present invention.

[0051] Figure 2 This is the flow chart of the present invention (client id2 initiates a red envelope).

[0052] Figure 3 This is a flow chart of obtaining red envelopes on a mobile phone of the present invention.

[0053] Figure 4 This is a schematic diagram of the spatial orientation angle matching of the animated red envelope image.

[0054] Figure 5 Configure a unit for the server (amount transfer execution unit).

[0055] Figure 6 Configure the unit for the server (animated red envelope image generation unit).

[0056] Figure 7 Configure the unit for the mobile end.

[0057] Figure 8 A schematic diagram of determining the corresponding ID of a client based on mobile phone action recognition.

[0058] Fig. 9 This is a flow chart of the red envelope issuing instruction in response to the maximum acceleration value of the mobile phone sensor of the present invention.

[0059] Fig.10 The flowchart is for determining the corresponding ID of the client based on mobile phone action recognition.

[0060] Fig.11 This is a system configuration diagram for determining the corresponding ID of a client based on mobile phone action recognition.

[0061] Fig.12 The flowchart is for determining the corresponding ID of the client based on the flash of the mobile phone.

[0062] Fig.13 This is a system configuration diagram for determining the corresponding ID of a client based on the mobile phone flash. DETAILED DESCRIPTION

[0063] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0064] Figure 1 101 is a schematic diagram of the principle of the present invention, 102 is a server, which is connected to the client through the Internet, 103 is a virtual image of a red envelope logo, the red envelope logo image can be a front image in an animated red envelope image, the red envelope logo virtual image does not exist in the real scene, and is represented by a dotted line in the figure, 104 is a human body, the client id1 mobile phone 101 obtains a video image of the real scene 104 through its camera, and displays an image 106 of the real scene 104 on the screen 105 of the mobile phone 101, 107 is the arm of the human body 104, and the display image 108 of the virtual image of the red envelope logo 103 on the screen of the mobile phone 101 is the red envelope logo image, the client id1 mobile phone recognizes the human body through image recognition software, further recognizes the arm, and further determines the position of the end of the arm, 107 is the arm image of the human body image 106, and after determining the position of the image at the end of the arm, the red envelope logo image 108 is superimposed and displayed on the end of the arm 107 and tracked with the arm movement, thus forming a scene image of holding a red envelope, 109 is the client idn mobile phone. The scenario formed in this way is: after each client detects that there is a red envelope, it obtains human body movements through the camera. Once someone raises his arm, it is assumed that a red envelope is about to be sent, and a red envelope logo image is superimposed at the end of the arm. The red envelope logo image tracks the movement at the end of the arm, but it is impossible to determine which client ID sends the red envelope, so it is necessary to clearly define the correspondence between the client ID that initiates the red envelope and the real mobile phone, as well as the spatial orientation of the red envelope and the geographical location restrictions (Chinese patent application 2016105984877 A method and system for AR has disclosed a method and system for AR red envelopes based on geographical location).

[0065] Therefore, the application scenarios of the present invention include: ① After each client detects a red envelope, it acquires the human body and action through the camera. Once someone (anyone) raises his or her arm, it is assumed that a red envelope is about to be sent, and a red envelope logo image is superimposed at the end of the arm. The red envelope logo image tracks the action at the end of the arm. When the red envelope issuing client determines to send the red envelope, each client displays an animated red envelope image superimposed on a scene video, and the red envelope image can be acquired by triggering in the scene of any person who raises his or her arm; ② Further, the mobile phone and holder corresponding to the client ID of the red envelope sending client are identified. Only the red envelope logo image is superimposed at the end of the arm of the person sending the red envelope, and only the scene video containing the person sending the red envelope displays the animated red envelope image superimposed on the scene video, and the red envelope image is acquired by triggering in the scene of the person sending the red envelope.

[0066] The red envelope logo image can be a red envelope image, a currency image, an ingot image, an ancient coin image, etc.; the animated red envelope image can also be a red envelope image, a currency image, an ingot image, an ancient coin image, etc. The animated red envelope image can be anthropomorphic in shape, such as adding wings.

[0067] The animated red envelope image is composed of a series of frames. The starting point of the animation playback coordinates is the red envelope logo image. The animation forms include changing from small to large (the visual effect is from far to near), screen space activities, image flipping, image deformation, etc.

[0068] Figure 2 This is a flow chart of the present invention (client id2 initiates a red envelope), including a client and a server, and the physical scene includes at least a first client id1 mobile phone and a second client id2 mobile phone. The second client id2 mobile phone initiates a red envelope request, and the client id2 mobile phone obtains the spatial orientation parameter through its sensor and sets the spatial orientation parameter of the red envelope image to be sent, that is, the setting parameter, and sets the amount, score setting, text description, etc. at the same time, and uploads the setting parameter to the server. The server generates an animated red envelope image with the spatial orientation parameter according to the setting parameter and publishes the animated red envelope image, and the client id1 mobile phone downloads the animated red envelope image to the local computer;

[0069] The red envelope receiving program of the client id1 mobile phone is started, and a red envelope logo image is prepared (the red envelope logo image is in the client red envelope receiving program). The client id1 mobile phone uses its camera to capture the scene containing the second client id2 mobile phone, and then uses the recognition software to determine the end-of-arm motion of the client id2 mobile phone holder and superimpose the red envelope logo image and the end-of-arm motion on the client id1 mobile phone screen. At the same time, the red envelope logo image tracks the end-of-arm motion;

[0070] The mobile phone of client id2 determines the red envelope and sends an instruction, and the mobile phone of client id1 obtains the local space orientation parameter. If the local space orientation parameter is outside the red envelope image parameter, that is, the set parameter threshold, the red envelope image will not be displayed (a certain display threshold is set); if the local space orientation parameter is within the red envelope image parameter, that is, the set parameter threshold, the animated red envelope image is superimposed on the scene video display, triggering the animated red envelope image to obtain the red envelope, and the mobile phone of client id1 uploads the electronic voucher and the server executes the amount transfer operation.

[0071] Trigger the red envelope frame in the animated red envelope image, that is, a sub-frame can be set in the animated red envelope image to carry the electronic certificate. The best solution is the front image of the red envelope. Of course, all the frames that make up the animated red envelope image can also carry the electronic certificate. In this way, after triggering any frame of the animated red envelope image, the red envelope can be obtained according to time priority. Of course, the electronic certificate in this solution carries a timestamp.

[0072] The client id2 mobile phone obtains the spatial orientation parameters through its sensor and sets the spatial orientation parameters of the red envelope image to be sent, that is, setting parameters. The steps are: the client id2 mobile phone holder obtains the scene direction map of the red envelope to be sent through its camera and displays it on its screen, and determines the spatial orientation of the red envelope after adjusting the framing. At this time, the setting parameters are determined through the set confirmation button. The parameters are obtained in real time by the mobile phone sensor, including the direction angle β obtained by the geomagnetic sensor and the inclination angle α obtained by the inclination sensor. After the setting parameters are uploaded to the server, the server determines the spatial orientation parameters of the generated animated red envelope image.

[0073] Red envelope image parameter threshold setting: a. Client id2 mobile phone obtains spatial orientation parameters through its sensor and sets the spatial orientation parameters of the red envelope image to be sent, that is, the spatial orientation parameters of the animated red envelope image generated by the server after the parameters are set, and sets a tolerance threshold for the spatial orientation parameters of the animated red envelope to be displayed on any client such as client id1; or b. Any client such as client id1 performs a display based on the determined spatial orientation parameters with the threshold as the screening, and the client id2 mobile phone obtains spatial orientation parameters through its sensor and sets the spatial orientation parameters of the red envelope image to be sent, that is, the tolerance threshold for the spatial orientation parameter range of the animated red envelope image generated by the server after the parameters are set; or c. The animated red envelope image parameters are dynamically displayed against the spatial orientation parameters of the local machine, and the client id2 mobile phone obtains spatial orientation parameters through its sensor and sets the spatial orientation parameters of the red envelope image to be sent, that is, the setting After the parameters are obtained, the server determines the spatial orientation parameters of the generated animated red envelope image. Any client, such as client id1, uses the real-time spatial orientation parameters of the mobile phone (obtained in real time by the mobile phone sensor) as a reference when displaying the animated red envelope image, and the spatial orientation parameters of the animated red envelope image as a benchmark (fixed at this spatial orientation for display, that is, the animated red envelope image is fixed in the relative position of the real scene). When superimposed display, if the spatial orientation of the mobile phone moves the animated red envelope image out of the framing range, it cannot be displayed, that is, the threshold is the framing range, and the steps are: ① Any client mobile phone obtains the animated red envelope image with spatial orientation parameters (α0, β0); ② The mobile phone obtains the real-time spatial orientation parameters (α1, β1) of the local sensor; ③ The animated red envelope image is displayed on the screen with parameters (α0, β0), and the real scene image is displayed on the screen with parameters (α1, β1), and the animated red envelope image is superimposed on the real scene image. If α1-α0 or β1-β0 is greater than the threshold, the animated red envelope image moves out of the framing range and is not displayed. Further: ① Any client mobile phone obtains an animated red envelope image with spatial orientation parameters (α0+-△α, β0+-△β); ② The mobile phone obtains the real-time spatial orientation parameters (α1, β1) of the local sensor; ③ The animated red envelope image is displayed floatingly on the screen with parameters (α0+-△α, β0+-△β), and the real-scene image is displayed on the screen with parameters (α1, β1), and the animated red envelope image is superimposed on the real-scene image. If α1-(α0+-△α) or β1-(β0+-△β) is greater than the threshold, the animated red envelope image moves out of the framing range and is not displayed. △α and △β are floating parameters set by the server, which make the animated image more jumping and increase the difficulty of the game.

[0074] The red envelope sending instruction can be triggered by the client ID2 mobile phone holder choosing an appropriate time on his mobile phone, or by the acceleration of the mobile phone sensor, or by the pixel acceleration of the video image at the end of the client ID2 mobile phone holder's arm taken by other mobile phones.

[0075] The dynamic display of the animated image parameters compared with the local spatial orientation parameters in scheme c is such as the existing electronic compass and level. The interface parameters of the compass and level are the local screen plane, the pointer and bubble setting parameters are the north-south direction and the vertical angle, and the display position of the pointer and bubble on the screen is controlled by the local sensor parameters.

[0076] Figure 3 The flowchart of obtaining red envelopes on the mobile phone of the present invention is as follows. Considering that the interactive red envelope distribution is limited to a small geographical location range, a geographical location restriction is added, including the following steps: 301, obtaining a real scene video image, obtained by a camera, and browsed on the mobile phone screen; 302, obtaining the current geographical location information, the mobile phone obtains the latitude and longitude, and altitude information of its own location, which can be from GPS or base station LBS; 303, obtaining the mobile phone's orientation information, the orientation information includes the horizontal inclination and direction angle of the mobile phone, the pointing posture of the mobile phone can be determined by the orientation information, the mobile phone's spatial orientation parameter is determined by the horizontal inclination and direction angle, the horizontal inclination parameter comes from the inclination sensor, and the direction angle comes from From the geomagnetic field sensor; 304, report the geographical location and orientation information, that is, report the latitude and longitude, altitude, horizontal inclination, direction angle and other information of the mobile phone's own location; 305, obtain the red envelope information within the current geographical location range, that is, obtain the red envelope information within the geographical location range set by the server; 306, obtain the animated red envelope image; 307 The animated red envelope image is superimposed on the real scene video image, and the superimposed image video is displayed on the mobile phone screen; 308, trigger waiting, wait for the corresponding trigger according to the server's action assignment; 309, trigger information upload, once the trigger is successful, the mobile phone will upload the trigger information (electronic certificate) to the server; 310, get the red envelope.

[0077] Figure 4 The figure is a schematic diagram of the spatial orientation azimuth angle matching of the animated red envelope image. 301 is the front frame of the animated red envelope image, and 302 is the oblique frame of the animated red envelope image. The animated red envelope image is composed of a series of action images, and the animation actions are reflected by different image perspectives of each frame, including horizontal movement, vertical movement, free fall movement, floating movement, or a combination of the above movements. The spatial orientation parameters of the animated image are: address longitude and latitude 31.252987, 121.4481, direction angle β is 20 degrees south by east, and horizontal angle α is 30 degrees. If the spatial orientation parameters of the client id1 mobile phone are: address longitude and latitude 31.262988, 121.4582, direction angle β is 21 degrees south by east, and horizontal angle α is 25 degrees, this position is assumed to be able to overlay the animated red envelope image (the geographic location range and orientation threshold meet the setting requirements).

[0078] Figure 5The server configuration unit (amount transfer execution unit) includes a trigger information acquisition unit, which is used to obtain trigger information from the client, such as touch screen clicks, taking photos, etc.; an acquisition and recognition unit, which is used to recognize the client's behavior of obtaining the animated red envelope image electronic certificate; and an amount transfer unit, which is used to perform accounting and settlement for each client.

[0079] Figure 6 The server configuration unit (animated red envelope image generation unit) includes a red envelope image library, an animated image generation unit, a push unit, a geographic location assignment unit, a geographic location range assignment unit, an action assignment unit, an amount assignment unit, a text adding unit, and an validity period assignment unit.

[0080] Figure 7 The mobile phone configuration unit, GPS, LBS unit, is used to obtain geographic location information; the network unit includes communication networks 3G, 4G, etc., and also includes WIFI wireless network; the animated red envelope image acquisition unit obtains the animated red envelope image according to the GPS, LBS unit and the network unit; the orientation acquisition unit obtains the real-time orientation posture of the mobile phone, that is, the spatial orientation, according to the direction angle sensor and the horizontal sensor (two-dimensional angle); the image superposition unit superimposes the animated red envelope image and the real scene video obtained by the camera on the mobile phone screen according to the orientation information, and the image display unit is responsible for the display of the mobile phone screen.

[0081] The orientation of a mobile phone is the spatial orientation parameter of the mobile phone, which refers to the normal direction angle of the mobile phone plane (or the front direction of the camera), including horizontal inclination (two-dimensional horizontal inclination) and direction angle (north-south orientation). It is the posture and orientation information of the mobile phone. In order to further improve the orientation information, the mobile phone sensor information is collected, including: acceleration, magnetic field, rotation vector, gyroscope, light sensor, pressure, etc.

[0082] Figure 8 This is a schematic diagram of determining the corresponding ID of the client based on mobile phone action recognition. Figure 1 In the schematic diagram, the mobile phone of client ID 1 cannot determine which client ID sent the red envelope, so the correspondence between the client ID that sent the red envelope and the real mobile phone must be clearly defined. In this way, assuming that the client ID 2 is held in the hand of the holder 104, refer to Fig.10The flowchart is for determining the corresponding id of the client based on the action recognition of the mobile phone. The mobile phone of client id1 establishes the scene coordinates X and Y by taking pictures (refer to the coordinates formed by the camera device through the array of photosensitive units). 107 is the captured arm of the holder. The human body is recognized by the image recognition software, and the arm is further recognized. Then, the end of the arm, that is, the position of the mobile phone, is further determined. When the physical mobile phone of client id2 is shaken left and right by the holder, a motion trajectory is generated. The motion trajectory is a function 802 of the displacement change in the X direction along the time t axis. The video captured by the mobile phone of client id1 also obtains a function 803 of the displacement change in the S direction along the time t axis. Function 803 is the image pixel motion trajectory S. The first-order derivative of S is calculated to calculate the image pixel speed V=S′. Further, the second-order derivative of S is calculated to calculate the image pixel acceleration A=S′′. The image pixel motion trajectory S, the image pixel speed V=S′, and the image pixel acceleration A=S′′ are uploaded to the server 102. At the same time, when the client id2 mobile phone moves, its built-in sensor obtains its acceleration vector a and uploads it to the server 102, and then calculates the velocity vector v=∫a*dt, and further calculates the motion trajectory s=∫∫a*(dt) 2 Thus, by comparing the consistency between the motion trajectory s and the action trajectory S, the consistency between the instantaneous velocity v and the image pixel velocity V, and the consistency between the acceleration a and the image pixel acceleration A, the position and coordinates of the client id2 mobile phone in the scene are determined.

[0083] The sensor used by the mobile phone to obtain the acceleration vector a is generally a gyroscope. The three-dimensional coordinates of X, Y, and Z are established. The acceleration vector a can be decomposed into ax, ay, and az. The motion trajectory s is synthesized by the x, y, and z trajectories, that is, x=∫∫ax*(dt) 2 y = ∫∫ay*(dt) 2 、z=∫∫az*(dt) 2 The motion trajectory s is a three-dimensional trajectory in space, and the image captured by the mobile phone is a plane image, so the projection of the motion trajectory s on a certain plane is matched and identified with the image pixel motion trajectory S, the projection of the instantaneous speed v on a certain plane is matched and identified with the image pixel speed V, and the projection of the acceleration a on a certain plane is matched and identified with the image pixel acceleration A.

[0084] The image motion speed is the pixel speed of the image. The image pixel coordinates are expressed in x and y coordinates. The image pixel speed is V=((Δx) 2 +(Δy) 2 ) 1 / 2 *f, Δx is the x-axis offset of the image pixel, Δy is the y-axis offset of the image pixel, f is the frame frequency, and the image pixel acceleration is A=(V1-V2)*f. It takes at least one frame to detect the image pixel speed, so V1 and V2 are the image pixel speeds of adjacent frames.

[0085] Component detection is a better solution. The coordinates are unified into horizontal and vertical coordinates X and Y. Assume that the camera plane is selected as the X and Y planes, ignore the Z coordinate, the mobile phone motion acceleration is ax and ay, the motion speed is vx=∫ax*dt, vy=∫ay*dt, and the motion trajectory is x=∫∫ax*(dt) 2 y = ∫∫ay*(dt) 2 ; Image pixel speed Vx=Δx*f, Vy=Δy*f, Ax=(Vx1-Vx2)*f, Ay=(Vy1-Vy2)*f, establish matching function:

[0086] Ax compares Ax = (Vx1-Vx2)*f;

[0087] ay comparison Ay=(Vy1-Vy2)*f;

[0088] vx = ∫ax*dt versus Vx = Δx*f;

[0089] vy=∫ay*dt versus Vy=Δy*f;

[0090] x=∫∫ax*(dt) 2 Compare X values;

[0091] y=∫∫ay*(dt) 2 Compare the Y values,

[0092] Numerical comparison is performed at each moment of a matching cycle, and the time interval can be selected as a frame period, so that trajectory matching, speed matching, and acceleration matching are converted into numerical comparison.

[0093] Considering that the matching and comparison program is completed in one matching cycle, the two matching quantities at any moment must ensure time consistency, so a timestamp is included in the transmitted signal data packet. The timestamp guarantees the comparison moment. Timestamp technology is a commonly used technical means in the industry.

[0094] Fig.10 The flowchart for determining the corresponding ID of a client based on mobile phone action recognition is as follows: Assume that the physical location coordinates of the mobile phone of client ID1 are (x1, y1), the physical location coordinates of the mobile phone of client ID2 are (2, 3), the physical location coordinates of the mobile phone of client ID3 are (2, 1), and the physical location coordinates of the mobile phone of client IDn are (xm, yn). The steps include:

[0095] The client id2 mobile phone is shaken in the hand of the holder, that is, it moves at the end of the holder's arm, and the sensor (such as a gyroscope) of the client id2 mobile phone obtains motion parameters and uploads the motion parameters to the server, which stores the motion parameters. The motion parameters at least include acceleration a;

[0096] Any client id1 mobile phone captures at least a scene video of the client id2 mobile phone through its camera and establishes X, Y coordinates, identifies the body and arm of the client id2 mobile phone holder and determines the body coordinates (2,3), tracks the end of the arm to obtain motion parameters such as image motion trajectory S, image speed V, image acceleration A, uploads the motion parameters and corresponding coordinates to the server, and the server stores the motion parameters;

[0097] The server matches and compares the obtained action parameters and motion parameters, including the motion trajectory s and the image action trajectory S, or the instant speed v and the image speed V, or the motion acceleration a and the image acceleration A. The server establishes the X and Y coordinates of the physical position of each client id mobile phone and determines the mobile phone coordinates of client id2 (2,3).

[0098] As for the matching method, the motion trajectory s and the image motion trajectory S can be matched, or the instantaneous speed v and the image speed V can be matched, or the motion acceleration a and the image acceleration A can be matched, or all three can be matched at the same time, or two out of the three can be matched.

[0099] The server also includes red envelope distribution based on tracking the mobile phone of client id2, and realizes the red envelope distribution display based on tracking the mobile phone of client id2 on the screen of the mobile phone of client id1.

[0100] In this way, any client idn mobile phone can determine the client id2 mobile phone holder through the above method to issue a red envelope.

[0101] The method of determining the corresponding ID of the client based on the action recognition of the mobile phone is based on image recognition, including existing mature feature recognition, object contour recognition, color recognition, action recognition, face recognition, human body recognition, human limb recognition, text recognition, etc., and corresponding recognition strategies are adopted in different environments. Since it is difficult to recognize the physical appearance of a mobile phone, the mature human body recognition technology is used to first determine the human body, and then the arm is recognized through the human body, and the end of the arm is further determined, so that the physical location of the mobile phone is indirectly determined, so as to realize the corresponding relationship between the client ID and its mobile phone.

[0102] Fig. 9This is a flow chart of the red envelope issuing instruction in response to the maximum acceleration value of the mobile phone sensor of the present invention, which is characterized in that the red envelope issuing instruction of the client id2 mobile phone is in response to the client id2 mobile phone obtaining the maximum acceleration value of its sensor, so that the client id2 mobile phone holder can issue the red envelope issuing instruction by throwing the mobile phone vigorously, and after the client receives the red envelope issuing instruction, an animated red envelope image and a real scene are superimposed and displayed, so that a continuous action of the client id2 mobile phone holder holding the red envelope logo image and throwing the animated red envelope image is obtained, and a frame image in the animated red envelope image is taken as the red envelope logo image, thereby forming a visual effect of throwing the red envelope by hand.

[0103] Since the mobile phone acceleration sensor has a maximum value, the maximum value can be used as the trigger signal for sending the red envelope, or an acceleration value can be set. As long as the sensor obtains an acceleration exceeding this value, the red envelope sending instruction will be executed. The setting value of the acceleration is generally a larger value, that is, the acceleration value of the hand throwing action.

[0104] Another implementation scheme for triggering the red envelope issuing instruction by acceleration is: the mobile phone of client id2 determines the red envelope issuing instruction in response to the maximum value of the calculated image pixel acceleration A=S′′, or the mobile phone of client id2 determines the red envelope issuing instruction in response to the set value of the calculated image pixel acceleration A=S′′. The set value of acceleration generally takes a larger value, that is, the acceleration value of the throwing action by hand.

[0105] If a certain number of red envelopes are distributed, it can be set to continuously throw out red envelopes as the acceleration value of the sensor is continuously triggered until the number of red envelopes distributed is completed. Or it can be set to continuously throw out red envelopes as the acceleration value of the image pixel is continuously triggered until the number of red envelopes distributed is completed.

[0106] Fig.11 This is a system configuration diagram for determining the client's corresponding ID based on mobile phone action recognition, including the client and server:

[0107] The client includes: a red envelope service system client, a sensor unit, an image recognition unit, an image tracking unit, and a motion parameter calculation unit. The red envelope service system client is such as the client disclosed in Chinese patent application 2016105984877. The sensor unit is responsible for obtaining the mobile phone motion signal, such as the gyroscope obtaining acceleration. The image recognition unit is responsible for identifying the human body and its arm and determining the end of the arm. The image tracking unit is responsible for tracking the determined end of the arm. The motion parameter calculation unit is responsible for calculating the motion parameters of the arm end. The motion parameters include image pixel motion trajectory, image pixel speed, and image pixel acceleration.

[0108] The server includes: a red envelope service system, client id mobile phone coordinates, and a client tracking unit. The red envelope service system is such as the server side disclosed in Chinese patent application 2016105984877. The client id mobile phone coordinates are the scene coordinates of the client id and the mobile phone obtained by the server, and the client tracking unit realizes the mobile tracking of a certain client id mobile phone. The server also includes a motion calculation unit, which is responsible for calculating motion parameters based on the sensor signals obtained from the client. The motion parameters include motion trajectory, motion speed, and motion acceleration. The server also includes a trajectory matching unit, a speed matching unit, and an acceleration matching unit.

[0109] The action parameter calculation unit of the client is responsible for calculating the action parameters of the end of the arm, and the action parameters are the image action trajectory. The motion calculation unit of the server is responsible for calculating the motion parameters based on the sensor signal obtained from the client, and the motion parameters are the motion trajectory. The trajectory matching unit is responsible for matching the image action trajectory and the motion trajectory to identify the consistency of the trajectory and determine the mobile phone corresponding to the client ID.

[0110] The action parameter calculation unit of the client is responsible for calculating the action parameters of the end of the arm, and the action parameters are the image action speed. The motion calculation unit of the server is responsible for calculating the motion parameters based on the sensor signals obtained from the client, and the motion parameters are the motion speed. The speed matching unit is responsible for matching the image action speed and the motion speed to identify the consistency of the speed and determine the mobile phone corresponding to the client ID.

[0111] The action parameter calculation unit of the client is responsible for calculating the action parameters of the end of the arm, and the action parameters are image action acceleration. The motion calculation unit of the server is responsible for calculating the motion parameters based on the sensor signals obtained from the client, and the motion parameters are motion acceleration. The acceleration matching unit is responsible for matching the image action acceleration and the motion acceleration to identify the consistency of the acceleration and determine the mobile phone corresponding to the client ID.

[0112] Fig.12 The flowchart for determining the corresponding ID of a client based on the flash of a mobile phone is as follows: Assume that the physical location coordinates of the mobile phone of client ID 1 are (x1, y1) and the physical location coordinates of the mobile phone of client ID 2 are (2, 3). The steps include:

[0113] (1) The client id2 mobile phone generates a modulation code using its id identification information and modulates it in the flash light wave and sends it out;

[0114] (2) Client id1 mobile phone obtains scene video through its camera to establish X and Y coordinates, and demodulates the change of flash image pixels of client id2 mobile phone over time to obtain the corresponding id2 identifier, and uploads the identifier and corresponding coordinates to the server;

[0115] Demodulate the image pixels over time: obtain the pixel brightness change parameters of each frame within a certain period of time, and perform amplitude demodulation or frequency demodulation according to the modulation method;

[0116] (3) The server establishes X and Y coordinates. After obtaining the id2 identifier and the corresponding coordinates, the server establishes a physical location map of the mobile phone.

[0117] (4) The server tracks the red envelope distribution of client id2 and displays the red envelope distribution on the mobile phone screen of client id1.

[0118] In this way, any client idn mobile phone can determine the client id2 mobile phone holder to issue a red envelope by parsing the id2 identifier transmitted by the client id2 mobile phone flash light.

[0119] Fig.13 The system configuration diagram for determining the corresponding ID of its client based on the mobile phone flash includes a client and a server. The client includes: a red envelope service system client, a coding unit, a modulation unit, and a pixel demodulation unit. The red envelope service system client is such as the client disclosed in Chinese patent application 2016105984877. The coding unit is responsible for converting the client ID into a transmittable transmission code, such as converting it into a binary code. The modulation unit is responsible for modulating the transmission code and loading it into the light wave of the mobile phone flash. The pixel demodulation unit is responsible for demodulating the video image of other mobile phone flash flashes from the mobile phone camera; the server includes: a red envelope service system, a client ID mobile phone coordinates, a client tracking unit, or a video pixel demodulation unit. The red envelope service system is such as the server disclosed in Chinese patent application 2016105984877. The client ID mobile phone coordinates are the scene coordinates of the client ID and the mobile phone obtained by the server in a one-to-one correspondence. The client tracking unit realizes the mobile tracking of a certain client ID mobile phone. The video pixel demodulation unit is responsible for demodulating the ID identification information loaded in the video image of the mobile phone flash flash in the client scene video.

[0120] The above application modes and rules do not limit the basic features of the method and system of the present invention, nor do they limit the scope of protection of the present invention. For example, the ID in the present invention may also refer to a unique code temporarily assigned by the server in the scenario. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An interactive AR red envelope method, comprising a client and a server, characterized in that: The physical scene includes at least a first client id1 mobile phone and a second client id2 mobile phone. The second client id2 mobile phone initiates a red envelope request. The client id2 mobile phone obtains the spatial orientation parameters through its sensor and sets the spatial orientation parameters of the red envelope image to be sent, i.e., setting parameters, and uploads the setting parameters to the server. The server generates an animated red envelope image with the spatial orientation parameters according to the setting parameters and publishes the animated red envelope image. After the client id1 mobile phone uses its camera to capture the scene containing the second client id2 mobile phone, it determines the end-of-arm motion of the client id2 mobile phone holder through recognition software and superimposes the red envelope logo image and the end of the arm on the client id1 mobile phone screen, while the red envelope logo image tracks the end-of-arm motion; The mobile phone of client ID2 determines that the red envelope sends an instruction, and the mobile phone of client ID1 obtains the local space orientation parameter. If the local space orientation parameter is outside the red envelope image parameter, i.e., the set parameter threshold, the red envelope image is not displayed. If the local space orientation parameter is within the red envelope image parameter, i.e., the set parameter threshold, the animated red envelope image is superimposed on the scene video display, and the animated red envelope image is triggered to obtain the red envelope. The mobile phone of client ID1 uploads the electronic voucher and the server performs the amount transfer operation; Its feature is that when the client ID2 mobile phone moves, its built-in sensor obtains its acceleration vector a, and then calculates the velocity vector v=∫a*dt, and further calculates the motion trajectory s=∫∫a*(dt) 2 The first client id1 mobile phone simultaneously obtains the video of the client id2 mobile phone in motion to obtain the image pixel motion trajectory S, and calculates the image pixel velocity V=S′ by taking the first-order derivative of S. Further, the image pixel acceleration A=S′′ is calculated by taking the second-order derivative of S. The coordinates of the client id2 mobile phone in the scene are determined by comparing the consistency of the motion trajectory s and the image pixel motion trajectory S, or the consistency of the instantaneous velocity v and the image pixel velocity V, or the consistency of the acceleration a and the image pixel acceleration A.

2. The interactive AR red envelope method according to claim 1, further characterized by: After the first client id1 mobile phone uses its camera to shoot a scene including the second client id2 mobile phone, it obtains the action parameters of the second client id2 mobile phone through the recognition software and uploads them to the server. At the same time, the second client id2 mobile phone obtains its motion parameters through its sensor and uploads them to the server. The server compares the consistency of the action parameters and motion parameters to determine the mobile phone corresponding to the second client id2. After recognition by the recognition software, the id2 mobile phone is tracked and the image is superimposed. The red envelope logo image is superimposed on the end of the arm of the person holding the client id2 mobile phone.

3. The interactive AR red envelope method according to claim 1, further characterized by: The client id2 phone modulates the identifier id2 in its flash light wave and sends it outward. The client id1 phone obtains the flash light wave signal through its camera and demodulates it to obtain the identifier id2. The client id1 phone determines the scene position of the client id2 phone on its screen and reports it to the server.

4. The interactive AR red envelope method according to claim 3 is further characterized by: The client id2 mobile phone modulates the identifier id2 in the flash light wave in an amplitude modulation manner and sends it outward; Or, the client id2 mobile phone modulates the identifier id2 in the flash light wave in a frequency modulation manner and sends it outward; Or, the client id2 mobile phone will modulate the identifier id2 in the f1\f2 dual-frequency mode in its flash light wave and send it outward, f1 represents 1 and f2 represents 0.

5. The interactive AR red envelope method according to claim 1, 2, 3 or 4, further characterized by: The client id2 mobile phone determines that the red envelope sends an instruction in response to the maximum or set value of the acceleration of its sensor.

6. The interactive AR red envelope method according to claim 1, 2, 3 or 4, further characterized by: The client id2 mobile phone determines that the red envelope is issued in response to the maximum value or set value of the calculated image pixel acceleration A=S′′.

7. The interactive AR red envelope method according to claim 1, 2, 3 or 4, further characterized by: The method is applied to instant messaging QQ and WeChat.

8. The system of the interactive AR red envelope method according to claim 1 comprises a server and a client, wherein: The server side includes a red envelope image library, an animation image generation unit, a push unit, a geographic location assignment unit, a geographic location range assignment unit, an action assignment unit, and a text adding unit; The server also includes: client id mobile phone coordinates, a client tracking unit, the client id mobile phone coordinates are scene coordinates of the client id and the mobile phone obtained by the server, the client tracking unit realizes the movement tracking of a mobile phone of a certain client id, and also includes a motion calculation unit, which is responsible for calculating motion parameters according to the sensor signal obtained from the client; The client includes at least an animated red envelope image acquisition unit, a geographic location positioning unit, a position acquisition unit, and an image superposition unit; The client also includes: a sensor unit, an image recognition unit, an image tracking unit, and an action parameter calculation unit. The sensor unit is responsible for acquiring the action signal of the mobile phone. The image recognition unit is responsible for identifying the human body and its arms and determining the end of the arm. The image tracking unit is responsible for tracking the determined end of the arm. The action parameter calculation unit is responsible for calculating the action parameters of the end of the arm. The action parameter calculation unit of the client is responsible for calculating the action parameters of the end of the arm, and the action parameters are action trajectories. The motion calculation unit of the server is responsible for calculating the motion parameters according to the sensor signals obtained from the client, and the motion parameters are motion trajectories. The server also includes a trajectory matching unit, which is responsible for matching the action trajectory and the motion trajectory to identify the consistency of the trajectories and determine the mobile phone corresponding to the client ID. or: The action parameter calculation unit of the client is responsible for calculating the action parameter of the end of the arm, and the action parameter is the action speed. The motion calculation unit of the server is responsible for calculating the motion parameter according to the sensor signal obtained from the client, and the motion parameter is the motion speed. The server also includes a speed matching unit, which is responsible for matching the action speed and the motion speed to identify the consistency of the speed and determine the mobile phone corresponding to the client ID; or: The action parameter calculation unit of the client is responsible for calculating the action parameters of the end of the arm, and the action parameters are action acceleration. The motion calculation unit of the server is responsible for calculating the motion parameters based on the sensor signals obtained from the client, and the motion parameters are motion acceleration. The server also includes an acceleration matching unit, which is responsible for matching the action acceleration and the motion acceleration to identify the consistency of the acceleration and determine the mobile phone corresponding to the client ID.

9. The system of the interactive AR red envelope method according to claim 1, comprising a server and a client, characterized in that: The server side includes a red envelope image library, an animation image generation unit, a push unit, a geographic location assignment unit, a geographic location range assignment unit, an action assignment unit, and a text adding unit; The server also includes: client id mobile phone coordinates, a client tracking unit, the client id mobile phone coordinates are scene coordinates of the one-to-one correspondence between the client id and the mobile phone obtained by the server, and the client tracking unit realizes the movement tracking of a mobile phone of a certain client id; The client includes at least an animated red envelope image acquisition unit, a geographic location positioning unit, a position acquisition unit, and an image superposition unit; The client also includes: an encoding unit, a modulation unit, and a pixel demodulation unit. The encoding unit is responsible for converting the client ID into a transmittable transmission code, the modulation unit is responsible for modulating the transmission code and loading it into the light wave of the mobile phone flash, and the pixel demodulation unit is responsible for demodulating the video image from the mobile phone camera and other mobile phone flash flashes.

10. A system for an interactive AR red envelope method according to claim 8 or 9, characterized in that: The system is connected to or embedded in instant messaging QQ or WeChat.

Images