A motion detection method and system based on mobile AR

Abstract

The present invention discloses a method and system for detecting motion based on mobile AR, the method comprising the following steps: obtaining the distance between the user and the virtual character during shooting, so that the distance is within a set distance range; ensuring that the angle between the user and the virtual character during shooting is within a set angle range, and the angle between the virtual character's orientation and the camera is within a set angle range; when the distance is within the set distance range, the angle is within the set angle range, and the angle is within the set angle range, the current motion of the virtual character is detected by a state machine, thereby determining whether the virtual character is performing a specific motion. The present invention determines whether there is a specific character in the picture and the character's animation state by performing program determination during the photo-taking process, thereby making it easier to perform photo detection and realize motion detection of the virtual character, reducing the difficulty and cost of motion recognition.

Description

Technical Field

[0001] The present invention relates to the field of AR motion recognition technology, and in particular to a motion detection method and system based on mobile AR. Background Art

[0002] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and video, and applies it to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0003] Common AR devices (AR glasses, AR projectors) all feature a range of environmental detection sensors, including depth cameras and color cameras, enabling them to recognize gestures, 3D objects, planes, and images, and register virtual information (text, images, 3D models, video, and sound) to real-world target locations. For example, Google's World AR and Baidu's Spatial AR aim to combine the VPS's 6DoF high-precision position and orientation with the SLAM tracking and positioning system on AR terminals to enable AR experiences in any spatial scenario. In general, large-scale AR relies on the VPS providing 6DoF information at a fixed frequency (e.g., 5 seconds per second) to the AR terminal's SLAM for tracking error correction, achieving a high-precision AR experience.

[0004] AR technology is applied in some applications, and virtual characters are provided in the AR application. The virtual characters can be objects with images of animals or cartoons, etc. The AR application can superimpose the virtual characters in the real environment in real time. For example, in the visual augmented reality, the user can observe the real environment surrounding the virtual character through the display. The AR application can also implement AR photography technology to capture the augmented reality after the real environment and the virtual character are superimposed on the same screen. During the AR photography process, the user can operate the virtual character, such as zooming in, zooming out, dragging, etc., which often requires the detection of actions in the photo. Usually, the detection of actions in the image is carried out, that is, first taking a photo, and then determining whether there is a specific action of a specific character in the photo. This adopts the method of image recognition. However, such action recognition requires a lot of preliminary machine learning and calculation, which greatly increases the recognition time and the cost of action recognition. Summary of the Invention

[0005] In view of the above technical problems, the purpose of the present invention is to provide a motion detection method and system based on mobile AR, so as to solve the problem that when detecting motion in traditional AR photos, a large amount of preliminary machine learning and calculation is required, resulting in long recognition time and high motion recognition cost.

[0006] The present invention adopts the following technical solutions:

[0007] A motion detection method based on mobile AR includes the following steps:

[0008] Obtaining the distance between the user and the virtual character during shooting, and adjusting the distance between the user and the virtual character so that the distance is within a set distance range;

[0009] Obtaining the angle between the user and the virtual character during shooting, and adjusting the angle between the user and the virtual character so that the angle is within a set angle range;

[0010] Obtaining the angle between the orientation of the virtual character and the camera during shooting, and adjusting the angle between the orientation of the virtual character and the camera so that the angle is within a set angle range;

[0011] When the distance is within the set distance range, the angle is within the set angle range, and the included angle is within the set included angle range, the state machine is used to detect the current action of the virtual character to determine whether the virtual character is performing a specific action.

[0012] Preferably, obtaining the angle between the user and the virtual character during shooting includes:

[0013] Through ray detection, the angle between the user and the virtual character during shooting is obtained.

[0014] Preferably, the method further comprises: when it is determined that the virtual character is not performing a specific action, controlling the action of the virtual character through a state machine.

[0015] Preferably, controlling the virtual character's actions through the state machine includes: using the state machine's Actor as the base class of the character script to carry the character's basic attributes; and using the state machine's base class ActorState to create the character's state.

[0016] Preferably, the controlling of the virtual character's actions through a state machine includes: decomposing the preset animation clips of the virtual character to obtain a first preset trajectory, a second preset trajectory, and a third preset trajectory, calculating parameter weights of the first preset trajectory, the second preset trajectory, and the third preset trajectory, fusing the first preset trajectory, the second preset trajectory, and the third preset trajectory according to the parameter weights to obtain a fused trajectory, and controlling the virtual character to act according to the fused trajectory.

[0017] Preferably, obtaining the angle between the user and the virtual character during shooting includes: emitting a first unity ray through the center of the camera to the virtual character, emitting a second unity ray through the center of the camera to the center direction of the user experience position, and using the angle between the first unity ray and the second unity ray as the angle between the user and the virtual character.

[0018] A motion detection system based on mobile AR, comprising:

[0019] a distance acquisition unit, configured to acquire the distance between the user and the virtual character during shooting, and adjust the distance between the user and the virtual character so that the distance is within a set distance range;

[0020] An angle acquisition unit, used to acquire the angle between the user and the virtual character during shooting, and adjust the angle between the user and the virtual character so that the angle is within a set angle range;

[0021] An angle obtaining unit, used to obtain the angle between the orientation of the virtual character and the camera during shooting, and adjust the angle between the orientation of the virtual character and the camera so that the angle is within a set angle range;

[0022] The action detection unit is used to detect the current action of the virtual character through a state machine when the distance is within a set distance range, the angle is within a set angle range, and the included angle is within a set included angle range, so as to determine whether the virtual character is performing a specific action.

[0023] Preferably, the angle acquisition unit is used to acquire the angle between the user and the virtual character during shooting through ray detection.

[0024] An electronic device comprises: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the mobile AR-based motion detection method.

[0025] A computer storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, it implements the motion detection method based on mobile AR.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention obtains the distance between the user and the virtual character during shooting, and adjusts the distance between the user and the virtual character so that the distance is within a set distance range; obtains the angle between the user and the virtual character during shooting, and adjusts the angle between the user and the virtual character so that the angle is within a set angle range; when the distance is within the set distance range, the angle is within the set angle range, and the included angle is within the set included angle range, the current action of the virtual character is detected by a state machine to determine whether the virtual character is performing a specific action, and thus by performing program judgment during the shooting process to determine whether there is a specific character in the picture and the character's animation state, photo detection can be performed more easily to achieve action detection of the virtual character, which reduces the difficulty and cost of action recognition compared to action detection through picture recognition learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a method for motion detection based on mobile AR provided by one embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a motion detection system based on mobile AR provided by one embodiment of the present invention;

[0030] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments or technical features described below may be arbitrarily combined to form new embodiments.

[0032] Example 1:

[0033] Please refer to Figure 1-3 As shown, Figure 1 The present invention shows a method for detecting motion based on mobile AR, comprising the following steps:

[0034] Step S1: obtaining the distance between the user and the virtual character during shooting, and adjusting the distance between the user and the virtual character so that the distance is within a set distance range;

[0035] In this embodiment, the maximum and minimum limits of the distance between the user and the virtual character can be set to ensure the size of the character in the captured image. For example, if the distance between the user and the virtual character is too large, the character in the captured image will be too small.

[0036] When the size of the character in the picture meets the preset size, it is beneficial to the accuracy of the character action detection. For example, when the size of the character is relatively small, the accuracy of the character action detection will be reduced.

[0037] In this embodiment, a mobile terminal such as a mobile phone or tablet can be used as the experience terminal.

[0038] Step S2: obtaining the angle between the user and the virtual character during shooting, and adjusting the angle between the user and the virtual character so that the angle is within a set angle range;

[0039] Preferably, obtaining the angle between the user and the virtual character during shooting includes:

[0040] Through ray detection, the angle between the user and the virtual character during shooting is obtained.

[0041] In this embodiment, the angle between the current user and the virtual character is obtained so that the angle is within a set angle range, thereby making the character's proportion in the shooting picture reach a preset proportion.

[0042] When the character's proportion in the shooting frame reaches a preset proportion, it is beneficial to the accuracy of character action detection. For example, when the character's proportion is relatively small, the accuracy of character action detection will be reduced.

[0043] Preferably, obtaining the angle between the user and the virtual character during shooting includes: emitting a first unity ray through the center of the camera to the virtual character, emitting a second unity ray through the center of the camera to the center direction of the user experience position, and using the angle between the first unity ray and the second unity ray as the angle between the user and the virtual character.

[0044] In this embodiment, during the AR experience, the user is holding the phone and moving the experience. Therefore, the ray cannot be emitted to the user. Therefore, the ray is emitted in the middle direction of the user experience position as the second unity ray. The user experience position can be specifically set.

[0045] During game development, collision detection is a crucial task. For example, checking whether a bullet hits an enemy in a shooting game or whether a loot item is picked up in an RPG game. The most commonly used tool for collision detection is rays. The Unity 3D physics engine provides a ray class and related function interfaces. Unity rays are provided by the Unity 3D physics engine. Specifically, the first parameter of a unity ray is the ray's starting point, and the second parameter is the ray's direction. By controlling the ray's starting point and direction, a first unity ray and a second unity ray can be generated. The angle between the first and second unity rays can then be calculated, serving as the angle between the user and the virtual character.

[0046] Step S3: obtaining the angle between the orientation of the virtual character and the camera during shooting, and adjusting the angle between the orientation of the virtual character and the camera so that the angle is within a set angle range;

[0047] Specifically, the angle between the camera-to-avatar vector and the avatar's own heading vector is calculated.

[0048] Step S4: When the distance is within the set distance range, the angle is within the set angle range, and the included angle is within the set included angle range, the state machine is used to detect the current action of the virtual character to determine whether the virtual character is performing a specific action.

[0049] Preferably, the method further comprises: when it is determined that the virtual character is not performing a specific action, controlling the action of the virtual character through a state machine.

[0050] Specifically, controlling the virtual character's actions through the state machine includes: using the state machine's Actor as the base class of the character script to carry the character's basic attributes; and using the state machine's base class ActorState to create the character's state.

[0051] It should be noted that the state machine is composed of state registers and combinational logic circuits. It can transfer states according to pre-set states based on control signals. It is the control center that coordinates related signal actions and completes specific operations. It can also detect the current action status of the virtual character.

[0052] Preferably, when it is determined that the virtual character is not performing a specific action, the action of the virtual character is controlled by a state machine.

[0053] Specifically, the method of controlling the actions of the virtual character through the state machine is as follows:

[0054] Actor is used as the base class of the character script, which carries the basic properties of the character, including the character ID, movement speed, coordinates, etc., so that the character can be controlled by a state machine. Therefore, the character's properties also include the character's current state, all states, state type, etc., which are implemented through the state machine, so that the state can be changed to drive the performance.

[0055] The state machine base class ActorState is used to create many states, such as standby, movement, attack, skill release, etc. Specifically, the state machine types include: enter state, update state, exit state, etc., and the user's various operations are implemented through subclasses.

[0056] Preferably, the controlling of the virtual character's actions through a state machine includes: decomposing the preset animation clips of the virtual character to obtain a first preset trajectory, a second preset trajectory, and a third preset trajectory, calculating parameter weights of the first preset trajectory, the second preset trajectory, and the third preset trajectory, fusing the first preset trajectory, the second preset trajectory, and the third preset trajectory according to the parameter weights to obtain a fused trajectory, and controlling the virtual character to act according to the fused trajectory.

[0057] In some embodiments, the information processing system of the virtual character may include a network, a control terminal, a user terminal, and a server. Specifically, the control terminal and the user terminal establish communication through the network. For example, the control terminal and the user terminal can communicate in the same local area network (for example, a network environment of the same router, etc.). Furthermore, the control terminal can be connected to the network via a wired (for example, a network cable, etc.) or wireless (for example, a cloud server, etc.), and the user terminal can establish a communication connection with the network via a wired or wireless (for example, WIFI, etc.). In some embodiments, the user terminal can send animation parameters of the virtual character to the control terminal and the server.

[0058] Furthermore, the control terminal and the server can play the animation clips of the virtual character on the user terminal. Furthermore, the control terminal and the server can obtain the motion library information of the motion capture technology through the network and the user terminal.

[0059] In this embodiment, the first preset trajectory can be achieved by detecting the position of the character in the first preset area, that is, the position of the character that triggers the action as the first position, and calculating the first distance to complete the first preset trajectory.

[0060] The first position can be obtained by inputting the first position of the trigger action. As an example, the first position may include the position information of the virtual character when performing the trigger action, for example, the position space coordinate value X1Y1Z1. In some embodiments, the first preset area may include a preset distance range of the virtual character, for example, the preset distance range includes an interval range of 0 to a (preset distance) from the virtual character (position space coordinate value X1Y1Z1), and the preset area may include a position space coordinate area (X1±a)(Y1±a)(Z1±a).

[0061] Similarly, the second preset trajectory can be completed by detecting the position of the character in the second preset area, taking the end point of the first preset trajectory as the starting point of the second preset trajectory, and calculating the second distance of the second preset trajectory.

[0062] Similarly, the third preset track can be completed by detecting the position of the character in the third preset area, taking the end point of the second preset track as the starting point of the third preset track, and calculating the third distance of the third preset track.

[0063] Then, parameter weights of the first preset trajectory, the second preset trajectory, and the third preset trajectory are calculated, and the first preset trajectory, the second preset trajectory, and the third preset trajectory are fused according to the parameter weights to obtain a fused trajectory.

[0064] For example, when a virtual character performs a jump, the system obtains the character's jump position information and detects whether there are any obstacles within a preset area of ​​2 meters around the character. If there are obstacles (such as a stone or a bird), the system calculates the distance between the character's jump position and the stone, and the distance between the character and the bird. Furthermore, based on this distance comparison information and preset conditions, the system calculates the parameter weights of the first, second, and third preset trajectories to obtain a fused trajectory, which is then used to update the jump trajectory for the action.

[0065] In the above implementation process, by performing program judgment during the photo-taking process, specifically, judging the distance between the user and the virtual character during the photo-taking, judging the angle between the user and the virtual character during the photo-taking, and judging the animation of the virtual character, it is possible to determine whether there is a specific character in the picture and the animation status of the character. This makes it easier to perform photo detection and realize action detection of the virtual character. Compared with action detection through image recognition learning, this reduces the difficulty and cost of recognition.

[0066] Example 2:

[0067] Please refer to Figure 2 As shown, Figure 2 The present invention shows a motion detection system based on mobile AR, including:

[0068] a distance acquisition unit, configured to acquire the distance between the user and the virtual character during shooting, and adjust the distance between the user and the virtual character so that the distance is within a set distance range;

[0069] An angle acquisition unit, used to acquire the angle between the user and the virtual character during shooting, and adjust the angle between the user and the virtual character so that the angle is within a set angle range;

[0070] An angle obtaining unit, used to obtain the angle between the orientation of the virtual character and the camera during shooting, and adjust the angle between the orientation of the virtual character and the camera so that the angle is within a set angle range;

[0071] The action detection unit is used to detect the current action of the virtual character through a state machine when the distance is within a set distance range, the angle is within a set angle range, and the included angle is within a set included angle range, so as to determine whether the virtual character is performing a specific action.

[0072] Preferably, the angle acquisition unit is used to acquire the angle between the user and the virtual character during shooting through ray detection.

[0073] Example 3:

[0074] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 The schematic diagram shown is used to describe an electronic device 100 for implementing the method of the present invention in an embodiment of the application.

[0075] As shown in FIG3 , an electronic device 100 includes one or more processors 102 and one or more storage devices 104. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Figure 3 The components and structure of the electronic device 100 shown are only exemplary and not restrictive. The electronic device may have Figure 3 Some of the components shown may also have Figure 3 Other components and structures not shown.

[0076] The processor 102 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0077] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the functions (implemented by the processor) in the embodiments of the present application described below and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the application.

[0078] The present invention also provides a computer storage medium having a computer program stored thereon. If the method of the present invention is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in the computer storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer storage medium, and when the computer program is executed by a processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer storage media does not include electric carrier signals and telecommunication signals.

[0079] Those skilled in the art can make various other corresponding changes and deformations based on the technical solutions and concepts described above, and all of these changes and deformations should fall within the scope of protection of the claims of the present invention.

Claims

1. A motion detection method based on mobile AR, characterized in that: The following steps are involved: Obtaining the distance between the user and the virtual character during filming, and adjusting the distance between the user and the virtual character so that the distance is within a set distance range, so that the size of the character in the picture meets a preset size; Obtaining the angle between the user and the virtual character during filming, and adjusting the angle between the user and the virtual character so that the angle is within a set angle range, thereby making the character's proportion in the filming image reach a preset proportion; Obtaining the angle between the orientation of the virtual character and the camera during shooting, and adjusting the angle between the orientation of the virtual character and the camera so that the angle is within a set angle range; When the distance is within the set distance range, the angle is within the set angle range, and the included angle is within the set included angle range, the state machine is used to detect the current action of the virtual character, thereby determining whether the virtual character is performing a specific action; When it is determined that the virtual character is not performing a specific action, controlling the action of the virtual character through a state machine; The method of controlling the action of a virtual character by a state machine includes: decomposing a preset animation clip of the virtual character to obtain a first preset trajectory, a second preset trajectory, and a third preset trajectory, calculating parameter weights of the first preset trajectory, the second preset trajectory, and the third preset trajectory, fusing the first preset trajectory, the second preset trajectory, and the third preset trajectory according to the parameter weights to obtain a fused trajectory, and controlling the virtual character to move according to the fused trajectory; determining whether there is a specific character in the picture and the animation status of the character by performing a program judgment during the photo shooting process; and obtaining the angle between the user and the virtual character during the photo shooting, including: emitting a first unity ray to the virtual character through the center of the camera, emitting a second unity ray through the center of the camera to the center of the user experience position, and using the angle between the first unity ray and the second unity ray as the angle between the user and the virtual character.

2. The method for detecting motion based on mobile AR according to claim 1, characterized in that: The method of controlling the virtual character's actions through the state machine includes: using the state machine's Actor as the base class of the character script to carry the character's basic attributes; and using the state machine's base class ActorState to create the character's state.

3. A motion detection system based on mobile AR, characterized in that: include: a distance acquisition unit, configured to acquire the distance between the user and the virtual character during filming, and adjust the distance between the user and the virtual character so that the distance is within a set distance range, thereby ensuring that the size of the character within the frame meets a preset size; An angle acquisition unit, used to acquire the angle between the user and the virtual character during shooting, and adjust the angle between the user and the virtual character so that the angle is within a set angle range, so that the proportion of the character shooting screen reaches a preset proportion; An angle obtaining unit, used to obtain the angle between the orientation of the virtual character and the camera during shooting, and adjust the angle between the orientation of the virtual character and the camera so that the angle is within a set angle range; an action detection unit, configured to detect the current action of the virtual character through a state machine when the distance is within a set distance range, the angle is within a set angle range, and the included angle is within a set included angle range, thereby determining whether the virtual character is performing a specific action; When it is determined that the virtual character is not performing a specific action, controlling the action of the virtual character through a state machine; The method of controlling the action of a virtual character by a state machine includes: decomposing a preset animation clip of the virtual character to obtain a first preset trajectory, a second preset trajectory, and a third preset trajectory, calculating parameter weights of the first preset trajectory, the second preset trajectory, and the third preset trajectory, fusing the first preset trajectory, the second preset trajectory, and the third preset trajectory according to the parameter weights to obtain a fused trajectory, and controlling the virtual character to move according to the fused trajectory; determining whether there is a specific character in the picture and the animation status of the character by performing a program judgment during the photo shooting process; and obtaining the angle between the user and the virtual character during the photo shooting, including: emitting a first unity ray to the virtual character through the center of the camera, emitting a second unity ray through the center of the camera to the center of the user experience position, and using the angle between the first unity ray and the second unity ray as the angle between the user and the virtual character.

4. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the mobile AR-based motion detection method described in any one of claims 1-2.

5. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the mobile AR-based motion detection method according to any one of claims 1 to 2.

Images