Virtual simulation interactive image recognition game monitoring method

Abstract

The invention provides a virtual simulation interactive image recognition game monitoring method, which comprises the following steps of: S1, capturing natural behaviors of players in an interactive game through high-resolution camera equipment, and generating video data representing game action moments in the interactive game of the players so as to provide input data with higher quality; s2, performing real-time analysis and processing on natural behaviors of players through frame processing, binaryzation and a deep learning target detection algorithm by utilizing the high-resolution video data acquired in the step S1; and S3, recognizing and responding to natural behaviors of the player, and mapping the behavior actions to the virtual characters in the game. According to the method, natural behaviors of players are accurately captured and analyzed in real time in combination with the Kinect camera, and shadow areas in the images are identified, so that shadow parts in the images in high-resolution video data are more accurately analyzed and identified, and richer information is provided for subsequent image processing and analysis.

Description

Technical Field

[0001] The present invention relates to the field of virtual simulation technology, and in particular to a virtual simulation interactive image recognition game monitoring method. Background Art

[0002] Most existing motion capture analysis systems use ordinary cameras to capture human motion. The images collected are relatively blurry and cannot process ghost images, black shadows, missing feet, etc., which easily affects the subsequent analysis and judgment results. In motion capture technology, the "black shadow" phenomenon is a key technical challenge. It refers to the data area that cannot be captured during the capture process due to sensor field of view limitations or technical limitations. This phenomenon is particularly prominent in gamification training because it directly affects the integrity of motion data and the accuracy of motion recognition.

[0003] For example, in a virtual reality football game, if the motion capture system cannot accurately capture every subtle movement of the player, such as a quick turn or subtle foot adjustment, then the virtual character in the game may not be able to reproduce these movements, resulting in a significant reduction in the player's in-game experience.

[0004] At this time, the incompleteness of this data not only reduces the accuracy of motion recognition and impairs the effect of game training, but may also increase the cost of the training system because additional sensors or more advanced capture technologies are needed to compensate for the missing data, resulting in a decrease in training effectiveness because players cannot learn all the necessary motion patterns from complete motion data. Summary of the invention

[0005] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a virtual simulation interactive image recognition game monitoring method to solve the problems raised in the above background technology.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solution: a virtual simulation interactive image recognition game monitoring method, comprising the steps of:

[0007] S1. Build a player motion capture strategy

[0008] Capturing the natural behavior of players in interactive games through high-resolution camera equipment, generating video data representing the game action moments in the player interactive game, so as to provide higher quality input data;

[0009] S2. Data Optimization

[0010] Using the high-resolution video data collected in step S1, the natural behavior of the players is analyzed and processed in real time through frame processing, binarization, and deep learning target detection algorithms to extract valid video data segments and prepare for the next step of behavior recognition analysis;

[0011] S3. Identify and respond to the player’s natural behavior, and map this behavior to the virtual character in the game.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] In order to solve the problem that most of the existing motion capture and analysis systems use ordinary cameras to capture human motion, the collected images are relatively blurry, and it is impossible to process phenomena such as ghosting, black shadows and missing feet, which easily affects the subsequent analysis and judgment results, the present invention combines the Kinect somatosensory camera, firstly, to achieve accurate capture and real-time analysis of the player's natural behavior, and secondly, by identifying the shadow area in the image, using deep learning targets to extract and analyze the player's movements, so that the present invention can more accurately analyze and identify the shadow part in the image in the high-resolution video data, and provide richer information for subsequent image processing and analysis, thereby providing rich and correct data support for game design and enhancing the interactivity between players and virtual environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The disclosure of the present invention is described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the accompanying drawings, the same reference numerals are used to refer to the same components. Among them:

[0015] Figure 1 It is a schematic diagram of the overall process of a virtual simulation interactive image recognition game monitoring method proposed in one embodiment of the present invention;

[0016] Figure 2 A schematic diagram of a player motion capture process proposed in an embodiment of the present invention;

[0017] Figure 3 A schematic diagram of a process for optimizing and processing high-resolution video data collected by a high-resolution camera device, as proposed in one embodiment of the present invention;

[0018] Figure 4 It is a schematic diagram of the process of the virtual simulation system proposed in one embodiment of the present invention to identify and respond to the player's actions;

[0019] Figure 5 It is a schematic diagram of a process of extracting and analyzing shadow parts in an acquired high-resolution video data image proposed in one embodiment of the present invention;

[0020] Figure 6 The figure is a flow chart of combining the player's actions captured by the Kinect motion sensing camera with the image analysis results of high-resolution video data through image processing technology proposed in one embodiment of the present invention. DETAILED DESCRIPTION

[0021] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific implementation modes and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or restriction to the technical solution of the present invention.

[0022] The present invention is further described in detail below with reference to the accompanying drawings, but is not intended to limit the present invention.

[0023] As an understanding of the technical concept and implementation principle of the present invention, the main purpose of a virtual simulation interactive image recognition game monitoring method proposed by the present invention is to solve the existing motion capture analysis system, which mostly uses ordinary cameras to capture human motion, and the collected images are relatively blurred, and it is impossible to process ghosting, black shadows, missing feet and other phenomena, which easily affects the subsequent analysis and judgment results. Therefore, the present invention combines the Kinect somatosensory camera, firstly, to achieve accurate capture and real-time analysis of the player's natural behavior, and secondly, by identifying the shadow area in the image, using deep learning targets to extract and analyze the player's actions, so that the present invention can more accurately analyze and identify the shadow part of the image in the high-resolution video data, and provide richer information for subsequent image processing and analysis, thereby providing rich and correct data support for game design and enhancing the interactivity between players and virtual environments.

[0024] When implementing it, Figure 1 As shown, a virtual simulation interactive image recognition game monitoring method is proposed, comprising the following steps:

[0025] S1. Build a player motion capture strategy

[0026] By capturing the natural behavior of players in interactive games through high-resolution cameras, video data representing the moments of game actions in the player's interactive game is generated to provide higher quality input data. It is understandable that when players play games, in order to more accurately capture and simulate the player's actions, they generally pay special attention to several key body parts, such as shoulders, hands, feet and knees. The actions of these parts are particularly important in the game, so they are identified with special markers, just like when doing a standard jump action, they will ensure that the actions of these parts are accurately captured, so that the characters in the game can imitate the player's real actions. By capturing the natural behavior of players in interactive games and generating high-quality video data, these data can be used as the basis for subsequent analysis and processing. Through high-resolution cameras, it can be ensured that the captured actions are rich in details and accurate, providing real player behavior input for the game.

[0027] like Figure 2-Figure 3 As shown, in one embodiment of the present invention, the natural behavior of the player in the interactive game is captured by marking the key action parts of the player in the game action process. The specific process is as follows:

[0028] S1-1. Based on the player's standing upright action, customize the body angle parameters and position parameters that characterize the player's standing upright, where the body angle parameters include: θ tor The tilt angle of the player's torso when standing upright, used to indicate whether the player's body is upright, θ nec The angle between the player's neck and torso when standing upright, indicating whether the player's head is in a straight line with the torso. kne The bending angle of the knee when the player stands straight, indicating whether the player's body is fully straight; the position parameter is the position coordinate A of the player's virtual character in the three-dimensional space in the game st , used to describe whether the player is within the predetermined standing area captured by the high-resolution camera device.

[0029] S1-2. Use gloves with built-in sensors to capture the player's hand and finger movements, which at least include the bending and rotation angles of the wrist, the overall posture angle of the hand, and the bending angle of each finger; synchronously use accelerometers, gyroscopes or magnetometer inertial sensors attached to the player's body to capture the player's head rotation α hea , waist bending β wai and leg movementγ hip Action, to fully capture the player's full body movements; head rotation action at least includes yaw α yaw , pitch α pit and tumbling α roo , the waist bending action at least includes the waist bending β ben and the rotation angle βrot , the leg movement includes at least the rotation angle γ of the hip joint hiprot , knee bending angle γ kneben , Ankle rotation angle γ ankrot .

[0030] Based on the above technical concept, it should be noted that in the process of action recognition, the player's body angle parameters, including the tilt or bending angle of the torso, neck and knees, are first captured by sensors to determine the player's basic standing posture. Then, the player's specific position in three-dimensional space is determined by position parameters to ensure that the player is within the effective capture range of the camera device. Further, the action coordinate points of the shoulders, hands, feet and knees can be used to capture the player's limb movements in detail, including the up and down movement of the shoulders, the opening and closing of the hands, the lifting of the feet and the bending of the knees. The changes in these coordinate points can reflect the dynamic process of the player when performing a specific action. By comprehensively analyzing these parameters, the system can accurately identify and respond to the player's natural behavior, thereby realizing real-time mapping and simulation of actions in the game, so that the system can better understand and simulate the player's actions and provide a more realistic and interactive game experience.

[0031] In one embodiment of the present invention, by capturing the player's body movements, gestures and facial expressions in the game, rich data can be collected, and these data can comprehensively record the player's interaction and experience process. After marking the player's action information, the collected action information can be edited and enhanced through a video editing server. The edited and enhanced video can provide higher quality input data, which helps to improve the accuracy of image recognition and analysis later. That is, high-quality video input data can improve the performance of image processing algorithms, because for image processing, the acquired video input data provides more contextual information, such as target tracking or behavior analysis. It can be understood that the complete action information in the instantaneous video data captured during the player's game action process is the basis for advanced image processing and analysis. Without this data, the subsequent analysis will be incomplete due to the black shadow edge or shadow area of ​​the image.

[0032] Therefore, the present invention further proposes that the specific process of generating instantaneous video data during the player's game action is as follows:

[0033] S1-3, using Kinect body sensing camera as a high-resolution camera device, and based on the predetermined high-resolution camera device to capture the standing area, set and install multiple Kinect body sensing camera nodes, and determine the position coordinates P of each node i are all precisely set and connected to the game server. At the same time, the association parameter L between each Kinect camera node and the standing area is established. i And the connection status parameter Ci To ensure that the player's action data can be accurately captured and transmitted,

[0034] P i =x i ,y i ,z i , represents the position coordinates of the i-th Kinect camera node in the game virtual scene,

[0035] L i = {P i ∈A st |i=1,2,...,N}, represents the i-th Kinect camera node P i Whether it is located at the position coordinate A of the player's virtual character in the three-dimensional space in the game st In the area, the connection status parameter C i The connection status between the i-th Kinect camera node and the game server is described in binary format, where 1 indicates a successful connection and 0 indicates a failed connection.

[0036] S1-4. When the player enters the virtual scene of the game, the virtual simulation system identifies the player through the virtual identity identifier worn by the player. The signal emitted by the virtual identity identifier is sent to the game server through the communication protocol in the game to collect the action video data of the player in the game process: after the game server receives the virtual identity identification signal of the player, it activates the Kinect motion camera corresponding to the player's position. The Kinect motion camera starts to capture the player's virtual character action and transmits the action data to the game server in the form of a video stream. The game server performs preliminary processing on the received video stream, including format conversion and encoding optimization, to adapt to the video playback standard in the game, and stores the processed video data in the shared directory of the game server in MP4 format to ensure the compatibility and accessibility of the video data;

[0037] S1-5, the game server calls the built-in video editing module to edit the collected video stream, add special effects or scene conversion editing operations, and add highlights, background images and game logos in the virtual scene in the process to enhance the viewing experience of the video and the immersion of the game;

[0038] S1-6. The edited video data is stored in a dedicated storage area of ​​the game server using distributed storage technology, and each video file is ensured to be associated with the player's virtual identity and game timestamp;

[0039] S1-7, the game server assigns a unique identifier to each video data based on the combination of the game timestamp and the player's virtual identity, so as to provide an index path for the subsequent real-time analysis of the player's natural behavior, and the operation is as follows:

[0040] The tagging formula in the process of constructing the index of video data is: [M_{tag}=g(S_{id},T_{timestamp})](1), where M_{tag} is the unique tag assigned to the video data, and g is the tagging function, which is generated by combining the player's virtual identity signal S_{id} and the timestamp T_{timestamp};

[0041] On the basis of the marked video data, a multi-dimensional index structure is constructed to complete the index path construction. The constructed index formula is as follows: [I_{index}=h(C_{class},M_{tag},P_{location})](2), where I_{index} is the established index and h is the index function, which is a retrieval path jointly created by the classification C_{class} of the video data, the tag M_{tag} and the storage location P_{location} of the video data in the server shared directory;

[0042] S1-8, by checking the position coordinates A of the player's virtual character in the three-dimensional space st Whether it exceeds the predefined game area boundary, construct the judgment condition L that the player leaves the game area lea , so that the virtual simulation system resets all video data states related to the player and prepares to capture new game actions. The logical formula is as follows:

[0043]

[0044] In the formula, if the player's position coordinate x pla ,y pla or pla Out of the minimum value x of the game area min ,y min 、z min or the maximum value x max ,y max 、x max , then the judgment condition L lea =1, indicating that the player has left the game area or completed the game action. At this time, when the player completes the game action or leaves the game area, the server automatically resets all video data states related to the player, thereby clearing the temporarily cached data, releasing storage space, and preparing for capturing new game actions; otherwise, the judgment condition L lea =0, indicating that the player is still in the game area.

[0045] S2. Data Optimization

[0046] Using the high-resolution video data collected in step S1, the natural behavior of the player is analyzed and processed in real time through frame processing, binarization and deep learning target detection algorithm to extract valid video data segments and prepare for the next step of behavior recognition analysis.

[0047] like Figure 4 As shown, in one embodiment of the present invention, when the natural behavior of the player is analyzed and processed in real time, the specific process of frame processing is:

[0048] S2-11. Use the VideoCapture function of the video processing library OpenCV to open the high-resolution video data, and store the returned high-resolution video data capture object in the variable cap. Define the variable N=5, where N represents extracting 1 frame every 5 frames to set the frame extraction interval. Create a variable frame_count and initialize it to 0 to track the number of frames that have been read.

[0049] S2-12, enter the loop, set each iteration to try to read a frame from the variable cap, and call the cap.read() function in this loop to return two values: Boolean value ret and frame frame, where if ret is True, the frame is displayed, indicating that the frame variable contains valid image data and can be used for subsequent processing or display, and the frame_count variable increases by 1; if ret is False, it means that all frames of high-resolution video data have been read, and the loop will terminate;

[0050] S2-13. If frame_count is a multiple of N, it is considered that the interval point for extracting the next frame has been reached. At this time, the current frame will be displayed using the cv2.imshow function and wait for user input. If the user presses the q key, the loop will exit. When the loop ends, the cap.release() function is used to release the high-resolution video data capture object, and the cv2.destroyAllWindows() function is used to close all open windows, thereby reducing the amount of computation required to process the video.

[0051] It should be noted that after using the video processing library OpenCV to perform frame processing on the high-resolution video data, it is also necessary to use Gaussian filtering or median filtering to remove the image noise of the high-resolution video data, and use the Otsu method to determine the threshold to convert the high-resolution video data image into a binary image.

[0052] In one embodiment of the present invention, when the natural behavior of the player is analyzed and processed in real time, the specific process of deep learning target detection is as follows:

[0053] S2-21. Construct a training data set containing feature vector F to describe the game actions of the player's virtual character in the game:

[0054] F=[θ tor ,θ nec ,θ kne ,A st (x,y,z),α yaw ,α pit ,α roo ,β ben ,β rot ,γ hip rot ,γ kne ben ,

[0055] γ ankrot ](4);

[0056] S2-22. For each feature f in the training data set, calculate its average value μ f and standard deviation σ f , complete the standardization of the feature vector F, where the average value μf is calculated as: Where N is the number of samples in the training data set, f i is the value of feature f in the i-th sample; standard deviation σ f The calculation formula is:

[0057] S2-23. Construct a convolutional neural network architecture CNN including multiple convolutional layers, pooling layers, fully connected layers and classification layers to extract features from the input high-resolution video data and classify them. At the same time, input the standardized feature vector F into the convolutional neural network architecture CNN for forward propagation and calculate the output, where:

[0058] In the convolution layer, the convolution kernel is applied to the input image data I to perform a convolution operation: (I*K)(x, y) = Σ m Σ n I(m,n)K(xm,yn) (7), where I is the input image data after frame processing and binarization, and represents the high-resolution video data of the player's game action, K is the convolution kernel, * is the convolution operation, m and n are the sliding positions of the convolution kernel K on the input image data I, x and y are the coordinates of the position in the convolution kernel K on the input data I, I(m,n) is the pixel value of the input image data I at the position (m,n), K(xm,yn) is the weight value of the convolution kernel K at the offset position (xm,yn) relative to its center position (x,y);

[0059] In the pooling layer, use the formula P(i, j) = max m，n I(i·sm, j·sn) (8) performs a pooling operation to reduce the amount of calculation and prevent overfitting, where P is the feature map after pooling, I is the feature map of the input image data, and S is the step size;

[0060] In the fully connected layer, the activation function ReLU is used to map the features to the final classification result: Z = W*A+b, where Z is the output, W is the weight matrix, A is the input, and b is the bias;

[0061] In the classification layer, the Softmax function is used and the formula Convert the output of the convolutional neural network architecture CNN into a probability distribution, where Z is the output of the fully connected layer, Softmax(Z) i is the probability of the i-th category, It is the sum of all category score index conversions, which is used for normalization to ensure that the sum of the probabilities of all categories is 1;

[0062] S2-24. Label the head rotation, waist bending and leg movement in the captured high-resolution video data, use the labeled data set to train the behavior recognition model, and adjust the behavior recognition model parameters through the back propagation algorithm Where θ is the recognition model parameter, η is the learning rate, J(θ) is the loss function, and the trained model is applied to the captured high-resolution video data to recognize the player's behavior.

[0063] Based on the above technical concept, it should be noted that the specific process of the trained behavior recognition model to recognize the player's behavior is as follows: first, the pre-processed high-resolution video data is input into the trained behavior recognition model, and the input data at least includes the player's hand and finger movements, head rotation α hea , waist bending β wai and leg movementγ hip The action recognition model analyzes each frame or fragment of the input and outputs the behavior category probability. (11), a time series analysis is performed on the prediction results of the behavior recognition model that represent the player's actions in the game to determine the final behavior label, where P t is the probability of the player's behavior category occurring at time t in the game, and T is the size of the time window considered; finally, the recognition results including behavior labels, timestamps, and related video frames are associated with high-resolution video data and stored to achieve accurate recognition of the player's game action behaviors.

[0064] like Figure 5-Figure 6As shown, as an embodiment of the present invention, since the Kinect body sensing camera is used as a high-resolution camera device to capture human motion and obtain high-resolution video data, it is essentially through projecting a specific light pattern (usually an infrared light spot pattern) into the game scene where the player is located. The Kinect body sensing camera captures the reflection of these light patterns on the surface of the player, but due to the different depths of different positions on the surface of the object, the distortion of the light pattern will also be different. That is, due to the different reflection characteristics of different surfaces to light, the clarity of the edge of the shadow and the internal brightness of the high-resolution video data image will be affected. For example, the edge of the shadow produced by the smooth surface may be clearer, while the edge of the shadow may be blurred due to diffuse reflection on the rough surface. In order to identify the shadow part in the image (the shadow / black shadow phenomenon is particularly prominent in gamification training because it directly affects the integrity of the action data and the accuracy of action recognition), it is first necessary to build a multi-spectral reflection model, which can decompose the reflected light in the image into a direct reflection component and a scattered reflection component, and then identify the shadow area by calculating the separation matrix of the shadow component. The specific implementation nodes are:

[0065] When analyzing and processing the natural behavior of players in real time, before using the deep learning target detection algorithm to accurately identify the player's game action behavior, the high-resolution video data image that has been converted into a binary image is divided and segmented to eliminate and extract the shadow part in the image of the high-resolution video data, providing accurate visual information for the subsequent virtual character animation. The process is as follows:

[0066] First, the high-resolution video data image is converted from the RGB color space to the YCrCb color space using the following conversion formula to extract the shadow part: Y = 0.299R + 0.587G + 0.114B, Cr = 0.713 (RY), Cb = 0.564 (BY), where Y is the brightness of the high-resolution video data image, Cr and Cb are the chromaticity of the high-resolution video data image, which are used to describe the color information of the high-resolution video data image. Each pixel of the high-resolution video data image is traversed, and the above formula is applied to the RGB value of each pixel to calculate the corresponding YCrCb value, and the converted YCrCb image is compared with the original RGB image to ensure the correct conversion;

[0067] Secondly, the black shadow area including the shadow part in the high-resolution video data image is identified by analyzing the reflection component analysis method. The operation is as follows: construct a multispectral reflection model, R(λ,φ)=R d (λ,φ)+R S (λ,φ) (12), decomposing the reflected light in the high-resolution video data image into the direct reflection component R d (λ, φ) and the scattered reflection component RS (λ,φ), where the direct reflection component R d (λ, φ) is the reflected light directly related to the light source, which is directly reflected by the player's body to the Kinect somatosensory camera. The scattered reflection component R S (λ, φ) is the reflected light related to ambient light or scattered light, including at least the light reflected from the surrounding environment onto the player's body;

[0068] Construct the shadow separation matrix S dow , separate and identify shadow areas in high-resolution video data images, S dow =(A T WA) -1 A T W(LE) (13), where A is the design matrix including the direct reflection component R d (λ, φ) and the scattered reflection component R S (λ, φ)-related eigenvectors, including the player's body posture and surface characteristics, W is the weight matrix, which is used to adjust the influence of different factors on the shadow part, including the light intensity, the distance between the player's body and the Kinect body camera, and the intensity of the ambient light, L is the brightness matrix, which represents the brightness value of each pixel obtained from the multi-spectral reflectance model, including the spectral response of the Kinect body camera, and E is the ambient light matrix, which represents the influence of ambient light on the brightness of each pixel, including the lighting conditions of the surrounding environment where the game player is located;

[0069] Solve the shadow separation matrix S using weighted linear least squares dow , extract shadow areas in high-resolution video data images;

[0070] Again, the direct reflection component R d The changes in (λ, φ) are analyzed as a time series data to identify the trend of the player's body posture change, ΔR d (t) = R d (t)-R d (t-1)(14), where ΔR d (t) is the direct reflection component R of the player's body posture change at time t d The change in (λ,φ), R d (t) is the direct reflection component of the player's body posture change at time t. At the same time, the scattered reflection component R S (λ, φ) to detect the degree of interaction between the player and the environment. If the scattered reflection component R S (λ, φ) increases significantly in a certain area, indicating that the player has entered a new lighting environment or interacted with objects in the environment. Image processing technology is used to measure the scattered reflection component R S(λ,φ) increases the light intensity of the player's area to determine if the player enters a brighter or darker environment;

[0071] Finally, the player's movements captured by the Kinect motion sensing camera are combined with the results of high-resolution video data image analysis. Through image processing technology, the two are aligned in space to ensure that the pixels of the two represent the same physical location and are integrated into a unified data structure for complete data fusion.

[0072] S3. Identify and respond to the player's natural behavior, and map this behavior to the virtual character in the game. The specific process is: when the player's game action is identified, it is also necessary to use the inverse kinematics algorithm to map these actions to the virtual character in the game. The target posture of the player's game action is calculated according to the inverse kinematics algorithm. The target posture includes the character's joint angles and body part positions. Finally, the animation of the virtual character is updated according to the calculated target posture, so that the character can reflect the player's natural behavior in the game environment.

[0073] The technical scope of the present invention is not limited to the contents in the above description. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical idea of ​​the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A virtual simulation interactive image recognition game monitoring method, characterized in that: Includes steps: S1. Build a player motion capture strategy Capturing the natural behavior of players in interactive games through high-resolution camera equipment, generating video data representing the game action moments in the player interactive game, so as to provide higher quality input data; S2. Data Optimization Using the high-resolution video data collected in step S1, the natural behaviors occurring in the player interactive game are analyzed and processed in real time through frame processing, binarization, and deep learning target detection algorithms to extract valid video data segments and prepare for the next step of behavior recognition analysis; S3. Identify and respond to the player’s natural behavior, and map this behavior to the virtual character in the game.

2. The virtual simulation interactive image recognition game monitoring method according to claim 1, characterized in that: In step S1, the specific process of capturing the natural behavior of players in interactive games is as follows: S1-1. Based on the player's standing upright action, customize the body angle parameters and position parameters when the player stands upright, where: Body angle parameters include: θ tor The tilt angle of the player's torso when standing upright, used to indicate whether the player's body is upright, θ nec The angle between the player's neck and torso when standing upright, indicating whether the player's head is in a straight line with the torso. kne The bending angle of the knees when the player is standing straight, indicating whether the player's body is fully straight; The position parameter is the position coordinate A of the player's virtual character in the three-dimensional space in the game. st , used to describe whether the player is in the standing area captured by the predetermined high-resolution camera device; S1-2, using gloves with built-in sensors to capture the player's hand and finger movements, wherein the hand and finger movements at least include the bending and rotation angles of the wrist, the overall posture angle of the hand, and the bending angle of each finger; Synchronously use accelerometers, gyroscopes or magnetometers inertial sensors attached to the player's body to capture the player's head rotation α hea , waist bending β wai and leg movementγ hip Action, to achieve comprehensive capture of the player's full body movements; the head rotation action at least includes yaw α yaw , pitch α pit and tumbling α roo The waist bending action at least includes waist bending β ben and the rotation angle β rot , the leg movement action at least includes the rotation angle γ of the hip joint hiprot , knee bending angle γ kneben , Ankle rotation angle γ ankrot .

3. The virtual simulation interactive image recognition game monitoring method according to claim 1, characterized in that: The specific process of generating instantaneous video data of the player's game action is as follows: S1-3, using Kinect body sensing camera as a high-resolution camera device, and based on the predetermined high-resolution camera device to capture the standing area, set and install multiple Kinect body sensing camera nodes, and determine the position coordinates P of each node i are all precisely set and connected to the game server. At the same time, the association parameter L between each Kinect camera node and the standing area is established. i And the connection status parameter C i To ensure that the player's action data can be accurately captured and transmitted, P i =x i ,y i ,z i , represents the position coordinates of the i-th Kinect camera node in the game virtual scene, L i = {P i ∈A st |i=1,2,...,N}, represents the i-th Kinect camera node P i Whether it is located at the position coordinate A of the player's virtual character in the three-dimensional space in the game st In the area, the connection status parameter C i The connection status between the i-th Kinect camera node and the game server is described in binary format, where 1 indicates a successful connection and 0 indicates a failed connection. S1-4, when the player enters the virtual scene of the game, the virtual simulation system identifies the player through the virtual identity identifier worn by the player, and the signal emitted by the virtual identity identifier is sent to the game server through the communication protocol in the game to collect the action video data of the player in the game process: after the game server receives the virtual identity identification signal of the player, it activates the Kinect somatosensory camera corresponding to the player's position, and the Kinect somatosensory camera starts to capture the player's virtual character action, and transmits the action data to the game server in the form of a video stream. The game server performs preliminary processing on the received video stream, including format conversion and encoding optimization, to adapt to the video playback standard in the game, and stores the processed video data in the shared directory of the game server in MP4 format to ensure the compatibility and accessibility of the video data; S1-5, the game server calls the built-in video editing module to edit the collected video stream, add special effects or scene conversion editing operations, and add highlights, background images and game logos in the virtual scene in the process to enhance the viewing experience of the video and the immersion of the game; S1-6. The edited video data is stored in a dedicated storage area of ​​the game server using distributed storage technology, and each video file is ensured to be associated with the player's virtual identity and game timestamp; S1-7, the game server assigns a unique identifier to each video data based on the combination of the game timestamp and the player's virtual identity, so as to provide an index path for the subsequent real-time analysis of the player's natural behavior, and the operation is as follows: The tagging formula in the process of constructing the index of video data is: [M_{tag}=g(S_{id},T_{timestamp})], where M_{tag} is the unique tag assigned to the video data, and g is the tagging function, which is generated by combining the player's virtual identity signal S_{id} and the timestamp T_{timestamp}; On the basis of the marked video data, a multi-dimensional index structure is constructed to complete the index path construction. The constructed index formula is as follows: [I_{index}=h(C_{class},M_{tag},P_{location})], where I_{index} is the established index and h is the index function, which is a search path jointly created by the classification C_{class} of the video data, the tag M_{tag} and the storage location P_{location} of the video data in the server shared directory; S1-8, by checking the position coordinates A of the player's virtual character in the three-dimensional space st Whether it exceeds the predefined game area boundary, construct the judgment condition L that the player leaves the game area lea , so that the virtual simulation system resets all video data states related to the player and prepares to capture new game actions. The logical formula is as follows: 1,if(x pla <x min or x pla >x max )or 0, oth In the formula, if the player's position coordinate x pla ,y pla or pla Out of the minimum value x of the game area min ,y min 、z min or the maximum value x max ,y max 、x max , then the judgment condition L lea =1, indicating that the player has left the game area or completed the game action. At this time, when the player completes the game action or leaves the game area, the server automatically resets all video data states related to the player, thereby clearing the temporarily cached data, releasing storage space, and preparing for capturing new game actions; otherwise, the judgment condition L lea =0, indicating that the player is still in the game area.

4. The virtual simulation interactive image recognition game monitoring method according to claim 1 or 3, characterized in that: In step S2, when the player's natural behavior is analyzed and processed in real time, the specific process of frame processing is as follows: S2-11, using the VideoCapture function of the video processing library OpenCV to open the high-resolution video data, and storing the returned high-resolution video data capture object in the variable cap, defining a variable N=5, where N represents extracting 1 frame every 5 frames to set the frame extraction interval, creating a variable frame_count and initializing it to 0 to track the number of frames that have been read; S2-12, enter the loop, set each iteration to try to read a frame from the variable cap, and call the cap.read() function in this loop to return two values: Boolean value ret and frame frame, where if ret is True, the frame is displayed, indicating that the frame variable contains image data that effectively represents the high-resolution video data and can be used for subsequent processing or display, and the frame_count variable increases by 1; if ret is False, it means that all frames of the high-resolution video data have been read, and the loop will terminate; S2-13. If frame_count is a multiple of N, it is considered that the interval point for extracting the next frame has been reached. At this time, the current frame will be displayed using the cv2.imshow function, and wait for user input. If the user presses the q key, the loop will exit. When the loop ends, the cap.release() function will be used to release the high-resolution video data capture object, and the cv2.destroyAllWindows() function will be used to close all open windows, thereby reducing the amount of computation required to process the video.

5. The virtual simulation interactive image recognition game monitoring method according to claim 4, characterized in that: After using the video processing library OpenCV to perform frame processing on the high-resolution video data, it is also necessary to use Gaussian filtering or median filtering to remove image noise from the high-resolution video data, and use the Otsu method to determine the threshold to convert the high-resolution video data image into a binary image.

6. The virtual simulation interactive image recognition game monitoring method according to claim 1, characterized in that: In step S2, when analyzing and processing the player's natural behavior in real time, the specific process of deep learning target detection is as follows: S2-21. Construct a training data set containing feature vector F to describe the game actions of the player's virtual character in the game: F=[θ tor ,the nec ,the kne ,A st (x,y,z), a yaw ,a pit ,a roo ,b ben ,b rot ,c hiprot ,c kneben ,c ankrot ]; S2-22. For each feature f in the training data set, calculate its average value μ f and standard deviation σ f , complete the standardization of the feature vector F, where the average value μf is calculated as: Where N is the number of samples in the training data set, f i is the value of feature f in the i-th sample; standard deviation σ f The calculation formula is: S2-23. Construct a convolutional neural network architecture CNN including multiple convolutional layers, pooling layers, fully connected layers and classification layers to extract features from the input high-resolution video data and perform classification. At the same time, input the standardized feature vector F into the convolutional neural network architecture CNN, perform forward propagation, and calculate the output, wherein, In the convolution layer, the convolution kernel is applied to the input image data I for convolution operation: (I*K)(x,y)=∑ m ∑nI(m,n)K(xm,yn), where I is the input image data after frame processing and binarization, and represents the high-resolution video data of the player's game action, K is the convolution kernel, * is the convolution operation, m and n are the sliding positions of the convolution kernel K on the input image data I, x and y are the coordinates of the position in the convolution kernel K on the input image data I, I(m,n) is the pixel value of the input image data I at the position (m,n), and K(xm,yn) is the weight value of the convolution kernel K at the offset position (xm,yn) relative to its center position (x,y); In the pooling layer, use the formula P(i, j) = max m，n I(i·sm, j·sn) performs pooling operation to reduce the amount of calculation and prevent overfitting. In the formula, P is the feature map after pooling, I is the feature map of the input image data, and S is the step size; In the fully connected layer, the activation function ReLU is used to map the features to the final classification result: Z = W*A+b, where Z is the output, W is the weight matrix, A is the input, and b is the bias; In the classification layer, the Softmax function is used and the formula The output of the convolutional neural network architecture CNN is converted into a probability distribution, where Z is the output of the fully connected layer, Softmax(Z)i is the probability of the i-th class, It is the sum of all category score index conversions, which is used for normalization to ensure that the sum of the probabilities of all categories is 1; S2-24. Label the head rotation, waist bending and leg movement in the captured high-resolution video data, use the labeled data set to train the behavior recognition model, and adjust the behavior recognition model parameters through the back propagation algorithm Where θ is the recognition model parameter, η is the learning rate, J(θ) is the loss function, and the trained model is applied to the captured high-resolution video data to recognize the player's behavior.

7. The virtual simulation interactive image recognition game monitoring method according to claim 6, characterized in that: The specific process of the trained behavior recognition model identifying the player's behavior is as follows: First, the pre-processed high-resolution video data is input into the trained behavior recognition model. The input data includes at least the player's hand and finger movements, head rotation α hea , waist bending β wai and leg movement γhip frames or segments of action; Secondly, the action recognition model analyzes each frame or clip of the input and outputs the action category probability; Again, by averaging the behavior category probabilities The prediction results of the behavior recognition model outputting the player's actions in the game are analyzed in time series to determine the final behavior label, where P t is the probability of the player's behavior category occurring at time t in the game, and T is the size of the time window considered; Finally, the recognition results including behavior labels, timestamps and related video frames are associated with high-resolution video data and stored to achieve accurate recognition of players' gaming actions.

8. The virtual simulation interactive image recognition game monitoring method according to claim 6, characterized in that: When analyzing and processing the natural behavior of players in real time, before using the deep learning target detection algorithm to accurately identify the player's game action behavior, it is also necessary to divide and segment the high-resolution video data image that has been converted into a binary image to eliminate and extract the shadow part in the image of the high-resolution video data, so as to provide accurate visual information for the subsequent virtual character animation. The process is as follows: First, the high-resolution video data image is converted from the RGB color space to the YCrCb color space using the following conversion formula to achieve the extraction of the shadow part: Y=0.299R+0.587G+0.114B, Cr=0.713(RY), Cb=0.564(BY), where Y is the brightness of the high-resolution video data image, Cr and Cb are the chromaticity of the high-resolution video data image, which are used to describe the color information of the high-resolution video data image. Each pixel of the high-resolution video data image is traversed, and the above formula is applied to the RGB value of each pixel to calculate the corresponding YCrCb value, and the converted YCrCb image is compared with the original RGB image to ensure the correct conversion; Secondly, the black shadow area including the shadow part in the high-resolution video data image is identified by analyzing the reflection component analysis method. The operation is as follows: construct a multispectral reflection model, R(λ, φ) = R d (λ,φ)+R s (λ, φ), decomposing the reflected light in the high-resolution video data image into the direct reflection component R d (λ, φ) and the scattered reflection component R S (λ, φ), where the direct reflection component R d (λ, φ) is the reflected light directly related to the light source, which is directly reflected by the player's body to the Kinect somatosensory camera. The scattered reflection component R s (λ, φ) is the reflected light related to the ambient light or scattered light, including at least the light reflected from the surrounding environment onto the player's body; Construct the shadow separation matrix S dow , separate and identify shadow areas in high-resolution video data images, S dow =(A T WA) -1 A T W(LE), where A is the design matrix including the direct reflection component R d (λ, φ) and the scattered reflection component R s (λ, φ)-related feature vectors, including the player's body posture and surface characteristics, W is a weight matrix used to adjust the influence of different factors on the shadow part, including the intensity of the light source, the distance between the player's body and the Kinect somatosensory camera, and the intensity of the ambient light, L is a brightness matrix, representing the brightness value of each pixel obtained from the multi-spectral reflectance model, including the spectral response of the Kinect somatosensory camera, and E is an ambient light matrix, representing the influence of ambient light on the brightness of each pixel, including the lighting conditions of the surrounding environment where the game player is located; Solve the shadow separation matrix S using weighted linear least squares dow , extract shadow areas in high-resolution video data images; Again, the direct reflection component R d The change of (λ, φ) is analyzed as a time series data to identify the trend of the player's body posture change, ΔRd(t) = Rd(t) - R d (t-1), where ΔR d (t) is the direct reflection component R of the player's body posture change at time t d The change of (λ, φ), R d (t) is the direct reflection component of the player's body posture change at time t. At the same time, the scattered reflection component R S (λ, φ) to detect the degree of interaction between the player and the environment. If the scattered reflection component R s (λ, φ) increases significantly in a certain area, indicating that the player has entered a new lighting environment or interacted with objects in the environment. Image processing technology is used to measure the scattered reflection component R s (λ, φ) increases the light intensity of the area where the player is located to determine whether the player enters a brighter or darker environment; Finally, the player's movements captured by the Kinect motion sensing camera are combined with the results of high-resolution video data image analysis. Through image processing technology, the two are aligned in space to ensure that the pixels of the two represent the same physical location and are integrated into a unified data structure for complete data fusion.

9. The virtual simulation interactive image recognition game monitoring method according to claim 1, characterized in that: When the player's game actions are identified, the inverse kinematics algorithm is needed to map these actions to the virtual character in the game, calculate the target posture of the player's game actions based on the inverse kinematics algorithm, and finally update the animation of the virtual character based on the calculated target posture, so that the character can reflect the player's natural behavior in the game environment.

Images