An image recognition-based game pausing method, system and storage medium

Abstract

The application discloses a game pause method and system based on image recognition and a storage medium, and comprises the following steps: acquiring multiple video images in a video stream; determining a pause picture in the multiple video images, and obtaining a training sample input quantity of model training; inputting the training sample input quantity into a pre-constructed pause recognition model for training; and when a player is disconnected or needs to call for a pause, recognizing and implementing the pause by using the trained model. The above scheme can recognize the video images recorded by the game, perform training without manual participation, determine correct pause operations in various scenes, and effectively improve the success rate of the pause.

Description

Technical Field

[0001] The present invention belongs to the field of computer software technology, and in particular relates to a game pause method, system and storage medium based on image recognition. Background Art

[0002] Current cloud gaming solutions often use fixed keystrokes. This approach is overly simplistic and ignores the player's state before the game is closed, leading to incorrect keystrokes and the inability to properly pause the game. For example, if the player has already entered the menu to pause the game, pressing a key at this point will result in an incorrect operation.

[0003] In terms of image recognition, if manual labeling is used, it will increase labor costs and affect commercial value.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention proposes a game pause method, system, and storage medium based on image recognition. This method addresses the high labor costs associated with manually adding tags for image recognition and reduces the error rate caused by manual keystrokes. This method, which can identify game pauses without relying on user interaction, helps cloud game players enhance their gaming experience.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] In one aspect, the present invention provides a game pausing method based on image recognition, the method comprising:

[0008] Get multiple frames of video images in the video stream;

[0009] Determine pause frames in multiple video frames to obtain training sample input for model training;

[0010] Input the training sample input into the pre-built pause recognition model for training;

[0011] Using the trained model, when a player is disconnected or needs to call a pause, the pause recognition model is used to identify and implement the pause.

[0012] Preferably, before obtaining multiple frames of video images in the video stream, the method further includes: obtaining the game video stream collected by the user through a video stream collection device and transmitted to the server by calling a service interface.

[0013] Preferably, the acquiring of multiple frames of video images in the video stream includes:

[0014] frame by frame reading real-time game video stream images, normalizing the images;

[0015] segmenting the normalized images to obtain a plurality of grids, and generating a detection box in each grid;

[0016] performing convolution layer processing on the detection box to obtain a detection box of a single frame image;

[0017] identifying the detection box to obtain a game pause control node;

[0018] storing the obtained game pause control node and the corresponding detection box.

[0019] Further, the frame by frame reading real-time video stream images, and normalizing the images include: obtaining a video stream address, and real-time de-framing; performing pause processing on the obtained picture to make the image meet the preset size requirement;

[0020] calculating the mean and variance of the image pixels;

[0021] subtracting the mean of each pixel value in the image from the mean and dividing by the variance to obtain the normalized image.

[0022] Preferably, the determination of the pause picture in the plurality of video images to obtain the training sample input quantity of the model training includes:

[0023] offline simulation of the game pause scene, taking the video frame, control information and time axis in the game as input, and taking the game pause control as output, taking the picture with consistent frame frequency characteristics and the picture with small frame frequency difference and other image feature data in the plurality of video images as the game pause control index, and determining the sample input quantity of the training model.

[0024] Further, the game pause index acquisition method is:

[0025] defining the control information execution time as a first preset time;

[0026] collecting each frame of video image on the time axis from the starting time point to the ending time point in the first preset time period, and judging whether the frame frequency of the video image in the first preset time period is less than a preset frame frequency threshold;

[0027] if less than the preset frame frequency threshold, collecting the low frequency characteristics of the current video image;

[0028] obtaining the picture with consistent low frequency characteristics and the picture with frame difference less than a preset threshold, and determining other image feature data in the plurality of video images.

[0029] Further, the determination of other image feature data in the plurality of video images includes:

[0030] Call OpenCV to locate the frames with consistent low-frequency features and frames with frame differences less than a preset threshold in multiple frames of video images;

[0031] Record the current frame image data and the previous frame image data near the low-frequency feature points respectively;

[0032] The current frame image data and the previous frame image data are preprocessed to collect other image feature data.

[0033] Furthermore, the preprocessing of the current frame image data and the previous frame image data to collect other image feature data includes:

[0034] The current frame image data and the previous frame image data near the low-frequency feature points of the current picture are subjected to translation alignment, and the corrected image data are subjected to difference processing and sum processing respectively to generate difference processed data and sum processed data.

[0035] In a second aspect, the present invention provides a game pause system based on image recognition, the system comprising:

[0036] An acquisition module is used to acquire multiple frames of video images in a video stream;

[0037] A determination module is used to determine pause images in multiple video frames to obtain training sample input for model training;

[0038] A training module, for inputting a training sample input into a pre-built pause recognition model for training;

[0039] The pause control module is used to use the trained model to identify and implement pause when the player is disconnected or needs to call pause through the pause recognition model.

[0040] A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the game pause method based on image recognition according to any one of claims 1 to 8 is implemented.

[0041] Compared with the closest prior art, the present invention has the following beneficial effects:

[0042] The solution of the present invention takes into account the particularity of the cloud gaming field and needs to consider the situation where the server-side game needs to be paused when the network is interrupted. A game pause method, system and storage medium based on image recognition are proposed, which include obtaining multiple frames of video images in a video stream; determining the pause screen in the multiple frames of video images to obtain the training sample input for model training; inputting the training sample input into a pre-built pause recognition model for training; using the trained model, when the player disconnects or needs to call a pause, the pause recognition model is used to identify and implement the pause. This avoids situations where the client disconnects from the network for various reasons while the server-side game continues, resulting in in-game tasks being "killed" and similar situations that exceed the client player's expectations.

[0043] This invention introduces trained image recognition to automatically identify recorded game video images, eliminating the need for manual training. It determines the correct pause operation for each scenario, improving the success rate of pauses. This effectively avoids manual pauses that result in incorrect key presses and inability to properly pause the game, thereby enhancing the gaming experience for game client players. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for the specific embodiments or the description of the prior art. Similar elements or parts are generally identified by similar reference numerals throughout the drawings. Elements or parts in the drawings are not necessarily drawn to scale.

[0045] Figure 1 This is a flow chart of a game pause method based on image recognition in a specific embodiment of the present invention;

[0046] Figure 2 This is a structural diagram of a game pause system based on image recognition in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following embodiments of the technical solution of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore only examples and are not intended to limit the scope of protection of the present invention.

[0048] It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0049] Example 1:

[0050] The specific embodiment of the present invention provides Figure 1 A game pausing method based on image recognition is shown, the method comprising:

[0051] S1 obtains multiple frames of video images in a video stream;

[0052] S2 determines the paused images in the multi-frame video images to obtain the training sample input for model training;

[0053] S3 inputs the training sample input into the pre-built pause recognition model for training;

[0054] S4 uses the trained model to identify and implement a pause when a player is disconnected or needs to call a pause.

[0055] Before executing step S1 to obtain multiple frames of video images in the video stream, the method further includes: obtaining the game video stream captured by the user through a video stream capture device and transmitted to the server by calling a service interface.

[0056] In step S1, obtaining multiple frames of video images in a video stream includes:

[0057] Reading real-time game video stream images frame by frame and performing normalization processing on the images;

[0058] The normalized image is segmented into multiple grids, and a detection box is generated in each grid;

[0059] Performing convolution layer processing on the detection frame to obtain a detection frame of a single frame image;

[0060] Identify the detection frame to obtain a game pause control node;

[0061] The obtained game pause control node and the corresponding detection frame are stored.

[0062] The process of reading the real-time video stream image frame by frame and normalizing the image includes: obtaining the video stream address and de-framing in real time; pausing the de-framing image to make the image meet the preset size requirement;

[0063] Calculate the mean and variance of image pixels;

[0064] Subtract the mean of each pixel in the image and divide it by the variance to obtain a normalized image.

[0065] In step S2, determining the paused images in the multi-frame video images and obtaining the training sample input for model training includes:

[0066] The game pause scene is simulated offline, video frames of a detection frame, control information and a time axis are taken as inputs, game pause control is taken as output, pictures with consistent frame frequency characteristics, pictures with small frame frequency differences and other image feature data in multiple video images are taken as game pause control indicators, and sample input of model training is determined.

[0067] The game pause indicator is obtained by:

[0068] The control information execution time is defined as a first preset time;

[0069] Each frame of video image on the time axis from a starting time point to an ending time point in the first preset time period in the game video history data is collected, and whether the frame frequency of the video image in the first preset time period is less than a preset frame frequency threshold is determined;

[0070] If the frame frequency is less than the preset frame frequency threshold, low frequency characteristics of the current video image are collected;

[0071] Pictures with consistent low frequency characteristics and pictures with frame differences less than a preset threshold are obtained, and other image feature data in multiple video images are determined.

[0072] The determination of the other image feature data in the multiple video images includes:

[0073] The opencv is called to locate the pictures with consistent low frequency characteristics and the pictures with frame differences less than the preset threshold in the multiple video images;

[0074] The current frame image data and the previous frame image data near the low frequency feature point are recorded respectively;

[0075] The current frame image data and the previous frame image data are preprocessed to collect the other image feature data.

[0076] Further, the preprocessing of the current frame image data and the previous frame image data to collect the other image feature data includes:

[0077] The current frame image data and the previous frame image data near the low frequency feature point of the current picture are translationally registered, and the corrected image data is respectively subjected to difference processing and addition processing to generate difference-processed data and addition-processed data.

[0078] In step S3, the minimization minE(w, v, θ, γ) of the global error function E is taken as the optimal solution, and the pause recognition model based on the BP neural network is constructed by the following formula:

[0079]

[0080] wherein x is a training sample, is the actual output of the network, y k (t) is the expected output of the network, w ij is the connection weight from input layer node i to hidden layer node j, i = 1, 2, ..., n, n is the number of input layer nodes, j = 1, 2, 3 ..., m; v jk is the connection weight from hidden layer node j to output layer node k, k = 1, 2, 3..., m; m is the number of hidden layer nodes, θ j is the threshold at hidden layer node j, γ t is the threshold at the output node t, t=1,2,3...,p; p is the number of nodes in the output layer, N1 is the number of neurons in the hidden layer, N is the number of neurons in the output layer, f(x) is the activation function, ε1 is the error preset threshold, R m ×p is an m-row and p-column matrix, R p×N1 is a matrix with p rows and N1 columns, R p is a matrix with p rows and 1 columns, R N It is a matrix with N rows and 1 column.

[0081] Example 2: Based on the same technical concept, the present invention provides a game pause system based on image recognition according to Example 1, such as Figure 2 As shown, the system includes:

[0082] An acquisition module is used to acquire multiple frames of video images in a video stream;

[0083] A determination module is used to determine pause images in multiple video frames to obtain training sample input for model training;

[0084] A training module, for inputting a training sample input into a pre-built pause recognition model for training;

[0085] The pause control module is used to use the trained model to identify and implement pause when the player is disconnected or needs to call pause through the pause recognition model.

[0086] Example 3: A computer-readable storage medium corresponding to Examples 1 and 2 is also provided; wherein the computer-readable storage medium includes a computer program stored thereon, and when the program is executed by the processor, the game pause method based on image recognition as described in any one of steps S1-S4 in Example 1 is implemented.

[0087] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0088] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0089] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit its scope of protection. Although the present application has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that after reading this application, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, and these changes, modifications or equivalent substitutions are all within the scope of the claims to be approved.

Claims

1. A game pause method based on image recognition, characterized in that: The method comprises: Get multiple frames of video images in the video stream; Determine pause frames in multiple video frames to obtain training sample input for model training; Input the training sample input into the pre-built pause recognition model for training; Using the trained model, when a player is disconnected or needs to call a pause, the pause recognition model is used to identify and implement the pause.

2. The method according to claim 1, wherein Before obtaining multiple frames of video images in the video stream, the method also includes: obtaining the game video stream collected by the user through a video stream collection device and transmitted to the server by calling a service interface.

3. The method according to claim 1, wherein The acquiring of multiple frames of video images in the video stream comprises: Reading real-time game video stream images frame by frame and performing normalization processing on the images; The normalized image is segmented into multiple grids, and a detection box is generated in each grid; Performing convolution layer processing on the detection frame to obtain a detection frame of a single frame image; Identify the detection frame to obtain a game pause control node; The obtained game pause control node and the corresponding detection frame are stored.

4. The method according to claim 3, wherein The process of reading the real-time video stream image frame by frame and normalizing the image includes: obtaining the video stream address and de-framing in real time; pausing the de-framing image so that the image meets the preset size requirement; Calculate the mean and variance of image pixels; Subtract the mean of each pixel in the image and divide it by the variance to obtain a normalized image.

5. The method according to claim 1, wherein Determining pause images in multiple frames of video images to obtain training sample input for model training includes: An offline simulation of the game pause scenario is performed, with the video frames of the detection frame, control information and timeline as input, and the game pause control as output. The pictures with consistent frame rate characteristics, pictures with small frame rate differences, and other image feature data in multi-frame video images are used as game pause control indicators to determine the sample input amount for model training.

6. The method according to claim 5, wherein The method for obtaining the game pause indicator is: defining a control information execution time as a first preset time; Collecting game video history data, for each frame of video image on a time axis from a start time point to an end time point within a first preset time period, and determining whether a frame rate of the video image within the first preset time period is less than a preset frame rate threshold; If it is less than the preset frame frequency threshold, collecting low-frequency features of the current video image; Obtain images with consistent low-frequency features and images with frame differences less than a preset threshold, and determine other image feature data in multiple frames of video images.

7. The method according to claim 6, wherein Determining other image feature data in the multi-frame video image includes: Call OpenCV to locate the frames with consistent low-frequency features and frames with frame differences less than a preset threshold in multiple frames of video images; Record the current frame image data and the previous frame image data near the low-frequency feature points respectively; The current frame image data and the previous frame image data are preprocessed to collect other image feature data.

8. The method according to claim 7, wherein The preprocessing of the current frame image data and the previous frame image data to collect other image feature data includes: The current frame image data and the previous frame image data near the low-frequency feature points of the current picture are subjected to translation alignment, and the corrected image data are subjected to difference processing and sum processing respectively to generate difference processed data and sum processed data.

9. A game pause system based on image recognition, characterized in that: The system comprises: An acquisition module is used to acquire multiple frames of video images in a video stream; A determination module is used to determine pause images in multiple video frames to obtain training sample input for model training; A training module, for inputting a training sample input into a pre-built pause recognition model for training; The pause control module is used to use the trained model to identify and implement pause when the player is disconnected or needs to call pause through the pause recognition model.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the game pause method based on image recognition described in any one of claims 1 to 8 is implemented.

Images