Method and system for animation in online game

Abstract

A method for dealing to a user during online gaming, the method comprising: receiving, from an end user device, a selection of a character from a pre-selected list of characters at the end user device by the user of the end user device; receiving, from an image capturing module, a plurality of images of a dealer / croupier; generating a three-dimensional (3D) animation of the dealer / croupier based on the received plurality of images of the dealer / croupier and the received selection of the character; and sending the generated 3D animation to the end user device for displaying to the user while the online game is in session.

Description

METHOD AND SYSTEM FOR ANIMATION IN ONLINE GAMETECHNICAL FIELD

[0001] The present disclosure generally relates to online gaming systems and methods. More particularly, embodiments of the present disclosure relate to animation of a dealer / croupier during online gaming.BACKGROUND

[0002] Currently available online casino games have limited digital effects or experiences for the users. While there are user interface overlays providing an online gaming experience similar to live casino table games, the users have limited or no option to view desired digital effects while the online game is in session. Existing solutions do not involve / allow capture of facial expressions and / or facial movements of the dealer / croupier in real-time for use during online gaming.

[0003] It is desirable to overcome or ameliorate one or more of the above-described difficulties, or to at least provide a useful alternative.SUMMARY OF THE INVENTION

[0004] According to an aspect, there is provided a method for dealing to a user during online gaming, the method comprising: receiving, from an end user device, a selection of a character from a pre-selected list of characters at the end user device, by the user of the end user device; receiving, from an image capturing module, a plurality of images of a dealer / croupier; generating a three-dimensional (3D) animation of the dealer / croupier based on the received plurality of images of the dealer / croupier and the received selection of the character; and sending the generated 3D animation to the end user device for displaying to the user while the online game is in session.

[0005] According to another aspect, there is provided a system for dealing to a user playing an online game on an end user device, the system comprising: an image capturing module configured to capture a plurality of images of a dealer / croupier; and a three-dimensional (3D) animation generation module communicatively connected to the end user device andSUBSTITUTE SHEET (RULE 26)the image capturing module, the 3D animation generation module configured to generate a 3D animation of the dealer / croupier based on at least the plurality of images captured by the image capturing module and a selection of a character by the user at the end user device from a pre-selected list while the online game is in session, wherein the generated 3D animation of the dealer / croupier is sent to the end user device for dealing to the user while the online game is in session.

[0006] The image capturing module may comprise a face capture camera and at least one motion capture camera.

[0007] The face capture camera may capture one or more images of a face of the dealer / croupier.

[0008] The at least one motion capture camera may capture at least one image of a body movement of the dealer / croupier.

[0009] The body movement may comprise at least a movement of a hand of the dealer / croupier.

[0010] The image capturing module may capture the plurality of images of the dealer / croupier while the online game is in session.

[0011] Generating the 3D animation may comprise masking out the hand of the dealer / croupier from the generated 3D animation.

[0012] The masking out of the hand may comprise using a plurality of layers of the at least one image of the body movement of the dealer / croupier.

[0013] Generating the 3D animation may further comprise using a machine learning model to process the plurality of images of the dealer / croupier.SUBSTITUTE SHEET (RULE 26)

[0014] The selection of the character from the pre-selected list of characters at the end user device by the user of the end user device may be performed while the online game is in session.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Preferred embodiments of the present disclosure are hereafter described, by way of non-limiting example only, with reference to the accompanying drawings, in which: a. Fig. 1 is a block diagram of a prior art online gaming system; b. Fig. 2 is a flow diagram of a method of online gaming according to an embodiment of the present disclosure; c. Fig. 3 is a block diagram of a system implementing the method of Fig. 2 according to an embodiment of the present disclosure; d. Fig. 4A, 4B and 4C exemplary screenshots of a user interface of a browser application running on the end user device providing the online gaming of Fig. 2 according to an embodiment of the present disclosure ; e. Fig. 5 is a block diagram of the image capturing module of Fig. 3 according to an embodiment of the present disclosure; f. Fig. 6 is detailed block diagram of a 3D animation generation module of Fig. 3 according to an embodiment of the present disclosure; g. Fig. 7A to 7C are in combination a detailed block diagram of the system of Fig. 3 according to an embodiment of the disclosure; h. Fig. 8A, 8B is a flow diagram of a method of online gaming according to an alternate embodiment of the present disclosure; i. Fig. 9 is a flow diagram of a method of capturing motion of a dealer / croupier according to an embodiment of the present disclosure;SUBSTITUTE SHEET (RULE 26)j. Fig. 10 is a flow diagram of a method of tracking face of a dealer / croupier according to an embodiment of the present disclosure; k. Fig. 11 is a flow diagram of a method of masking out unwanted parts from the 3D gaming environment according to an embodiment of the present disclosure; l. Fig. 12 is a flow diagram of a method of generating video stream output over webRTC according to another embodiment of the present disclosure; m. Fig. 13 shows an exemplary game setup for detecting gaming card(s) using machine learning according to an embodiment of the present disclosure; n. Fig. 14 shows a game setup for detecting gaming card(s) using a barcode scanner according to another embodiment of the present disclosure; and o. Fig. 15 shows a game setup for detecting roulette using in-built sensors according to another embodiment of the present disclosure.DETAILED DESCRIPTION

[0016] Fig. 1 is a block diagram of a prior art online gaming system 100 comprising an end user device 102, a video camera 104, a gaming table 106, a card detector 108, a studio computer 110, a game server 112 and a casino aggregator 114. The device 102 (such as, but not limited to, a personal computer, a laptop, a personal digital assistant, a smart phone, etc.) may host an online game in the form of an end user web browser. A web browser is an application for accessing websites. For the currently available online games, a non- customizable recording of an actual dealer / croupier dealing to the user (such as bets, sits, hits etc.) is sent to the end user device 102. A dealer / croupier is a person who is conducting a gambling game (such as Blackjack, Baccarat, Roulette or in game shows), including distribution of cards and pay out of wins after each hand or round. For card-based games, the dealer / croupier deals the cards used, collects bets from users (i.e. players who are playing the game) and monitors suspicious behavior to prevent cheating. Card(s) being dealt may be detected by a card detector 108 (such as a barcode scanner if the cards have a barcode, orSUBSTITUTE SHEET (RULE 26)any other standard card detection mechanisms). A video camera 104 of the system 100 records videos (consisting of sequences of images) of a gaming table 106 as well as of the detected card(s) being dealt by the dealer / croupier and sends a streamed data to a studio computer 110. The video camera 104 may be a professional level camera with fixed position and lens field of view (FOV) or lens focus. However, it will be appreciated that any standard video camera may be used.

[0017] The studio computer 110 sends the streamed data to the game server 112. The end user browser application running on the device 102 can access online game data via the game server 112. In other words, a user (not shown) can access the online game by opening the end user browser application on the device 102 and receive online game data from the game server 112 to which the device 102 is connected. In the prior art online gaming system 100, a recording of the actual dealer / croupier and a gaming environment including the gaming table and the cards is received by the user of the end user device 102.

[0018] One or more online game-related actions of the user (such as placing a bet) may be sent from the device 102 to the game server 112 in the form of data packets using a suitable communications protocol (e.g., WebSocket connection). The game server 112 may interact with a communication server (for e.g. a WebSocket server) which listens for any incoming connections (e.g., web socket connection), processes the connections and checks for a valid authentication token. The communication server may open up a bi-directional Transmission Control Protocol (TCP) connection between a browser running on the end user device 102 and the game server 112. This may allow the user of the end user device 102 to send data to the game server 112 and the server 112 to send data to the end user device 102, all in near real-time while the online game is in session. The game server 112 may process any incoming data from the end user device 102 via the communication server and then send any relevant online game data back to the end user device 102 via the communication server. The communications protocol (e.g., WebSocket connection) is used to send and receive data between the game server 112 and the browser on the end user device 102 in real-time. This data can include a time when the user sits at the gaming table, makes a bet, performs an action such as hit, stand and so on. The data from the server can include information aboutSUBSTITUTE SHEET (RULE 26)each user and where they are seated, each user's score, whose turn it is and the overall online game state / status.

[0019] The game server 112 may interact with a casino aggregator 114 to connect to one of a plurality of online casinos handled by the casino aggregator. The casino aggregator 114 may be an external casino aggregator. The game server 112 may communicate with the casino aggregator 114 to receive the details of the end user such as, but not limited to, username, identification number, display name, current credits. The details can vary between each external casino aggregator 114. The game server 112 may also update the external casino aggregator when the user makes a bet with a bet amount to subtract from the user's credits balance. The game server 112 may also communicate with the external casino aggregator 114 to update the user's credit balance with any amount the user wins or loses or to keep the credit balance the same if the user draws. The game server 112 and the casino aggregator 114 may communicate via a HTTP API call. In some embodiments, the game server 112 and the casino aggregator 114 may communicate via a more dynamic communications protocol such as a WebSocket connection.

[0020] Fig. 2 is a flow diagram of a method 200 of online gaming according to an embodiment of the present disclosure. When a user is playing an online game on the device 102, the user may have an option to select a character for a dealer / croupier to be dealt by. The user may select, while the online game is in session, a character from a pre-selected list of characters presented to the user on a user interface of the online game on the device 102. An online game is considered to be in session when the end user device 102 is connected to the game server 112 (via a suitable communication network) and there is exchange of game data and user actions between the end user device 102 and the game server 112. More specifically, an online game is said to be in session when the user of the end user device 102 is playing the online game, is seated at a gaming table and has a credit balance. Such a selection may occur at the start of the online game session. Further, such a selection of character may occur at any point in time during the online game session i.e., while the online game is in session. Accordingly, at step 202, a selection of a character from the pre-selected list by the user of the online game is received. A plurality of images of the dealer / croupier is received at step 204. At step 206, a three-dimensional (3D) animation of theSUBSTITUTE SHEET (RULE 26)dealer / croupier is generated based on the received selection of the character and the plurality of images of the dealer / croupier. Finally, at step 208, the generated 3D animation of the dealer / croupier is sent on the end user device 102 to deal to the user while the online game is in session.

[0021] Fig. 3 is a block diagram of a system 300 for implementing the method 200 according to an embodiment of the present disclosure. The system 300 comprises an image capturing module 302 and a 3D animation generation module 304. The image capturing module 302 captures a plurality of images of a dealer / croupier. The user selects a character from a preselected list via a user interface of the online game running on the device 102 while the online game is in session. The 3D animation generation module 304 receives the selection of the character from the device 102 (step 202 of Fig. 2 and shown by the arrow 306) while the online game is in session and the plurality of images of the dealer / croupier from the image capturing module 302 (step 204 of Fig. 2 and shown by the arrow 308) to generate a 3D animation of the dealer / croupier while the online game is in session (step 206 of Fig 2). The generated 3D animation of the dealer / croupier is then sent to the device 102 (step 208 of Fig. 2 and shown by arrow 310) such that the user sees the 3D animation of the dealer / croupier dealing to them during the online game session. In various embodiments, the steps 202, 204, 206 and 208 may be performed in real-time while the online game is in session. In various embodiments, steps 204 to 208 may be triggered upon occurrence of step 202. In some embodiments, the plurality of images of the dealer / croupier may be received continuously.

[0022] Fig. 4A, 4B and 4C are exemplary screenshots 400A, 400B and 400C respectively, of a user interface of a browser application running on the end user device providing the online gaming of Fig. 2, wherein a user can select and be dealt to by a 3D animation of the dealer / croupier while the online game is in session. Once the user starts playing the online game on the end user device 102 (for e.g. via a web browser application hosting the online game or via a dedicated software application running the online game), selects a gaming table and a credit balance, the online game is then considered to be in session and the user receives a user interface such as the one shown in Fig. 4A to select a character to be their dealer / croupier from a pre-selected list. Pre-selection of the list of characters may be basedSUBSTITUTE SHEET (RULE 26)on any one or more of the following: theme, type of game, tier. In an embodiment, one or more lists of pre-selected characters may be generated by the 3D animation generation module 304 and stored as batch(es) along with matching accessories and / or environments based on character themes, game themes etc. In another embodiment, one or more lists of pre-selected characters may be generated by the 3D animation generation module 304 based on one or more tiers of the game, each tier being based on a minimum bet size of the gaming table. Premium tables may only have a single user being dealt to by a single dealer / croupier. In an embodiment, there may be three tiers of the game, with the higher bet size of the table having more characters to select from and / or availability of certain types of characters in the pre-selected list that would not be available in the list(s) presented to users in the lower tiers. In some embodiments, the pre-selected list may be based on the type of game for e.g., Roulette and Baccarat have no limit to number of users so characters having their own themes may be provided in the list(s). Once the user selects a character, the 3D animation generation module 304 receives the selection of the character from the device 102 (step 202 of Fig. 2 and shown by the arrow 306) and a plurality of images of the actual dealer / croupier from the image capturing module 302 (shown by the arrow 308, step 204 of Fig. 2) to generate the 3D animation of the dealer / croupier while the online game is in session (step 206 of Fig 2). The generated 3D animation of the dealer / croupier is then sent to the device 102 (shown by arrow 310, step 208 of Fig. 2) such that the user sees the 3D animation of the dealer / croupier dealing to them during the online game as shown in Fig. 4B (for placing the bet), Fig. 4C (for deciding if the user wants to double, hit, stand or split the bet) etc.

[0023] Fig. 5 is a detailed block diagram of the image capturing module 302 of Fig. 3 according to an embodiment of the present disclosure. The image capturing module 302 comprises a camera module 504 including a face capture camera 508 and one or more motion capture cameras 510. A real person assigned to be a dealer / croupier for the online game wears a motion capture suit 506 in a studio. In a preferred embodiment, the motion capture suit 506 worn by the dealer / croupier may comprise an Xsens motion capture suit, a mocap beanie and a mocap foot wrap. Additional motion capture accessories (e.g. motion capture gloves) may be used for different gaming experiences. In some embodiments, each live gaming table may be assigned different motion capture accessories to reduce or avoid interference with the real gaming features e.g. cards, roulette wheel, gaming table etc. TheSUBSTITUTE SHEET (RULE 26)face capture camera 508 records images of the face of the dealer / croupier wearing the motion capture suit 506. The face capture camera 508 may track and map out the face of the dealer / croupier and based on a live footage recorded, the face capture camera 508 may create a mesh face to determine the type of facial expression made by the dealer / croupier. In an exemplary embodiment, the face capture camera 508 may use a software (such as AR Kit) to create the mesh face and determine the facial expression based on the mesh face. The motion capture camera 510 records one or more body movements of the dealer / croupier wearing the motion capture suit 506 (such as movement of the hand(s), turning of the body to the sides, walking etc.). The one or more motion capture cameras 510 may detect infrared (IR) signals and may track reflective sources. A plurality of motion capture cameras 510 may track a single reflective source to calculate position of the source in 3D space. In an exemplary embodiment, the motion capture suit has over 41 reflective sources. In an exemplary embodiment, the dealer / croupier puts on the motion capture suit 506 that has 41 markers placed thereon and the one or more motion capture cameras 510 track the dealer / croupier moving in 3D space. Finger articulation of the dealer / croupier may also be captured by the one or more motion capture cameras 510. In an embodiment, a single software may be used by the one or more motion capture cameras 510 to track the movements of the dealer / croupier as well as the finger articulation of the dealer / croupier. In another embodiment, dedicated software applications compatible with each other and running at the same time may each be used to track body movement of the dealer / croupier and finger articulation of the dealer / croupier (for e.g., OptiTrack Motive software for tracking body movement of the dealer / croupier and MANUS™ software for finger articulation of the dealer / croupier). One or more images captured by the face capture camera 508 are then fed into the video camera 104 and the studio computer 512, while one or more images captured by the motion capture camera 510 are then fed into a studio computer 512. The face capture camera 508 may be a smart phone or similar such as an iPhone™. In a preferred embodiment, around 100 images of the face of the dealer / croupier may be captured by the face capture camera 508. The motion capture camera 510 may be a Primexmotion capture camera. In an embodiment, the motion capture camera 510 may be a Primex13 camera that may be suitable for high-speed, precise tracking for medium-sized areas. In another embodiment, the motion capture camera 510 may be a Primex13W camera that maySUBSTITUTE SHEET (RULE 26)be suitable for high-speed, wall-to-wall tracking optimized for compact volumes. In one or more embodiments, the motion capture camera 510 may be used along with additional accessories such as a focusing tool (e.g. Primex13 focus tool for focusing lens on Primex13 camera, Primex13W focus tool for focusing lens on Primex13W camera) for improved motion capturing. The motion capture camera 510 may be calibrated to establish a 3D reference for the camera 510 using a calibration wand. The calibration wand may be a rod with markers set at fixed distance from each other. In an embodiment, a CW-500 calibration wand kit may be used to calibrate the motion capture camera 510. In some embodiments, a plurality of motion capture cameras 510 may be used. In a preferred embodiment, 12 to 17 motion capture cameras 510 may be used. The number of motion capture cameras 510 may depend on an accuracy of data (e.g. data relating to motion, orientation, etc.) obtained from the motion capture suit 506, the mocap beanie and / or mocap foot wrap etc.

[0024] Fig. 6 is a detailed block diagram of a 3D animation generation module 304 of Fig. 3 according to an embodiment of the present disclosure. The 3D animation generation module 304 comprises a game engine 602, a motion capture plugin 604, a 3D character animation module 606 and a 3D character composition module 608. The game engine 602 receives the output from the image capturing module 302 shown as 514, which comprises one or more images of the face of the dealer / croupier from the face capture camera 508 via the video camera 104 and converts data related to the one or more images of the face into facial expressions. In an exemplary embodiment, the game engine 602 may include a face capture plugin (such as an Augmented Reality Kit (ARKit) plugin) to convert data related to the one or more images of the face into facial expressions. In a preferred embodiment, around 100 facial expressions may be created by the game engine 602. The game engine 602 further receives one or more images of the body movement(s) of the dealer / croupier from the motion capture camera 510 via the studio computer 512, which is part of the output from the image capturing module 302.

[0025] Main footage from the video camera 104 may contain a plurality of images of the dealer / croupier, the gaming table and of the cards (from the card detector). The game engine 602 creates a plurality of layers for each of the images of facial expression(s), body movement(s), the gaming table as well as the card(s) along with a plurality of layers eachSUBSTITUTE SHEET (RULE 26)for a gaming background, selection of character, shadow catchers, user interface and visual effects (VFX) related to digital environments. It will be appreciated that each of these elements are their own separate layers that, when composited together, may create the illusion of a 3D character dealing cards on a real life gaming table in a virtual environment. The motion capture plugin 604 may work as a bridge to allow tracked data (for e.g., related to the characters, rigid bodies of the dealers / croupiers etc.), from the image capturing module 302 to the game engine 602. In an exemplary embodiment, the game engine 602 is Unreal Engine 5.

[0026] A 3D character animation module 606 is configured to run a machine learning model that receives the plurality of layers for each of the facial expression(s) image(s), body movement(s) image(s), the gaming table image(s) as well as the card image(s) from the game engine 602. The machine learning model then generates segmentation data of the dealer / croupier detected from the image(s), creates a mask and uses the mask to merge the captured frames and blend them with a static image of the gaming background without the dealer / croupier in it. In other words, the machine learning model masks out the images of the actual dealer / croupier to avoid or reduce the possibility of the dealer's / croupier's body parts (such as their hand, fingers etc.) appearing along with the animated character in the online game. In one or more embodiments, the machine learning model is a convolutional neural network with a plurality of convolutional layers. In some embodiments, the machine learning model may include a feature pyramid network configured to take a single-scale image of an arbitrary size as input and output proportionally sized feature maps at multiple levels for extracting features from images. In a preferred embodiment, a YOLOv8 model may be used as the machine learning model. Due to the dealer / croupier wearing a motion capture suit 506, the machine learning model may need to be trained with real life data (images / video) of dealers / croupiers wearing the motion capture suit in as many different poses, environments and lighting conditions as possible. This may allow the machine learning model to be faster and more accurate in detecting the dealer / croupier. Each image used for training may also need extra data such as a polygon mask of the dealer / croupier. The trained model may then be used to process video frames in real time. The machine learning model returns a polygon mask of the dealer / croupier, which may then be combined with the current video frame, and a previous image of the environment without theSUBSTITUTE SHEET (RULE 26)dealer / croupier in it. In an embodiment, the 3D character animation module 606 may create a skeleton with animations in it, which contain bone translation, rotation and / or scale all in real time to create a movable character. The resulting image of the movable character may then be streamed to the 3D character composition module 608. Preferably, the 3D character composition module 608 is Unreal Engine 5. It will be appreciated that every single operation would need to be as fast as possible (preferably within 40ms) to be able to keep up with a desired frame rate. In an embodiment, operations of the 3D animation generation module 304 may be processed on a graphics processing unit (GPU) while operations of the image capturing module 302 may be processed on a central processing unit (CPU), both of which may impact the processing abilities of the studio computer 512.

[0027] The 3D character composition module 608 then receives the masked images of the dealer / croupier and composes a 3D character based on the selection of the character by the user from the pre-selected list via the device 102. The 3D character composition module 608 may receive data from an environment camera setup configured to mimic position(s) of real cameras in real-time and to create an illusion of the animated character seeming to appear in the real world. The 3D character is composed as a live video footage including positioning ofthe 3D animated dealer / croupier (e.g., behind, on top ofthe real-world gaming table, cards etc.) and is sent to the online game running on the end user device 102 along with the environment, the user interface and any suitable visual effects.

[0028] Fig. 7A, 7B and 7C in combination are detailed block diagrams of the system of Fig. 3 according to an embodiment of the disclosure and will be explained in detail below.

[0029] Fig. 7A is a detailed block diagram of the image capturing module 302 of Fig. 3 and includes a face capture camera 704, three motion capture cameras 706a, 706b, 706c, a gaming table 710, a card detector 712, a video camera 714, and a studio computer 716. In some embodiments, a single motion capture camera 706a may be used. In some embodiments, the image capturing module 302 may include an additional bird's eye view camera 740 to detect what cards are being dealt to which user such that correct game calculations may be made and next available options may be given to the dealer / croupier and / or the user as will be explained in more detail below. For example, in Fig. 4C, the bird's eye view camera 740 would detect that the user has a 4 of hearts and a 7 of hearts, and sendsSUBSTITUTE SHEET (RULE 26)data related to this detection to the studio computer 716 via a direct connection 742. A motion capture suit 708 is worn by the dealer / croupier, the details of which are described hereinbefore. Working of the face capture camera 704 and of the motion capture cameras 706a, 706b, 706c are also described hereinbefore. One or more images captured by the face capture camera 704 is sent as streamed data to the studio computer 716 (shown by arrow 738). The video camera 714 records video of the gaming table 710 and the card detector 712 and sends the video data directly to the studio computer 716 (shown by arrow 736). The video camera 714 may be a professional level camera with fixed position and lens field of view or focus. The studio computer 716 may be a high-end gaming computer including GPU and graphics card(s) required to run the operations with less latency and higher quality visuals than standard computers. In an exemplary embodiment, the studio computer 716 may have 16 core processors, 4.5GHz base clock speed, 64GB Random Access Memory (RAM) and a GPU with base clock speed of 2.21GHz and 9728 cores (e.g. NVIDIA CUD A® cores).

[0030] Fig. 7B is a detailed block diagram of the 3D animation generation module 304 of Fig. 3 and includes a game engine 718, a 3D character animation module 720, a motion capture plugin 722 and a 3D character composition module 724. The studio computer 716 and the video camera 714 send the captured plurality of images of the dealer / croupier, gaming table 710 and the card(s) 712 to the game engine 718 (shown by arrow 732). In some embodiments where the image capturing module 302 includes a bird's eye view camera 740, data related to the image(s) obtained by the camera 740 sent to the studio computer 716 would then be sent to a card tracking application 744. The card tracking application 744 may use a machine learning model to identify and track each card on the gaming table using a custom-trained set of data. A more accurate view of the cards captured by the bird's eye view camera 740 without any bias that the video camera 714 may have, would allow the machine learning algorithm in the card tracking application 744 to recognize the number and suit of the cards. In a preferred embodiment, a YOUOv8 model may be used as the machine learning model in the card tracking application 744. The application 744 may take a live video stream and process each frame using a machine learning model to determine what each card is (type) and where they are on the table (position) and then sends that information (746) to the game server 728. For e.g., in Fig. 4C, the card tracking application 744 would determine that the user currently has a total of 11 of hearts and send that information to the game server 728SUBSTITUTE SHEET (RULE 26)for corresponding options to then be displayed in front of the animated dealer / croupier for the user to choose from. One or more images of the body movement of the dealer / croupier is sent to the 3D character animation module 720 via the motion capture plugin 722 and processed along with the image(s) comprising facial expressions of the dealer / croupier sent from the game engine 718 to the 3D character animation module 720, data from one or more environment cameras (setup to match real cameras) and custom designed anime characters. The game engine 718 creates a plurality of layers as described hereinbefore. The 3D character animation module 720 includes a machine learning module as described hereinbefore. The 3D character composition module 724 then composes a 3D character as described hereinbefore. In other words, a 3D animation of the dealer / croupier is generated by the 3D character composition module 724 by putting the 3D animated character in between layers of other visual graphics such as background environment, the live footage, visual effects and the user interface.

[0031] Fig. 7C is a detailed block diagram showing interaction between the elements of Fig. 7B and an end user device 726, a game server 728 and a casino aggregator 730. Live video footage of the 3D character animation of the dealer / croupier is sent from the 3D character composition module 724 as a streamed data (e.g., streamed footage via webRTC) to the end user device 726. In a preferred embodiment, video streaming technologies such as WebRTC may be used to stream data between the 3D character composition module 724 and the end user device 726. Gaming information such as, but not limited to, type, location and number of cards i.e. the type and number of cards at each user location on the gaming table (identified by the card tracking application 744) is sent to the game server 728. In an embodiment, the gaming information may be sent as encoded data packets. In a preferred embodiment, the gaming information may be sent as JavaScript Object Nation (JSON) encoded data packets via a persistent WebSocket connection. The game server 728 receives user actions from the end user device 726 and sends game data to the end user device. In a preferred embodiment, data between the game server 728 and the end user device 726 may be sent as JavaScript Object Nation (JSON) encoded data packets via a persistent WebSocket connection. A standard HTML or JavaScript web application or website such as a react application may run in the browser at the end user device 726. A video stream from the studio computer 716 may be sent to the browser at the end user device 726 via a WebRTC real-timeSUBSTITUTE SHEET (RULE 26)communication protocol. The web application displays the video / audio stream in the browser to the user.

[0032] The game server 728 may send and receive data to / from a casino aggregator 730 that connects the game server 728 to a plurality of external casinos and exchanges user provider information credits (adding or removing credits) and / or information about win, loss, bet etc. In a preferred embodiment, the game server 728 may be connected to a communication server (such as Node.js WebSocket server) as detailed above.

[0033] Fig. 8A, 8B is a flow diagram of a method 800 of online gaming according to an alternate embodiment of the present disclosure. The dealer / croupier may be wearing a motion capture suit 801. The motion capture suit may be custom-made with passive and / or active markers. In some embodiments, the motion capture suit may include a green overlay material that can be masked out using green screen masking technology. The dealer / croupier may additionally wear gloves with passive and / or active markers that can track each individual finger. In an embodiment, each individual finger may be tracked using OPTITRACK cameras. The studio environment may facilitate one or more types of game specific setups 802 as will be explained in more detail with reference to Figs. 13 to 16. The game setup may include games such as, but not limited to, Blackjack, Baccarat, Roulette.

[0034] A face capture camera 803 may be used to capture / track the dealer's / croupier's facial expressions and / or facial movements and / or position of the facial expressions and / or facial movements. In various embodiments, an off-the-shelf web camera or mobile phone device may be used as a face capture camera 803. A face tracking computer 804 may be set-up to be connected / connectable to the face capture camera 803 and configured to capture the dealer's / croupier's facial expressions via the connected face capture camera. One or more key points of the dealer's / croupier's face (e.g., points such as eyes, nose, mouth, eyebrows, cheeks etc.) may be translated and sent to one or more Unreal PC’s running Unreal App and then be translated to a virtual 3D character's face.

[0035] The studio may use one or more motion capture cameras 805 (805a, 805b, 805c) that may be placed around the environment to be able to accurately capture / track the motion tracking markers. The motion capture cameras 805 may be configured to detect markers i.e., passive and / or active sensors around the dealer's / croupier's motion capture suit. TheSUBSTITUTE SHEET (RULE 26)motion capture cameras may interface directly with suitable software (e.g., OptiTrack software) to translate the positions from physical world to virtual world. A motion capture computer 806 may be set up to receive the raw motion tracking data from the motion capture cameras (e.g., using OptiTrack Motive software). The raw data may be merged and translated into virtual coordinates and sent to one or more Unreal PCs. The raw data may include tracking for dealers / croupiers, cameras and any physical item (e.g., items around the gaming table / wheel, items used on the card dealing shoe etc.) which may need to be tracked in the environment.

[0036] The studio may use one or more high quality studio cameras 807 to capture the live (real-time) scene including the dealer / croupier and game specific items such as, but not limited to, tables, cards, roulette wheels. In various embodiments, a high quality studio camera may be a camera that is pointed towards the game and may be configurable to capture sharp details clearly within the range of assets (such as, but not limited to, the dealers / croupiers, shufflers, tables, cards, chips, shoes, discard racks and any other props pertaining to the respective game) within the game scene. A live feed of the game generated by the studio cameras may be received by the footage computer(s). The studio cameras may provide one or more views of the scene and may have tracking markers attached to match the virtual cameras with the physical cameras. In various embodiments, a virtual camera may be a camera that is created within the Unreal engine scene for mimicking / tracking the position, and / or angle of the camera in the real world. The virtual camera may be tied to the real life camera by means of motion capture rigid bodies and may be tethered by a live link connection that may be picked up by the Unreal Engine. Accordingly, adjustments made to the real life camera may also be made to the virtual camera. The virtual cameras may be calibrated to match the lens profile of the physical camera as a means to more closely align the virtual world with the physical world. In an embodiment, the studio may use a hardware device 808 (e.g., hardware ultimatte device) to mask / remove greenscreen backgrounds and characters which are not to be seen, while retaining the game specific details such as cards, tables, roulette wheels. The output of the studio cameras may be fed directly into the ultimatte device. In an embodiment, the ultimatte device may be configured to remove a specific shade of green. The ultimatte hardware may reduce the latency that software based keying and composition can introduce.SUBSTITUTE SHEET (RULE 26)

[0037] A masking computer 809 may be set up to receive the output of the ultimatte hardware device and perform further processing to clean up any artefacts and provide a seamless view between the physical environment and the virtual environment. In an embodiment, the masking computer 809 may be set up to receive the output of the studio cameras directly. The output of the masking computer may then be transmitted to one or more Unreal PC 810. Unreal PC 810 is a high end gaming computer setup to be able to run the Unreal Game executable at a high quality with a stable minimum of 30 frames per second output. The Unreal PC 810 may require a high-end graphics card to be able to process the 3D environment and 3D character. The Unreal PC 810 may be further configured to obtain output from the face tracking computer 804 and the motion capture computer 806. A game engine executable 811 may be set up on the Unreal PC 810. The game engine executable may be created with Unreal Engine and may be used to capture and merge some or all of the data from external sources (relating to the virtual environment created in Unreal Engine which transforms what is in the real environment to be the animated environment) to create a scene that blends the physical environment with the virtual environment and render the output in real time.

[0038] Face capture camera tracking data 812, originating from the face tracking computer 804 may be translated by the game engine executable 811 to the face of the virtual 3D character to provide facial movement of some or all parts of the face (such as mouth, eyes etc.) and to display the expressions of the dealer / croupier on the 3D model. A 3D character animation module 813 may be configured to combine dealer / croupier motion capture data 814 from the motion capture cameras 805 (via the motion capture computer 806, the Unreal PC 810, game engine executable 811 and optionally an OptiTrack Motive plugin 815) and dealer / croupier face capture camera tracking data 812 (via the face capture camera 803, face tracking computer 804, the Unreal PC 810, game engine executable 811) and translate the points (i.e., the markers being tracked on the face and / or body of the dealer / croupier or via the face and / or motion capture camera) to a corresponding 3D character point on the 3D model. Doing so may allow the 3D character to match the movements of the dealer / croupier and may show seamless interaction between the 3D world and the physical world. In an embodiment, the dealer's / croupier's motion capture data 814 may be streamed from the motion capture computer 806 to the Unreal PC executing the game engine executable. OneSUBSTITUTE SHEET (RULE 26)or more passive and / or active markers on the dealer's / croupier's suit may be directly translated to points on the 3D character's skeleton model to provide synced animation between the dealer / croupier and the 3D model. OptiTrack Motive plugin 815 is a plugin for Unreal Engine that may be used to translate the dealer's / croupier's tracking point data to the 3D character's skeleton.

[0039] Output from the masking computer 809 may be sent to the Unreal PC 810 and the game engine executable 811 may generate masked camera footage 816 that may be used in the final composition (in the 3D character composition module 818 as will be explained further below) to show the real world physical objects in the 3D scene. A 3D environment817 may be rendered inside the scene to provide a virtual world in which the 3D characters may be placed. The 3D environment may be designed to blend (including generating a virtual character based on the physical person within the created virtual environment) with the physical world and the 3D characters. A 3D character composition module 818 may blend the virtual world and the physical world together with the 3D character to provide a visually appealing output that the end user can see.

[0040] A WebRTC stream 819 may take the output of the 3D character composition module818 and stream the video footage using WebRTC technology to a streaming server 820. The streaming server 820 may take the output stream and broadcast it to users (potentially 1000s) seamlessly with as minimal latency as possible. Minimal latency may refer to the speed between occurrence of an action on the dealer's / croupier's end such as dealing a card and when that action may be seen at the end-user's end. In an embodiment, the minimal latency may be less than one second response time. In an embodiment, this streaming may be done using a plurality of servers that may be geolocated close to the end users. Eive video stream 821 may be sent via the streaming server using WebRTC protocol / technology (or similar real-time streaming protocol) to the end user's web browser application. The end user's device 822 may be any device configurable to run a web browser and capable of displaying video streams, such as but not limited to, a mobile phone, a tablet or a personal computer. The web application 823 may be made using any suitable standard web frameworks to provide the user interface and video stream to allow the user to interact with the online game . Game data may be sent between the web application 823 and the game server 825 via websockets 824 to provide instant feedback and interaction with the game. A game serverSUBSTITUTE SHEET (RULE 26)825 may handle the logic for the flow of the game, validate and / or act on the user actions sent from the web application. The game server 825 may also handle any game specific data from the studio e.g., which cards have been dealt, what number the ball has landed on in roulette etc. API rest calls 826 may be provided by an external casino aggregator 827 that the game server may communicate with, to get user information, add and / or subtract credits from the users balance when they make bets and / or win rounds. An external casino aggregator 827 may be a middleware between the game server, online casino and the end user. The aggregator 827 may provide a list of games and functionality to retrieve and / or adjust a user's credit balance. In some embodiments, game-specific data (e.g., the dealt card, the number the ball has landed on in roulette etc.) may be sent from the game specific setup 802 to the game server 825 in real time via websocket(s) 828.

[0041] Fig. 9 is a flow diagram of a method 900 of capturing motion of a dealer / croupier according to an embodiment of the present disclosure. In method 900, motion of a dealer / croupier wearing a motion capture suit 902 may be captured using one or more motion capture cameras 904. As described elsewhere in the description, the one or more motion capture cameras 904 may be placed around the gaming environment to be able to accurately track one or more motion tracking markers associated with the motion capture suit 902. In an embodiment, three motion capture cameras may be used. In various embodiments, three to twelve motion capture cameras may be used. The motion capture cameras may interface directly with suitable software (e.g., OptiTrack software) to translate the positions from physical world to virtual world. A motion capture computer 906 may be set up to receive the raw motion tracking data from the motion capture cameras (e.g., using OptiTrack Motive software). The raw data may be merged and translated into virtual coordinates. The raw data may include tracking for dealers / croupiers, cameras and any physical item that may need to be tracked in the environment. In an embodiment, the motion capture computer 906 may use an OptiTrack Motion capture software 908 for processing the raw data. Motion of the dealer / croupier may comprise movements of the body and / or parts of the body.

[0042] Fig. 10 is a flow diagram of a method 1000 of tracking face of a dealer / croupier according to an embodiment of the present disclosure. One or more face capture camera(s) 1004 may track face of the dealer / croupier 1002 to capture the facial expressions and movements of the dealer / croupier (as described elsewhere in the disclosure). Output of theSUBSTITUTE SHEET (RULE 26)face capture camera 1004 may then be sent to a face tracking computer 1006 that may use a face tracking software 1008 to translate the actual or physical face and related movements and / or expressions into the virtual 3D character’s face as described elsewhere in the disclosure.

[0043] Fig. 11 is a flow diagram of a method 1100 of masking out unwanted parts from the 3D gaming environment according to an embodiment of the present disclosure. Machine learning models (e.g., Y OLOv8) configured for object detection may be used 1102 to detect various gaming-related elements such as face of card, dealer' s / croupier's hands, fingers, game table etc. Output of the machine learning model may be the detected gaming element boundaries that is then input to the studio camera(s) 1104. The studio camera(s) may be configured to mask / remove unwanted elements and / or parts such as greenscreen backgrounds, characters that are not to be seen, dealer's / croupier's hands etc., while retaining the game specific details such as cards, tables, roulette wheels. In various embodiments, machine learning model(s) may be used to detect the dealer's / croupier's hand and / or fmger(s) that may not be removable by using motion capture software such as masking software used in an ultimatte hardware device. The output of the studio cameras may be fed directly into an ultimatte device 1106. In an embodiment, the ultimatte device 1106 may be configured to remove a specific shade of green. The ultimatte hardware may reduce the latency that software based keying and composition can introduce. The ultimatte device may mask out a chosen color (e.g., chosen by an operator, end user etc.). In an example where a green screen is used, the receiving computer may get two feeds coming in - one feed being the normal footage with all the green turned to black, removed from the scene, and the other feed being a key where the chosen color (e.g., green) shows up as white, while everything else that is not green is shown as black. These images may then be sent into the Unreal PC to be manipulated into the final composition image. In an example where the dealer / croupier is wearing a green suit, the above process may also make the dealer / croupier invisible. With respect to alpha and masking, the material editor in the Unreal engine may show anything that is white and hide anything that is black. Accordingly, with the key image received from the ultimatte, the key may be inverted so that the green screen is made to be black and the gaming table and other elements may be made white. Doing so may render the background of the footage to be transparent and may allow for the virtual background to appear behindSUBSTITUTE SHEET (RULE 26)the footage. A masking computer 1108 may be set up to receive the output of the ultimatte hardware device 1106 and perform further processing to clean up any artefacts and provide a seamless view between the physical environment and the virtual environment. The output of the masking computer 1108 may then be transmitted to one or more Unreal PC as discussed elsewhere.

[0044] Fig. 12 is a flow diagram of a method 1200 of generating video stream output over webRTC according to another embodiment of the present disclosure. The Unreal PC 1202 may run a game engine executable 1204 whose output is fed into a 3D game environment 1206, video composition module 1208 and a 3D character animation module 1214. The 3D character animation module 1214 may also receive input in the form of face tracking data 1210 and motion capture data 1212 (optionally via Optitrack motion capture plugin 1216). The video composition module 1208 receives input from the 3D character animation module 1214 as well as masked video footage 1218 and the 3D game environment 1206 to then blend the virtual world and the physical world together with the 3D character to provide a visually appealing output which the end user can see i.e., the video composition module creates a 3D character composition. The generated video stream may then be sent over webRTC 1220 to a streaming server as described elsewhere in the disclosure. In an embodiment, the masked footage may be generated via a module configured to remove the physical human and any other objects from the raw video footage such as with any combination of the following modules, both external to the game engine and inside the game engine itself: keying out the dealer / croupier / environment using green suit / greenscreen (e.g., on a hardware Ultimatte device or a software based solution) and removing any additional artifacts using machine learning to detect and replace undesired objects and / or dealers / croupiers from the scene.

[0045] Fig. 13 shows an exemplary game setup 1300 for detecting gaming card(s) using machine learning according to an embodiment of the present disclosure. Card(s) being dealt on a gaming table 1304 may be detected by a card detector 1302 (such as a barcode scanner if the cards have a barcode, or any other standard card detection mechanisms). A birds eye view camera 1306 (e.g., an overhead camera) may capture images of the card detector and / or the gaming card(s) and / or the gaming table 1304. Output of the bird's eye view camera may be sent to a computer 1308 where a suitable machine learning model may be used to detectSUBSTITUTE SHEET (RULE 26)face of card(s) 1310. Output of the machine learning model 1310 (e.g., which cards have been dealt) may then be broadcast to a game server 1312 for further processing as discussed elsewhere in the disclosure.

[0046] Fig. 14 shows another game setup 1400 for detecting gaming cards using a barcode scanner according to an embodiment of the present disclosure. Card(s) being dealt on a gaming table 1404 may be detected by a barcode scanner 1406 that performs card detection 1402 if the cards have a barcode, using barcodes printed on playing cards and a barcode scanner built into the gaming table 1404 to scan the card as it is drawn from the deck) . Output of the barcode scanner 1406 may be sent to a computer 1408 (e.g., Raspberry Pi or any computing device that can process small amounts of data) and then be sent as websocket data to a game server 1410 for further processing as discussed elsewhere in the disclosure. In an embodiment, each card drawn may be broadcast to the game server 1410. In an embodiment, card tracking may be performed using artificial intelligence (Al) for reading cards more accurately (e.g., for games such as Baccarat).

[0047] Fig. 15 shows a game setup 1500 for detecting roulette using in-built sensors according to an embodiment of the present disclosure. A roulette wheel 1504 may have a plurality of sensors 1506 in-built to detect one or more of the ball position 1502, the wheel rotation, landing number. The in-built sensors 1506 may be connected to a computer 1508 (e.g., Raspberry Pi via serial USB connection). Output from the computer 1508 may be sent / broadcast to the game server 1510 in the form of websocket data for further processing as discussed elsewhere in the disclosure.

[0048] Some possible advantages of the present disclosure may include selection of digital effects by the user while the online game is in session such as choosing their dealer / croupier to be viewed as a desired animated character during the online game, more realistic animation due to masking of the actual images of the dealer / croupier using machine learning algorithms, avoiding the requirement of post-production or post-editing thereby improving the speed of delivery of the digital effects while the online game is in session and subsequent improved user experience. The present disclosure may advantageously live stream images of the face and body of the dealer / croupier such that the user is dealt to by a 3D animation of the dealer / croupier displayed as the character selected by the user playing an online mixed reality game (e.g. mixed reality card game) on the web browser at the end user device.SUBSTITUTE SHEET (RULE 26)

[0049] Throughout this specification and claims which follow, unless the context requires otherwise, the word "comprise", and variations such as "comprises" and "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

[0050] The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgment or admission or any form of suggestion that that prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

[0051] Many modifications will be apparent to those skilled in the art without departing from the scope of the present invention.SUBSTITUTE SHEET (RULE 26)

Claims

THE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS:

1. A method for dealing to a user during online gaming, the method comprising: receiving, from an end user device, a selection of a character from a pre-selected list of characters at the end user device by the user of the end user device; receiving, from an image capturing module, a plurality of images of a dealer / croupier; generating a three-dimensional (3D) animation of the dealer / croupier based on the received plurality of images of the dealer / croupier and the received selection of the character; and sending the generated 3D animation to the end user device for displaying to the user while the online game is in session.

2. The method according to claim 1, wherein the image capturing module comprises a face capture camera and at least one motion capture camera.

3. The method according to claim 2, wherein the face capture camera captures one or more images of a face of the dealer / croupier.

4. The method of claim 2 or 3, wherein the at least one motion capture camera captures at least one image of a body movement of the dealer / croupier.

5. The method of claim 4, wherein the body movement comprises at least a movement of a hand of the dealer / croupier.

6. The method of any one of claims 1 to 5, wherein the image capturing module captures the plurality of images of the dealer / croupier while the online game is in session.

7. The method of claim 5, wherein generating the 3D animation further comprises masking out the hand of the dealer / croupier from the generated 3D animation.

8. The method of claim 7, wherein the masking out comprises using a plurality of layers of the at least one image of the body movement of the dealer / croupier.

9. The method of any one of claims 1 to 8, wherein generating the 3D animation comprises using a machine learning model to process the plurality of images of the dealer / croupier.

10. The method of any one of claims 1 to 9, wherein the selection of the character from the pre-selected list of characters at the end user device by the user of the end user device is performed while the online game is in session.

11. A system for dealing to a user playing an online game on an end user device, the system comprising: an image capturing module configured to capture a plurality of images of a dealer / croupier; and a three-dimensional (3D) animation generation module communicatively connected to the end user device and the image capturing module, the 3D animation generation module configured to generate a 3D animation of the dealer / croupier based on at least the plurality of images captured by the image capturing module and a selection of a character by the user at the end user device from a pre-selected list while the online game is in session, wherein the generated 3D animation of the dealer / croupier is sent to the end user device for dealing to the user while the online game is in session.

12. The system according to claim 11, wherein the image capturing module comprises a face capture camera and at least one motion capture camera.

13. The system according to claim 12, wherein the face capture camera captures one or more images of a face of the dealer / croupier.

14. The system of claim 12 or 13, wherein the at least one motion capture camera captures at least one image of a body movement of the dealer / croupier.

15. The system of claim 14, wherein the body movement comprises at least a movement of a hand of the dealer / croupier.

16. The system of any one of claims 11 to 15, wherein the image capturing module captures the plurality of images of the dealer / croupier while the online game is in session.

17. The system of claim 15, wherein generating the 3D animation further comprises masking out the hand of the dealer / croupier from the generated 3D animation.

18. The system of claim 17, wherein the masking out comprises using a plurality of layers of the at least one image of the body movement of the dealer / croupier.

19. The system of any one of claims 11 to 18, wherein generating the 3D animation comprises using a machine learning model to process the plurality of images of the dealer / croupier.

20. The system of any one of claims 11 to 19, wherein the selection of the character from the pre-selected list of characters by the user at the end user device is performed while the online game is in session.

Images