Mahjong tile real-time pose estimation method, electronic device and storage medium

Abstract

The application provides a mahjong real-time pose estimation method, an electronic device and a storage medium, and belongs to the technical field of pose estimation. The method comprises the following steps: acquiring a simulation data set and a real data set, wherein the data amount of the real data set is less than the data amount of the simulation data set; training a mahjong detection model through the simulation data set and the real data set, and training a pose estimation model through the simulation data set; when a to-be-detected image is received, inputting the to-be-detected image into the mahjong detection model, and acquiring a target detection frame output by the mahjong detection model, wherein the target detection frame is used for framing each mahjong; inputting image data in the target detection frame into the pose estimation model, and acquiring pose data corresponding to each mahjong in the target detection frame output by the pose estimation model. The application can effectively improve the accuracy and real-time performance of mahjong recognition and pose estimation.

Description

Technical Field

[0001] This application relates to the field of pose estimation technology, and in particular to a method for real-time pose estimation of mahjong tiles, an electronic device, and a storage medium. Background Technology

[0002] Object pose estimation systems, utilizing computer vision technology, accurately estimate the position and orientation of 3D objects in space, and are widely used in robotic grasping. When a robot participates in a mahjong game, this system needs to accurately identify and locate the position and orientation of the mahjong tiles. However, the pose estimation system in this scenario faces several challenges: First, the mahjong tiles are highly similar and have complex and detailed patterns, increasing the difficulty of accurate identification; second, the tiles are closely arranged during the game, and partial occlusion is common, further interfering with the recognition effect; third, the lighting conditions on the mahjong table are variable, and the tiles are prone to glare and shadows, interfering with visual recognition. Finally, the system needs to complete the identification and pose estimation of a large number of mahjong tiles in a short time, which places extremely high demands on real-time performance.

[0003] Currently, object pose estimation techniques are mainly divided into two categories: those based on hand-designed models and those based on deep learning. Methods based on hand-designed models rely on manually designed specific feature points, are sensitive to changes in lighting and occlusion, and perform poorly on objects with little texture. While deep learning-based methods offer superior performance, they depend on a large amount of real pose annotation data. However, the dense arrangement and similarity of mahjong tiles make manual annotation inefficient and difficult to meet practical application needs. Furthermore, different mahjong tiles are difficult to distinguish when viewed from the back, compromising the accuracy of automatic annotation results.

[0004] Therefore, given the limitations of existing technologies in estimating the pose of mahjong tiles, there is an urgent need to develop a real-time object pose estimation system specifically for mahjong tiles to improve the accuracy and real-time performance of mahjong tile recognition. Summary of the Invention

[0005] The purpose of this application is to provide a method, electronic device and storage medium for real-time pose estimation of mahjong tiles to solve the above problems.

[0006] To achieve the above objectives, firstly, this application proposes a method for real-time pose estimation of mahjong tiles, the method comprising:

[0007] Obtain a simulation dataset and a real dataset, wherein the amount of data in the real dataset is less than the amount of data in the simulation dataset;

[0008] A mahjong tile detection model is obtained by training the simulation dataset and the real dataset, and a pose estimation model is obtained by training the simulation dataset.

[0009] When an image to be detected is received, the image to be detected is input into the mahjong tile detection model, and the target detection box output by the mahjong tile detection model is obtained. The target detection box is used to select each mahjong tile.

[0010] The image data within the target detection box is input into the pose estimation model, and the pose data corresponding to the mahjong tiles within each target detection box output by the pose estimation model are obtained.

[0011] In some implementations, prior to obtaining the simulation dataset and the real dataset, the method further includes:

[0012] The mahjong tile model data was obtained by performing a 3D scan on the target mahjong tiles;

[0013] The mahjong tile model data is imported into the Blender tool, and a mahjong simulation environment is built within the Blender tool to obtain a first subset of simulation data; and

[0014] The mahjong tile model data is imported into the Isaac Sim tool, and a mahjong table simulation environment is built in the Isaac Sim tool to obtain a second subset of simulation data.

[0015] A simulation dataset is formed based on the first and second subsets of simulation data.

[0016] In some implementations, obtaining mahjong tile model data by performing a three-dimensional scan of the target mahjong tiles includes:

[0017] Initial model data was obtained by performing a 3D scan on the target mahjong set;

[0018] Texture analysis was performed on the initial model data to identify the high-reflectivity areas;

[0019] Texture filling is performed on the high-reflectivity areas to obtain mahjong tile model data.

[0020] In some implementations, the step of importing the mahjong tile model data into the Blender tool and constructing a mahjong simulation environment within the Blender tool to obtain a first subset of simulation data includes:

[0021] Import the mahjong tile model data into the Blender tool, and configure rigid body properties for the mahjong tile model data using the physics engine;

[0022] A hierarchical sampling mechanism was used to initialize the pose of the mahjong tile model data configured with rigid body properties;

[0023] Background data for various mahjong interactive scenarios is generated using texture blending technology.

[0024] A first lighting system is formed by configuring a planar light source with dynamically changing light intensity and a point light source with dynamically adjusted color temperature.

[0025] Under the first lighting system, the first simulation data subset is obtained by acquiring simulated color images and simulated depth images of the mahjong tile model data under various background data after pose initialization by implementing a progressive sampling strategy on the camera system.

[0026] In some implementations, the process of importing the mahjong tile model data into the Isaac Sim tool and constructing a mahjong table simulation environment within the Isaac Sim tool to obtain a second subset of simulation data includes:

[0027] Import the mahjong tile model data into the Isaac Sim tool, and configure rigid body properties for the mahjong tile model data using the physics engine;

[0028] Background data for various mahjong interactive scenarios is generated using texture blending technology.

[0029] A second lighting system is formed by configuring point light sources with dynamically changing light intensity, color temperature, and light source position;

[0030] A mahjong table simulation environment is built in the Isaac Sim tool, and a regionalized random strategy is adopted in the mahjong table simulation environment to generate various rotation postures of mahjong tiles based on mahjong tile model data configured with rigid body properties. The tablecloth of the mahjong table simulation environment maintains optical consistency with the background data in each mahjong interaction scene by implementing a shadow projection system.

[0031] Under the second lighting system, a second set of simulation data is obtained by acquiring simulated color images and simulated depth images of a mahjong table simulation environment with mahjong tiles in various rotation postures under various background data through a dynamic sampling strategy implemented on the vision acquisition system.

[0032] In some implementations, forming a simulation dataset based on the first subset of simulation data and the second subset of simulation data includes:

[0033] The mahjong tiles in the first and second simulation data subsets are labeled with detection information. The labeling results include the tile classification and the two-dimensional rectangles surrounding the mahjong tiles.

[0034] The mahjong tiles in the first and second simulation data subsets are used to construct a physical coordinate system with the center of the mahjong tile as the origin for pose information annotation. The pose information annotation results include the three-dimensional translation vector and the three-dimensional rotation matrix of the mahjong tile.

[0035] A simulation dataset is formed based on the first and second subsets of simulation data that have been annotated with detection and pose information.

[0036] In some embodiments, the method further includes:

[0037] When annotating detection information, detect the quadrilaterals projected onto the image plane by each mahjong tile;

[0038] When the quadrilateral is an inverted quadrilateral, or when the area of ​​the quadrilateral is less than a preset pixel value, the corresponding mahjong tile is removed from the detection information labeling.

[0039] In some implementations, the step of inputting the image to be detected into the mahjong tile detection model upon receiving the image to be detected, and obtaining the target detection bounding box output by the mahjong tile detection model, includes:

[0040] Upon receiving an image to be detected, the image is input into the mahjong tile detection model, and the initial detection box output by the mahjong tile detection model is obtained.

[0041] Initial detection boxes with a confidence level greater than a preset confidence threshold are selected as target detection boxes.

[0042] Secondly, to achieve the above objectives, this application also proposes an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the real-time pose estimation method for mahjong tiles as described above.

[0043] Thirdly, to achieve the above objectives, this application also proposes a computer storage medium storing executable instructions, which, when executed by a processor, cause the processor to perform the real-time pose estimation method for mahjong tiles as described above.

[0044] Compared with the prior art, the beneficial effects of this application include:

[0045] Firstly, by using a simulated dataset and a small amount of real-world data for training, the reliance on real data is reduced, thereby lowering the workload and cost of data annotation. Secondly, the simulated dataset provides diverse scenes and lighting conditions, overcoming problems such as lighting changes and occlusion that may be encountered during real-world data collection, thus improving the quality of the training data. Thirdly, by employing a cascaded approach of a mahjong tile detection model and a pose estimation model, while ensuring accurate identification of mahjong tiles using the detection model, a single pose estimation model can estimate the pose of all mahjong tiles. This avoids training a separate network for each type of mahjong tile, significantly reducing GPU and system memory usage, improving the system's real-time response capability, and meeting the needs for rapid identification and pose estimation in mahjong game scenarios. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0047] Figure 1 This is a flowchart illustrating a method for real-time pose estimation of mahjong tiles in one embodiment;

[0048] Figure 2 This is a schematic diagram of a real-color image captured in a real environment in one embodiment;

[0049] Figure 3 This is a schematic diagram of another real-color image captured in a real environment in one embodiment;

[0050] Figure 4 This is a schematic diagram of another real-color image captured in a real environment in one embodiment;

[0051] Figure 5 This is a schematic diagram illustrating the annotation of detection information on a real color image in one embodiment;

[0052] Figure 6 This is a schematic diagram illustrating the annotation results of manually annotating a real color image in one embodiment;

[0053] Figure 7 This is a schematic diagram of the annotation results of a real color image annotated using a mahjong tile detection model in one embodiment;

[0054] Figure 8 This is a visualization of the three-dimensional pose estimation result of a pose estimation model in one embodiment.

[0055] Figure 9This is another visualization of the 3D pose estimation result performed by the pose estimation model in one embodiment.

[0056] Figure 10 This is a flowchart illustrating the real-time pose estimation method for mahjong tiles in another embodiment;

[0057] Figure 11 This is a schematic diagram of model processing in one embodiment, which obtains mahjong tile model data by performing a 3D scan of the target mahjong tile.

[0058] Figure 12 This is a schematic diagram of the annotation results of the detection information annotation using the Blender tool in one embodiment;

[0059] Figure 13 This is a schematic diagram of the annotation results of detection information annotation using the Isaac Sim tool in one embodiment;

[0060] Figure 14 This is a schematic diagram of the annotation result using the Blender tool in one embodiment;

[0061] Figure 15 This is another illustration of the annotation result obtained by using the Blender tool in one embodiment;

[0062] Figure 16 This is a schematic diagram of the electronic device involved in the real-time pose estimation method for mahjong tiles in the embodiments of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] All terms used in this application (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0065] For example, the terms "first," "second," etc., used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For instance, without departing from the scope of this application, a first subset of simulation data may be referred to as a second subset of simulation data, and similarly, a second subset of simulation data may be referred to as a first subset of simulation data. Both the first subset of simulation data and the second subset of simulation data are subsets of simulation data, but they are not the same subset of simulation data.

[0066] For example, the terms "comprising" or "including" used in this application indicate the presence of features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0067] As mentioned earlier, object pose estimation systems utilize computer vision technology to accurately estimate the position and orientation of 3D objects in space, and are widely used in the field of robot grasping. When a robot participates in a mahjong game, the system needs to accurately identify and locate the position and orientation of the mahjong tiles. However, the pose estimation system in this scenario faces many challenges: First, the mahjong tiles are highly similar and have complex and detailed patterns, increasing the difficulty of accurate identification; second, the mahjong tiles are closely arranged during the game, and partial occlusion is common, which further interferes with the recognition effect; third, the lighting conditions on the mahjong table are variable, and the mahjong tiles are prone to reflection and shadows, interfering with visual recognition. Finally, the system also needs to complete the identification and pose estimation of a large number of mahjong tiles in a short period of time, which places extremely high demands on real-time performance. Currently, object pose estimation techniques are mainly divided into two categories: methods based on hand-designed models and methods based on deep learning. Methods based on hand-designed models rely on specific hand-designed feature points, are sensitive to changes in lighting and occlusion, and perform poorly on objects with little texture; while methods based on deep learning have superior performance, they rely on a large amount of real pose annotation data. However, the dense arrangement and similarity of mahjong tiles lead to low efficiency in manual annotation, making it difficult to meet practical application needs. Furthermore, different mahjong tiles are difficult to distinguish when viewed from the back, compromising the accuracy of automatic annotation. Therefore, given the limitations of existing technologies in mahjong tile pose estimation, there is an urgent need to develop a real-time object pose estimation system specifically for mahjong tiles. To this end, this application proposes a real-time pose estimation method, electronic device, and storage medium for mahjong tiles, which can effectively improve the accuracy and real-time performance of mahjong tile recognition.

[0068] like Figure 1 As shown in the figure, this application provides a method for real-time pose estimation of mahjong tiles, the method including the following steps:

[0069] Step S10: Obtain the simulation dataset and the real dataset.

[0070] In this embodiment, the simulation dataset refers to a collection of image data generated through computer simulation technology, containing a large number of images simulating mahjong tiles in various mahjong interaction scenarios. For example, mahjong tile model data can be obtained by performing a 3D scan of the target mahjong tile, and then imported into simulation software such as Blender, Isaac Sim, 3ds Max, and Maya. A mahjong tile simulation environment can be constructed in the simulation software, and various simulation parameters (such as light source position, light intensity, background data, shooting angle, etc.) can be set in the mahjong tile simulation environment to simulate real lighting, background, occlusion, and other factors, generating a large number of simulated color images and simulated depth images. After detection information annotation and pose information annotation, a simulation dataset is formed to train the model to adapt to different situations.

[0071] A real-world dataset refers to a collection of true-color images of mahjong tiles collected in a real-world environment, such as... Figures 2 to 4 As shown. Each frame in the real dataset also contains annotation information, such as the detection information of the mahjong tiles in the image annotated using annotation tools (such as LabelMe) (e.g., the 2D detection boxes surrounding the mahjong tiles and the annotation information of the tile classification), such as... Figure 5 As shown, the quadrilateral wireframe corresponds to a 2D detection box, and the text above the box (i.e., annotation information) indicates the tile classification. For example, annotation "t1" represents a single tile, annotation "b3" represents a three of dots, and annotation "w5" represents a five of characters. Furthermore, the real dataset has a smaller data volume than the simulation dataset, requiring only a relatively small number of real color images to narrow the gap between the simulation training results and the real environment. For example, 800 real color images of mahjong tiles under different environments can be collected, such as drawing tiles, holding tiles, and opponents discarding tiles, simulating changes in ambient lighting as much as possible to simulate different tile situations. The collected real color images are then labeled with 2D detection boxes to form a classification system, as shown below. Figure 5 The real dataset shown.

[0072] By combining simulation datasets and real datasets, we can improve the model's generalization ability to different scenarios while ensuring data quality, reduce dependence on real datasets, and reduce data collection costs.

[0073] Step S20: A mahjong tile detection model is trained using the simulation dataset and the real dataset, and a pose estimation model is trained using the simulation dataset.

[0074] The mahjong tile detection model in this embodiment is a model used to detect the position of mahjong tiles in the image to be detected and generate detection boxes. It can be the YOLOv11 detection model.

[0075] In some implementations, simulated and real datasets can be mixed and divided into training and test sets according to a preset ratio (e.g., 95:5, 98:2, 90:10, etc.). This ensures that both simulated and real datasets are distributed across both sets, thereby improving the model's generalization ability. The training set is used to train object detection algorithms such as YOLOv11. When the loss function of the model on the test set fails to decrease for several consecutive rounds, training automatically stops to prevent overfitting and ensure the model's performance on unseen data, resulting in a mahjong tile detection model.

[0076] During the training process of the mahjong tile detection model, various data augmentation strategies can be implemented on the training set. These strategies include, but are not limited to: adjusting the illumination angle and intensity of each image in the training set to simulate different lighting conditions; randomly flipping each image in the training set to increase sample diversity; cropping each image in the training set to enhance the robustness of the mahjong tile detection model to local features; and stitching together images from different scenes in the training set to enrich the data distribution. Through these data augmentation strategies, the mahjong tile detection model's adaptability to complex environments can be effectively improved.

[0077] Furthermore, since the annotations of mahjong tiles are prone to errors, with an actual error rate exceeding 2%, it is necessary to re-annotate incorrectly labeled data. To reduce the cost of manual review, this embodiment can use a pre-trained mahjong tile detection model to verify the manually annotated real dataset. If the Intersection over Union (IoU) between the manually annotated 2D detection boxes and the detection boxes output by the mahjong tile detection model is found to be greater than a preset IoU (e.g., 0.5), the incorrectly annotated image data is saved and re-annotated for subsequent model training. Figure 6 and Figure 7 As shown, Figure 6 These are real image data with manual annotations. Figure 7 This is the detection result output by the mahjong tile detection model for unlabeled real image data. It can be seen that... Figure 6 The six of thousands in the mahjong tile detection model was manually labeled as nine of thousands (label information "w9"), which does not match the output of the mahjong tile detection model. Figure 6 Save the file separately for subsequent review. If the review result indicates an error in the labeling, then... Figure 6 Re-label the data for use in model training.

[0078] For example, the test results are shown in Table (1) below:

[0079] Table (1)

[0080]

[0081] Here, mAP50 stands for "mean Average Precision at IoU=0.5," representing the average precision across all categories when the Intersection over Union (IoU) threshold is set to 0.5. It measures the mahjong tile detection model's ability to locate and classify mahjong tiles; a higher value (maximum 1) indicates better model performance. IoU=0.5 means that the overlap between the predicted and ground truth bounding boxes must be at least 50% of their total area for a detection to be considered successful.

[0082] mAP50-95 stands for "Mean Average Precision from IoU 0.5 to 0.95". It is calculated by taking the average of the average precision (AP) at IoU values ​​from 0.5 to 0.95 (10 thresholds in 0.05 increments). This metric is more stringent than a single IoU threshold (such as mAP50) and comprehensively reflects the model's detection capability under different positioning accuracy requirements; a higher value indicates better model performance.

[0083] The pose estimation model in this embodiment is a model used to estimate the position and orientation of the mahjong tiles detected by the mahjong tile detection model in three-dimensional space, and can be the HiPose pose estimation model.

[0084] In some implementations, pose estimation algorithms such as HiPose can be trained using only simulation datasets to obtain pose estimation models, thus fully utilizing the diversity and accuracy of simulation data and improving pose estimation performance. It should be noted that this embodiment ignores the differences between different mahjong tiles; that is, different tile faces are represented and learned using a uniform pose.

[0085] During the training process of the pose estimation model, various image enhancement strategies can be implemented on the training set. These strategies include: adding Gaussian noise and salt-and-pepper noise to simulate sensor errors in the simulated depth image; randomly adjusting the illumination angle and intensity of the simulated color image to adapt to different lighting environments; randomly cropping each image in the training set based on the random scaling and offset of the detection box to enhance the model's robustness to local features; and superimposing occlusion blocks to improve the pose estimation model's ability to estimate pose under occlusion conditions.

[0086] Furthermore, to ensure the validity of the image data in the training set, this embodiment automatically removes mahjong tile samples where the visible portion is less than 0.2 due to occlusion, ensuring that the features learned by the model are representative and have generalization ability. According to experimental data, the training process of this pose estimation model consumed 9.66GB of GPU memory, and the final checkpoint (ckpt, used to store model parameters and other information during training) size was 40.6MB. The pose estimation model size was 425MB. The ADD metric was 90.5%. ADD is a commonly used metric for pose estimation, referring to the percentage of pose offsets less than 0.1 times the object diameter on the test set being 90.5%.

[0087] Step S30: When the image to be detected is received, the image to be detected is input into the mahjong tile detection model, and the target detection box output by the mahjong tile detection model is obtained. The target detection box is used to select each mahjong tile.

[0088] In this embodiment, the image to be detected refers to the image for which mahjong tile detection and pose estimation are required, and can be image data from a camera, video stream, or other image sources. The target detection box refers to a two-dimensional rectangular box output by the mahjong tile detection model, used to select the detected mahjong tiles in the image and label the corresponding tile face classification.

[0089] The mahjong tile detection model can quickly and accurately locate mahjong tiles in an image, providing a target region for further pose estimation, narrowing the search range for pose estimation, and improving the efficiency of the entire system.

[0090] In some implementations, step S30 includes: upon receiving an image to be detected, inputting the image to be detected into the mahjong tile detection model and obtaining the initial detection bounding box output by the mahjong tile detection model; selecting initial detection bounding boxes with a confidence level greater than a preset confidence threshold (e.g., 0.7) as target detection bounding boxes. Here, confidence level refers to the degree of confidence of the mahjong tile detection model in identifying objects within the initial detection bounding box as mahjong tiles, and its value is between 0 and 1.

[0091] For example, in a scenario where a robot automatically plays mahjong, a camera captures an image of the mahjong table; this image is the image to be detected. After inputting this image into a mahjong tile detection model, the model outputs multiple initial detection boxes, each with a corresponding confidence level. If the preset confidence threshold is 0.7, only initial detection boxes with a confidence level greater than 0.7 will be retained as target detection boxes; the others will be discarded. The image data within these target detection boxes will be extracted for subsequent pose estimation. For example, initial detection box A has a confidence level of 0.8, which is greater than the threshold of 0.7, and is therefore retained as a target detection box; while initial detection box B has a confidence level of 0.6, which is less than the threshold, and is discarded. This implementation ensures that the input for subsequent pose estimation is a relatively reliable mahjong tile region, improving the accuracy and efficiency of the entire system.

[0092] Step S40: Input the image data within the target detection box into the pose estimation model, and obtain the pose data corresponding to the mahjong tiles within each target detection box output by the pose estimation model.

[0093] The pose estimation model in this embodiment employs a parallel inference strategy, enabling it to process multiple target detection boxes simultaneously, significantly improving inference efficiency. During inference, the pose estimation model performs 3D pose estimation on the mahjong tiles within each target detection box, outputting pose data of the mahjong tiles in the camera coordinate system (such as 3D translation vectors and rotation matrices). The visualization results are shown below. Figure 8 and Figure 9 As shown, Figure 8 and Figure 9 The bright areas in the color depth image are due to the characteristics of depth images, which only capture the front of the hand and mahjong tile, leaving the back empty and appearing as bright areas with missing color. Figure 8 and Figure 9 The mahjong tiles in the image are all enclosed in 3D bounding boxes (green 3D boxes in the figure) for pose estimation.

[0094] This embodiment uses the coordinates of the target detection box and the tile classification detected by the mahjong tile detection model, as well as the pose data estimated by the pose estimation model, to provide accurate input data for subsequent robot grasping and operation.

[0095] The real-time pose estimation method for mahjong tiles proposed in this application has several advantages. First, by using a simulated dataset and a small amount of real datasets for training, the reliance on real data is reduced, thereby lowering the workload and cost of data annotation. Second, the simulated dataset provides diverse scenes and lighting conditions, overcoming problems such as lighting changes and occlusion that may be encountered during real data acquisition, thus improving the quality of training data. Third, by employing a cascaded approach of a mahjong tile detection model and a pose estimation model, while ensuring accurate identification of mahjong tiles using the detection model, only one pose estimation model is needed to estimate the pose of all mahjong tiles. This avoids training a separate network for each type of mahjong tile, significantly reducing GPU and system memory usage, improving the system's real-time response capability, and meeting the needs for rapid identification and pose estimation in mahjong game scenarios.

[0096] In one embodiment, such as Figure 10 As shown, prior to step S10, the procedure further includes:

[0097] Step S01: Obtain mahjong tile model data by performing a 3D scan on the target mahjong tile.

[0098] In this embodiment, the target mahjong set refers to the mahjong tiles that require 3D scanning to obtain model data. This embodiment uses Sichuan mahjong as an example, using 108 tiles, removing the wind tiles (East, South, West, North), arrow tiles (Red Dragon, Green Dragon, White Dragon), and flower tiles (Spring, Summer, Autumn, Plum Blossom, Orchid, Bamboo, Chrysanthemum). Only the three suits (Circles, Bamboo, Characters) are retained, with 36 tiles of each suit, totaling 108 tiles. Each suit has nine possible values ​​(1-9), with four identical tiles of each value. The mahjong tile model data refers to the 3D model data of the mahjong tiles obtained through 3D scanning, including information such as geometric shape and texture.

[0099] In some embodiments, step S01 includes: obtaining initial model data by performing a three-dimensional scan of the target mahjong tile; performing texture analysis on the initial model data and identifying the high-gloss reflection area, which refers to the area where light reflection is strong due to the material characteristics of the mahjong tile, resulting in a high gloss; and filling the high-gloss reflection area with texture based on the texture pattern and color around the high-gloss reflection area to obtain mahjong tile model data.

[0100] For example, such as Figure 11As shown, a 3D scanner can be used to scan the "Nine of Characters" tile to obtain its initial model data. During the scanning process, due to the reflective material on the surface of the mahjong tile, some areas of the scanned texture data exhibited specular reflections, resulting in incomplete texture information in these areas. The initial model data was then imported into texture analysis software. The software analyzed parameters such as brightness, contrast, and reflectivity to identify the specular reflection areas. These areas typically appear as brighter, more reflective parts of the texture, such as the pattern of the "Nine of Characters" tile. Next, the initial model data and texture analysis results were imported into texture restoration software with rendering capabilities. Taking Blender as an example, texture editing tools were used to repair the specular reflection areas. Specifically, based on the texture pattern and color surrounding the specular reflection area, the corresponding fill algorithm and parameters were selected to fill the area. For example, interpolation algorithms could be used to estimate and fill the missing parts based on the color and pattern of surrounding pixels. Finally, the repaired initial model data became the complete mahjong tile model data, including accurate geometry and complete texture information.

[0101] The mahjong tile model data generated in this embodiment can be used for subsequent simulation data generation, such as building a mahjong simulation environment in Blender or Isaac Sim to generate simulation image data in various scenarios, ensuring the authenticity and accuracy of the simulation data. This data can then be used to train the mahjong tile detection model and pose estimation model, thereby improving the training effect and performance of the model.

[0102] Step S02: Import the mahjong tile model data into the Blender tool, and obtain the first simulation data subset by building a mahjong simulation environment in the Blender tool.

[0103] The Blender tool used in this embodiment is an open-source 3D modeling and rendering software that can be used to generate high-quality simulation images. The first subset of simulation data is a collection of simulation image data generated in Blender, including simulated color images and simulated depth images under various mahjong tile interactive scenes. When building a mahjong simulation environment in Blender, data diversity can be improved through multi-dimensional randomization strategies. These multi-dimensional randomization strategies include: for mahjong tile model data, various pose sampling mechanisms combined with position perturbation and multi-angle rotation combinations can be used. Point light sources and planar light sources can also be built to simulate ambient lighting, texture blending technology can be used to generate procedural backgrounds and adjust material optical properties, and an adaptive camera system can be configured to achieve multi-view capture and intelligent obstacle avoidance.

[0104] Specifically, step S02 may include: importing the mahjong tile model data into the Blender tool and configuring rigid body properties on the mahjong tile model data through the physics engine.

[0105] The physics engine is a module in Blender used to simulate motion and interactions in the physical world, such as collision detection and gravity. Rigid body properties describe the characteristics of objects (mahjong tiles) in the physical simulation, such as mass, inertia, and coefficient of friction. For example, the mass of each mahjong tile model can be set to 100 grams, the coefficient of kinetic friction to 0.3, and a 20-substep physics simulation can be enabled to ensure stacking stability. By configuring rigid body properties, the mahjong tiles can correctly respond to physical forces and interactions in the simulation environment, improving the realism of the simulation.

[0106] A hierarchical sampling mechanism is employed to initialize the pose of mahjong tile model data configured with rigid body properties. Hierarchical sampling is a method for initializing object position and pose through tiered sampling, ensuring sample diversity and coverage. Taking the tabletop where the mahjong tile model data is placed as the first standard coordinate system in the XY plane as an example, the hierarchical sampling mechanism can include: generating reference coordinates within the XY plane interval [-0.3m, 0.3m], superimposing ±1cm position noise, and generating mahjong tile model data with a combined pose including a 90° vertical flip (40% probability) and ±30° planar rotation through Euler angle sampling. Pose initialization using the hierarchical sampling mechanism ensures the diversity and representativeness of the generated mahjong tile poses, avoids excessive concentration of samples, and improves the quality of training data.

[0107] By employing texture blending technology, background data for various mahjong interactive scenarios is generated. For example, 2-4 materials (such as wood, stone, etc.) are randomly selected for each scenario and alpha blending is performed, with material roughness (0-1.0) and specular reflectivity (0.2-0.8) parameters dynamically configured. Generating diverse background data enhances the richness and realism of the mahjong simulation environment, enabling the subsequently trained model to better adapt to different real-world scenarios.

[0108] A first lighting system is formed by configuring a planar light source with dynamically changing light intensity and a point light source with dynamically adjusted color temperature. The light intensity of the planar light source can be 3-6 W / m². 2 The color temperature of the point light source can be dynamically adjusted within the range of 5000-6500K, and the positions of the planar light sources are randomly distributed on a plane at a height of 10m perpendicular to the XY plane. The positions of the point light sources are also randomly distributed within a hemispherical shell with a radius of 1-1.5m. This dynamically changing first lighting system can simulate changes in lighting conditions in a real environment, improving the diversity of the first simulation dataset and the model's adaptability to different lighting conditions.

[0109] Under the first lighting system, the first simulation data subset is obtained by acquiring simulated color images and simulated depth images of the mahjong tile model data under various background data after pose initialization by implementing a progressive sampling strategy on the camera system.

[0110] The camera system is used to capture image data in a mahjong simulation environment, simulating the function of a camera. A progressive sampling strategy generates initial points on an observation sphere with a radius of 0.1-0.75m. Based on a point of interest (POI) algorithm, the observation angle is dynamically adjusted according to the initial points. The focal length parameter of the camera system can be adaptively selected between 24-35mm based on the distribution density of the target objects (mahjong tile model data). Furthermore, a minimum object distance constraint of 0.3m can be set to prevent the mahjong tile model data from being too close to the camera system, resulting in inconsistencies in the image. By acquiring diverse simulation image data through the progressive sampling strategy, the training simulation dataset is enriched, improving the ability of the subsequently trained model to recognize different scenes, lighting conditions, and tile poses.

[0111] Step S03: Import the mahjong tile model data into the Isaac Sim tool, and obtain the second simulation data subset by building a mahjong table simulation environment in the Isaac Sim tool.

[0112] The Isaac Sim tool used in this embodiment is a high-performance simulation tool used to provide a highly realistic simulation environment. The second subset of simulation data is a collection of simulation image data generated in Isaac Sim, containing highly realistic mahjong table and mahjong tile image data. When building a mahjong table simulation environment in Isaac Sim, the real-time stability of the dense mahjong tile model data interaction can be ensured through physics engine optimization. A regionalized random distribution strategy can also be used to generate various rotational postures of mahjong tiles in the mahjong table simulation environment. Furthermore, ambient lighting simulation can be achieved by configuring point light sources with dynamically changing light intensity, color temperature, and light source position; procedural backgrounds can be generated using texture blending technology and the optical properties of materials can be adjusted; and a multi-degree-of-freedom vision acquisition system can be configured to achieve dynamic acquisition of simulation image data.

[0113] Specifically, step S03 includes: importing the mahjong tile model data into the Isaac Sim tool, and configuring rigid body properties for the mahjong tile model data through the physics engine.

[0114] The physics engine, a module in Isaac Sim, simulates motion and interactions in the physical world, such as collision detection and gravity. Rigid body properties describe the characteristics of objects (mahjong tiles) in the physical simulation, such as mass, inertia, and coefficient of friction. For example, the mass of each mahjong tile model can be set to 100 grams and the coefficient of kinetic friction to 0.3, making its motion characteristics approximate those of a real mahjong tile. By configuring rigid body properties, the mahjong tiles can correctly respond to physical forces and interactions in the simulation environment, improving the realism of the simulation.

[0115] By employing texture blending technology, background data for various mahjong interactive scenarios is generated. For example, 2-4 texture patterns (such as wood texture, fabric texture, etc.) can be automatically blended in each simulation round, and the material reflectivity can be randomly adjusted (0.2-0.8). Generating diverse background data enhances the richness and realism of the simulation environment, enabling the trained model to better adapt to different real-world scenarios. A second lighting system is formed by configuring point light sources with dynamically changing light intensity, color temperature, and light source position. For example, point light sources can be configured programmatically or using Isaac Sim's built-in tools, with their light intensity dynamically varying between 500-1500 lumens and their color temperature dynamically varying between 3000-6500K. Simultaneously, the light source position is randomly located within a hemispherical space with a radius of 1 meter. This dynamically changing second lighting system simulates changes in lighting conditions in the real environment, improving the diversity of the second simulation dataset and the model's adaptability to different lighting conditions.

[0116] A mahjong table simulation environment is built in the Isaac Sim tool, and a regionalized random strategy is adopted in the mahjong table simulation environment to generate various rotational postures of mahjong tiles based on mahjong tile model data configured with rigid body properties.

[0117] The mahjong table simulation environment refers to a 3D scene built in Isaac Sim that simulates a real mahjong table and its surrounding environment. This can be achieved by importing 3D reconstructed mahjong table model data. By constructing a highly realistic mahjong table simulation environment, scene support can be provided for generating a highly realistic second simulation dataset, enabling the trained model to better adapt to actual mahjong interaction scenarios. The regionalized randomization strategy refers to a strategy that randomly generates the position and posture of mahjong tiles within a preset area (such as the tabletop) of the mahjong table simulation environment.

[0118] For example, a second standard coordinate system is established using the mahjong table simulation environment's tabletop as the XY plane. Reference coordinates can be generated within the X-axis ±0.5m and Y-axis ±0.4m intervals in the XY plane, with ±1cm positional offset noise superimposed. Simultaneously, various mahjong tile rotation postures (including upright, inverted, and ±15-degree deflection combinations) are generated through quaternion random sampling. Generating diverse mahjong tile rotation postures improves the diversity of the second simulation dataset and the model's ability to recognize different tile states.

[0119] Furthermore, the tablecloth in the mahjong table simulation environment maintains optical consistency with the background data in each mahjong interaction scene through the implementation of a shadow projection system. A shadow projection system is a system used to generate shadows in a simulated environment, enabling objects to produce realistic shadow effects under lighting. Optical consistency means that the lighting effects in the scene match the lighting conditions and object positions, conforming to real optical laws.

[0120] For example, a shadow projection system can be enabled in Isaac Sim, allowing the tablecloth in the mahjong table simulation environment to generate realistic shadows based on the lighting parameters of the second lighting system and the positions of the mahjong tiles. Maintaining optical consistency through the shadow projection system improves the realism of the simulation environment, enabling the trained model to better adapt to changes in light and shadow in real-world scenes.

[0121] Under the second lighting system, a second set of simulation data is obtained by acquiring simulated color images and simulated depth images of a mahjong table simulation environment with mahjong tiles in various rotation postures under various background data through a dynamic sampling strategy implemented on the vision acquisition system.

[0122] The visual acquisition system is used to capture image data in a mahjong table simulation environment, simulating the function of a camera. The dynamic sampling strategy includes configuring a movable multi-angle camera for the visual acquisition system. Its observation point is based on coordinates (0.7, -0.5, 1.2) in a second standard coordinate system, with random offsets of ±20cm across the X / Y / Z axes superimposed. The dynamic adjustment range of the focal length is 24-35mm to simulate different depth-of-field effects. By acquiring diverse simulation image data through this dynamic sampling strategy, the simulation dataset is enriched, and the model's ability to recognize different scenes, lighting conditions, and tile poses is improved.

[0123] Step S04: Based on the first simulation data subset and the second simulation data subset, a simulation dataset is formed.

[0124] In this embodiment, the simulation dataset is a complete simulation dataset formed by merging the first and second simulation datasets, used to train the mahjong tile detection model and the pose estimation model. During the merging process, unified detection information and pose information annotations can be performed on the mahjong tiles in both the first and second simulation datasets.

[0125] Specifically, it can be like Figure 12 and Figure 13 As shown, detection information is labeled for mahjong tiles in the first and second simulation data subsets. The labeling results include the tile's face classification and a two-dimensional bounding box surrounding the tile. The face classification refers to the tile's face, such as 3 of Wan, 5 of Tiao, 8 of Tong, etc., and the two-dimensional bounding box is used to select the two-dimensional coordinate region of the mahjong tile in the image. This detection information labeling provides accurate training data for subsequent model training, improving the accuracy and recall of the detection model, enabling it to more accurately identify and locate mahjong tiles.

[0126] For the mahjong tiles in the first and second subsets of simulation data, a physical coordinate system is constructed with the center of the mahjong tile as the origin to annotate their pose information. The pose information annotation results include the three-dimensional translation vector and the three-dimensional rotation matrix of the mahjong tile. The three-dimensional translation vector describes the positional offset of the mahjong tile in three-dimensional space, and the three-dimensional rotation matrix describes the rotational state of the mahjong tile in three-dimensional space. Figures 14 to 15 As shown, each mahjong tile is selected using a 3D bounding box, and then a physical coordinate system is constructed with the center of the mahjong tile as the origin to annotate its pose information. Finally, a simulation dataset is formed based on the first and second subsets of simulation data with both detection and pose information annotations.

[0127] In some implementations, when annotating detection information, the quadrilaterals projected onto the image plane by each mahjong tile are detected; when the quadrilateral is an inverted quadrilateral, or when the area of ​​the quadrilateral is less than a preset pixel value, the corresponding mahjong tile is removed from the detection information annotation.

[0128] In this context, the image plane refers to the plane containing the two-dimensional image data from the first and second subsets of simulation data, which is also the plane on which the camera images. The projected quadrilateral is the quadrilateral formed by projecting the four points (top left / top right / bottom right / bottom left) of the mahjong tile's front face onto the image plane. A reversed quadrilateral is one where the vertices are arranged counter-clockwise, indicating the back of the mahjong tile faces the camera; in this case, the tile's classification cannot be determined from the color image. The preset pixel value is a pre-defined pixel area threshold used to determine if the quadrilateral is too small. If the area is less than the preset pixel value (e.g., 500 pixels), the mahjong tile's label is removed to avoid unclear tile details affecting the model's recognition performance. Furthermore, the visible portion ratio of each mahjong tile can be detected. If the visible portion ratio is less than a preset ratio (e.g., 0.2), the corresponding mahjong tile is removed from the detection information labeling to prevent the model from failing to accurately identify the mahjong tile's features due to insufficient visible portion. The visible portion ratio refers to the ratio of the area of ​​the visible portion of the mahjong tile in the image to the entire area of ​​the mahjong tile.

[0129] In the real-time pose estimation method for mahjong tiles proposed in this application, firstly, a first and a second subset of simulation data are generated using two simulation tools, Blender and Isaac Sim, respectively, and then merged to form a simulation dataset. Blender emphasizes environmental diversity, while Isaac Sim emphasizes physical realism, forming a complementary simulation data generation system. This ensures both the diversity of simulation data in terms of scene and lighting conditions, and the reliability in terms of physical characteristics and realism, enabling the trained model to better adapt to various real-world scenarios. Simultaneously, the construction of the simulation dataset reduces reliance on real data, avoiding the high cost and difficulty of collecting large amounts of data in real-world environments, and also reduces the impact of inaccurate annotation of real data.

[0130] Secondly, this application constructs a standardized simulation dataset construction process, which abstracts and unifies the representation of mahjong tiles with highly similar geometric shapes. This enables joint training of multiple categories of objects (mahjong tiles with different faces) through cascaded networks, without having to train a separate network for each category of object (mahjong tiles with different faces). This fundamentally solves the problems of high memory consumption and long inference time caused by independent training in other methods, and greatly improves the real-time response capability of the system.

[0131] Thirdly, this application generates background data for various mahjong interaction scenarios using texture blending technology, enabling the model to learn the object's behavior under different material characteristics. This not only improves the model's ability to recognize the target object (mahjong tiles) in different environments but also enhances the model's robustness to other elements in the background. In practical applications, the model needs to handle various complex scenarios, including interactions between human hands and mahjong tiles. Because the simulation dataset of this application generates rich and diverse background data during its construction phase, the trained model, when encountering complex elements such as human hands in real-world scenarios, can focus on recognizing the target object (mahjong tiles) without being disturbed, thanks to its strong adaptability. For example, in a scenario where a robot automatically plays mahjong, even with hands or other interfering objects, the model can still accurately recognize the mahjong tiles.

[0132] In one embodiment, a computer storage medium is provided that stores executable instructions that, when executed by a processor, cause the processor to perform the steps in the above method embodiments.

[0133] In one embodiment, an electronic device is also provided, including one or more processors; and a memory storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the steps in the above-described method embodiments. For example, the electronic device can be any device such as the aforementioned thermal management device or computing service device.

[0134] In one embodiment, such as Figure 16 The diagram illustrates the structure of an electronic device used to implement an embodiment of this application. The electronic device includes a central processing unit (CPU) 101, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 102 or a program loaded from a storage portion 108 into a random access memory (RAM) 103. The RAM 103 also stores various programs and data required for the operation of the electronic device. The CPU 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0135] The following components are connected to I / O interface 105: an input section 106 including a keyboard, mouse, etc.; an output section 107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 108 including a hard disk, etc.; and a communication section 109 including a network interface card such as a LAN card, modem, etc. The communication section 109 performs communication processing via a network such as the Internet. A drive 110 is also connected to I / O interface 105 as needed. A removable medium 111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 110 as needed so that computer programs read from it can be installed into storage section 108 as needed.

[0136] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer-readable medium carrying instructions that, in such embodiments, can be downloaded and installed from a network via communication section 109, and / or installed from removable medium 111. When the instructions are executed by central processing unit (CPU) 101, the various method steps described in this application are performed.

[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0138] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, any of the embodiments or implementations claimed above can be used in any combination. The information disclosed in this background section is intended only to enhance the understanding of the general background of this application and should not be construed as an admission or in any way implying that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for real-time pose estimation of mahjong tiles, characterized in that, The method includes: The mahjong tile model data was obtained by performing a 3D scan on the target mahjong tiles; The mahjong tile model data is imported into the Blender tool, and a mahjong simulation environment is built within the Blender tool to obtain a first subset of simulation data; and The mahjong tile model data is imported into the Isaac Sim tool, and a mahjong table simulation environment is built in the Isaac Sim tool to obtain a second subset of simulation data. A simulation dataset is formed based on the first subset of simulation data and the second subset of simulation data; Obtain a simulation dataset and a real dataset, wherein the amount of data in the real dataset is less than the amount of data in the simulation dataset; A mahjong tile detection model is obtained by training the simulation dataset and the real dataset, and a pose estimation model is obtained by training the simulation dataset. When an image to be detected is received, the image to be detected is input into the mahjong tile detection model, and the target detection box output by the mahjong tile detection model is obtained. The target detection box is used to select each mahjong tile. The image data within the target detection box is input into the pose estimation model, and the pose data corresponding to the mahjong tiles within each target detection box output by the pose estimation model are obtained.

2. The method for real-time pose estimation of mahjong tiles according to claim 1, characterized in that, The process of obtaining mahjong tile model data by performing a 3D scan of the target mahjong tiles includes: Initial model data was obtained by performing a 3D scan on the target mahjong set; Texture analysis was performed on the initial model data to identify the high-reflectivity areas; Texture filling is performed on the high-reflectivity areas to obtain mahjong tile model data.

3. The real-time pose estimation method for mahjong tiles according to claim 1, characterized in that, The process involves importing the mahjong tile model data into the Blender tool and constructing a mahjong simulation environment within the Blender tool to obtain a first subset of simulation data, including: Import the mahjong tile model data into the Blender tool, and configure rigid body properties for the mahjong tile model data using the physics engine; A hierarchical sampling mechanism was used to initialize the pose of the mahjong tile model data configured with rigid body properties; Background data for various mahjong interactive scenarios is generated using texture blending technology. A first lighting system is formed by configuring a planar light source with dynamically changing light intensity and a point light source with dynamically adjusted color temperature. Under the first lighting system, the first simulation data subset is obtained by acquiring simulated color images and simulated depth images of the mahjong tile model data under various background data after pose initialization by implementing a progressive sampling strategy on the camera system.

4. The real-time pose estimation method for mahjong tiles according to claim 1, characterized in that, The process involves importing the mahjong tile model data into the Isaac Sim tool and constructing a mahjong table simulation environment within the Isaac Sim tool to obtain a second subset of simulation data, including: Import the mahjong tile model data into the Isaac Sim tool, and configure rigid body properties for the mahjong tile model data using the physics engine; Background data for various mahjong interactive scenarios is generated using texture blending technology. A second lighting system is formed by configuring point light sources with dynamically changing light intensity, color temperature, and light source position; A mahjong table simulation environment is built in the Isaac Sim tool, and a regionalized random strategy is adopted in the mahjong table simulation environment to generate various rotation postures of mahjong tiles based on mahjong tile model data configured with rigid body properties. The tablecloth of the mahjong table simulation environment maintains optical consistency with the background data in each mahjong interaction scene by implementing a shadow projection system. Under the second lighting system, a second set of simulation data is obtained by acquiring simulated color images and simulated depth images of a mahjong table simulation environment with mahjong tiles in various rotation postures under various background data through a dynamic sampling strategy implemented on the vision acquisition system.

5. The real-time pose estimation method for mahjong tiles according to claim 1, characterized in that, The process of forming a simulation dataset based on the first subset of simulation data and the second subset of simulation data includes: The mahjong tiles in the first and second simulation data subsets are labeled with detection information. The labeling results include the tile classification and the two-dimensional rectangles surrounding the mahjong tiles. The mahjong tiles in the first and second simulation data subsets are used to construct a physical coordinate system with the center of the mahjong tile as the origin for pose information annotation. The pose information annotation results include the three-dimensional translation vector and the three-dimensional rotation matrix of the mahjong tile. A simulation dataset is formed based on the first and second subsets of simulation data that have been annotated with detection and pose information.

6. The real-time pose estimation method for mahjong tiles according to claim 5, characterized in that, The method further includes: When annotating detection information, detect the quadrilaterals projected onto the image plane by each mahjong tile; When the quadrilateral is an inverted quadrilateral, or when the area of ​​the quadrilateral is less than a preset pixel value, the corresponding mahjong tile is removed from the detection information labeling.

7. The real-time pose estimation method for mahjong tiles according to claim 1, characterized in that, The step of receiving an image to be detected, inputting the image to be detected into the mahjong tile detection model, and obtaining the target detection bounding box output by the mahjong tile detection model includes: Upon receiving an image to be detected, the image is input into the mahjong tile detection model, and the initial detection box output by the mahjong tile detection model is obtained. Initial detection boxes with a confidence level greater than a preset confidence threshold are selected as target detection boxes.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the mahjong tile real-time pose estimation method as described in any one of claims 1 to 7.

9. A computer storage medium, characterized in that, The storage medium stores executable instructions, which, when executed by a processor, cause the processor to perform the real-time pose estimation method for mahjong tiles as described in any one of claims 1 to 7.

Images