Intelligent auxiliary building method based on solid tenon-and-mortise building blocks and mobile terminal

Abstract

The invention discloses an intelligent auxiliary building method based on entity tenon-and-mortise building blocks and a mobile terminal, and the mobile terminal in the method can accurately recognize building block information based on unique special marks on the surface of each building block, quickly position abnormal building blocks with deviations in position or orientation, and visually display an adjustment process in an animation form. By implementing the application, the user can obtain phased visual feedback and dynamic construction guidance by means of instant difference detection and visual guidance. The user operation threshold is obviously reduced, and the building efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to home entertainment product software, and more particularly to an intelligent auxiliary building method and mobile terminal based on physical mortise and tenon building blocks. Background Art

[0002] As a traditional construction toy with a long history, mortise and tenon building blocks are widely popular due to their exquisite structural design and unique connection method.

[0003] In related technologies, building instructions for mortise and tenon building blocks are mainly provided through paper instructions or video tutorials. Paper instructions usually contain a series of static images arranged according to the building steps, and users need to refer to the images to complete the splicing of the building blocks. Video tutorials, on the other hand, dynamically display the building process, and users can follow the rhythm of the video. In addition, some manufacturers have also developed simple apps that provide digital building guides through real-time rendering of preset building block component models or simple AR technology.

[0004] However, mortise and tenon building blocks, which interlock through precise concave and convex joints to form a stable three-dimensional structure, often require specific sequences, angles, and precise positioning during construction, creating a high technical barrier for users, especially children whose spatial cognition is still developing. These technologies are unable to effectively identify problems during construction and require users to frequently switch their attention between physical manipulation and pictorial understanding. When faced with difficulties, users can only make adjustments through repeated attempts or by comparing the final product. Summary of the Invention

[0005] The present application provides an intelligent auxiliary construction method and a mobile terminal based on physical mortise and tenon building blocks, which are used to provide error operation query and dynamic construction guidance.

[0006] In a first aspect, the present application provides an intelligent assisted building method based on physical mortise and tenon building blocks, which is applied to a mobile terminal. The method includes: in the process of using physical mortise and tenon building blocks to build a target module, a first block layout image of a built block pile is collected by a camera of the mobile terminal; a special mark different from other blocks is fixed on the surface of each block according to a preset orientation relationship; based on the special mark of each block in the first block layout image, each block in the block pile is compared with the corresponding block in the preset three-dimensional digital model of the target module, and the blocks in the block pile with inconsistent position coordinates and orientation information are treated as abnormal blocks a set of abnormal blocks; in response to a user clicking on a first abnormal block in the abnormal block set highlighted on a first block layout image, determining, based on the preset three-dimensional digital model, an order of removing associated blocks in the block stack that restrict adjustment of the first abnormal block; the position coordinates of the first abnormal block are consistent with those of the corresponding block but the orientation information is inconsistent; and displaying on the mobile terminal an animation of correcting the first abnormal block in the block stack, the animation comprising at least a first animated image of removing the associated blocks in the removing order and a second animated image of correcting the first abnormal block with an arrow indicating a rotation direction and an angle value.

[0007] In the above-mentioned embodiment, the mobile terminal can capture the layout of the building blocks during the construction process through a camera and accurately identify them based on the unique markings on each block's surface. By comparing the actual building block stack with the preset three-dimensional digital model, it can quickly locate abnormal blocks with deviations in position or orientation, and intuitively display the adjustment sequence and specific adjustment methods through animation. This instant difference detection and visual guidance solution greatly reduces the difficulty of users in troubleshooting during the building process, improving construction efficiency and accuracy.

[0008] In combination with some embodiments of the first aspect, in some embodiments, based on the type of building blocks corresponding to each identified special mark, a one-to-one mapping relationship between the actual mark and the preset building blocks in the preset three-dimensional digital model is established; based on the position and deformation characteristics of the special mark in the image, combined with the camera parameters of the mobile terminal, the position coordinates and orientation information of each building block in the three-dimensional space are calculated; for the building blocks set as adjacent in the preset three-dimensional digital model, the relative position relationship between the special marks of the adjacent building blocks in the first building block layout image is calculated, and the relative position relationship includes the distance and angle relationship between the actual marks; if the relative position relationship between the special marks of a pair of adjacent building blocks exceeds a preset tolerance range, the pair of adjacent building blocks are added to the abnormal building block set at the same time, and the preset tolerance range is a pre-set allowable position deviation threshold, which is usually determined according to the building block size and splicing accuracy requirements.

[0009] In the above embodiment, the mobile terminal achieves precise calculation of the spatial position of building blocks and strict control of their relative positional relationships. For blocks that are preset as adjacent, a one-to-one mapping relationship is established between the actual markings and the preset model. The relative positional relationship between their special markings, including the actual distance and angle, is calculated. Any adjacent building block pairs that fall outside the preset tolerance range are simultaneously marked as abnormal. This dual detection mechanism, based on the preset tolerance range, ensures the precision of the splicing between the building blocks and the structural stability.

[0010] In combination with some embodiments of the first aspect, in some embodiments, a criticality score for each abnormal building block is calculated based on the block position deviation, orientation angle error, and the number of adjacent building blocks, where the criticality score is higher for larger position deviations, larger orientation angle errors, and more adjacent building blocks; the abnormal building blocks are sorted from high to low according to the criticality scores to generate a set of abnormal building blocks with a priority correction order.

[0011] In the above embodiment, the mobile terminal calculates a criticality score for each abnormal block by comprehensively considering multiple dimensions, including the block's position deviation, orientation angle error, and the number of adjacent blocks. This multi-dimensional ranking mechanism prioritizes abnormal blocks with large impacts and high deviations, ensuring the efficiency of the overall correction process.

[0012] In combination with some embodiments of the first aspect, in some embodiments, in response to a user clicking on a first abnormal building block in a set of abnormal building blocks highlighted on a first building block layout image, mortise and tenon structure information of the first abnormal building block is obtained from a preset three-dimensional digital model, including a tenon position, a mortise position, and a connection method. Based on the mortise and tenon structure information of the first abnormal building block, a building block in the building block stack that is directly connected to the first abnormal building block is identified as a preliminary associated building block. Based on the connection position and connection strength between the preliminary associated building block and the first abnormal building block, associated building blocks that physically hinder the orientation adjustment of the first abnormal building block are determined, where the physical obstacles include meshing restrictions of the mortise and tenon structure and mutual interference in spatial positions. The connection strength includes contact area calculation and meshing depth judgment, where a larger contact area indicates a higher connection strength, and a larger depth indicates a tighter connection. Based on the disassembly direction and removal path of the mortise and tenon connection determined by the preset three-dimensional digital model, an order for removing the associated building blocks is determined.

[0013] In the above embodiment, the mobile terminal obtains information about the mortise and tenon joints of the abnormal building blocks, accurately identifying directly connected building blocks. Based on the connection location and strength, it determines the associated building blocks that will physically hinder adjustment. It then analyzes the contact area and engagement depth to accurately assess the connection strength, and, based on a pre-set 3D digital model, determines the optimal disassembly path. This intelligent analysis mechanism based on the mortise and tenon joint structure ensures the feasibility and safety of the adjustment process.

[0014] In combination with some embodiments of the first aspect, in some embodiments, after determining the order in which associated building blocks are to be removed, based on comparing the actual orientation of the first abnormal building block with orientation information of the first abnormal building block in a preset three-dimensional digital model, a rotation angle that needs to be adjusted for the first abnormal building block is calculated, where the rotation angle is a three-dimensional spatial difference represented by Euler angles; after determining the removal of the associated building blocks based on the rotation angle, the rotation direction and angle value of the first abnormal building block are corrected.

[0015] In the above embodiment, the mobile terminal calculates the specific rotation angle that needs to be adjusted for the abnormal building block in three-dimensional space through Euler angles, and based on this calculation result, rationally plans the order of removing related building blocks and the adjustment plan for the abnormal building block, ensuring the accuracy of the building block orientation adjustment.

[0016] In combination with some embodiments of the first aspect, in some embodiments, based on the geometric feature data of the snap-fit ​​mortise and tenon structure extracted from a preset three-dimensional digital model, associated building blocks containing special structures are identified in the building block pile, and the special structures include elastic snaps, rotating locking mechanisms, and sliding locking mechanisms; based on the precise positioning of the coordinate positions and orientations of these special structures in the building block pile, a flashing warning sign is added to the first animation image.

[0017] In the above-mentioned embodiment, the mobile terminal establishes an intelligent recognition and warning mechanism for building blocks with special structures. Based on the geometric feature data in the pre-set three-dimensional digital model, it can accurately identify and locate building blocks containing special structures such as elastic snaps, rotating locking mechanisms, and sliding locking mechanisms. By adding flashing warning icons to the animation, the user is intuitively reminded of these structures requiring special attention, effectively preventing damage to the building blocks caused by improper operation.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, after the user confirms that the first abnormal building block has been corrected, a second building block layout image of the corrected building block stack is captured by a camera of the mobile terminal; Based on the special mark of each building block in the second building block layout image, each building block in the building block pile is compared with the corresponding building block in the preset three-dimensional digital model of the target module; after confirming that the position coordinates and orientation information of the first abnormal building block are consistent with the corresponding building block in the preset three-dimensional digital model, the abnormal building block set after removing the corrected first abnormal building block is displayed on the mobile terminal.

[0019] In the above embodiment, the mobile terminal implements a closed-loop verification mechanism for the block correction process. By re-collecting the corrected block layout image and comparing it with the preset 3D digital model, it ensures that the abnormal block has indeed achieved the expected position and orientation requirements. This verification mechanism based on real-time feedback not only ensures the correction quality of each abnormal block, but also promptly updates the set of abnormal blocks to be corrected, providing clear feedback on the correction progress.

[0020] In a second aspect, an embodiment of the present application provides a mobile terminal, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, the one or more processors calling the computer instructions to enable the mobile terminal to execute the method described in the first aspect and any possible implementation of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a mobile terminal, enables the mobile terminal to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions. When the instructions are executed on a mobile terminal, the mobile terminal executes the method described in the first aspect and any possible implementation of the first aspect.

[0023] It is understandable that the mobile terminal provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. Thanks to the adoption of a block building status monitoring mechanism based on real-time camera capture and special marker recognition, the system can accurately identify abnormal blocks with inconsistent position coordinates and orientation information, and display adjustment plans through animation. This effectively solves the problems of the existing technology that cannot timely detect block installation deviations and the unintuitive adjustment methods, thereby realizing real-time monitoring and precise guidance of the block building process.

[0025] 2. Due to the use of three-dimensional spatial mapping based on special markers and the calculation mechanism of the relative position relationship of adjacent building blocks, the system can accurately obtain the spatial position information of each building block and strictly control the assembly accuracy between adjacent building blocks, realizing high-precision positioning and quality control during the building block construction process.

[0026] 3. Due to the use of special structure recognition and flashing warning mechanism based on geometric feature data, the system can accurately identify and locate building blocks with special connection structures and provide eye-catching visual reminders, effectively solving the problem in the existing technology that special structure building blocks are easily damaged due to improper operation, and thus realizing intelligent protection and safety guidance for special structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of an intelligent auxiliary building method based on physical mortise and tenon building blocks in an embodiment of the present application; Figure 2 This is another flow chart of an intelligent assisted building method based on physical mortise and tenon building blocks in an embodiment of the present application; Figure 3 It is a schematic diagram of a physical device structure of a mobile terminal in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms used in the following examples of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification of the present application, the singular expressions "a", "an", "above", "the", and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations of one or more of the listed items.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0030] See also Figure 1 , which is a flow chart of an intelligent assisted building method based on physical mortise and tenon building blocks in an embodiment of the present application.

[0031] S101. In the process of building a target module using physical mortise and tenon building blocks, a first building block layout image of a built building block stack is captured by a camera of a mobile terminal; a special mark different from other building blocks is fixed on the surface of each building block according to a preset orientation relationship.

[0032] Among them, solid mortise and tenon building blocks refer to solid building blocks with tenon and mortise structures, which enable the building blocks to be firmly combined through complementary concave and convex connections; target modules refer to the complete structure or partial structure that the user expects to build; mobile terminals refer to portable electronic devices, such as smartphones, tablets, and other devices used to run applications; preset orientation relationships refer to the fixed position and direction of each special mark on the surface of the corresponding building block, which is used to determine the correct orientation of the building block in three-dimensional space.

[0033] This step is performed when the user is building with physical mortise and tenon building blocks and requires auxiliary guidance from the mobile terminal. The user launches the auxiliary building application on the mobile terminal and points the mobile terminal's camera at the pile of built blocks. The mobile terminal's camera captures an image of the entire pile of blocks from an appropriate angle, which clearly captures the special markings on the surface of each block. These special markings are fixed to the surface of each block in a predetermined position and orientation when the blocks are manufactured. The markings on each block are different from those on other blocks.

[0034] It is understood that the target module can be determined in a variety of ways: In some embodiments, the user can independently select the target module on the mobile terminal.

[0035] In other embodiments, based on a first block layout image captured by a mobile terminal camera, special marks on the surface of each mortise and tenon block in the image can be identified, and matching can be performed with a preset block layout scheme based on the special marks identified in the first block layout image. The stage in the preset building scheme whose matching degree exceeds a predetermined threshold is determined to be a target module. The target module is a building stage arranged in a standard building order in the preset building scheme. Each building stage has a specific block layout state, including a specifically marked block type, its spatial position coordinates, and orientation angle, which are not limited here.

[0036] It is understood that image acquisition can be achieved in a variety of ways: in some embodiments, the user opens the building assistance software on the mobile terminal, actively triggers the shooting function, places the building block pile within the camera's field of view, ensures uniform lighting to clearly display the markings and outlines on the building block surface, and the software automatically saves the captured image and marks the shooting time; in other embodiments, the user turns on the device's automatic acquisition mode, and when it detects that the building block pile has moved or the building action is paused, the camera automatically starts shooting. It is also possible to expand the shooting angle through external auxiliary equipment or pre-process the image to remove irrelevant edge areas, which are not limited here.

[0037] S102: Based on the special mark of each building block in the first building block layout image, compare each building block in the building block pile with the corresponding building block in the preset three-dimensional digital model of the target module, and identify the building blocks in the building block pile with inconsistent position coordinates and orientation information as an abnormal building block set.

[0038] Among them, the preset three-dimensional digital model represents the precise digital expression of the target module established in advance, including the special markings, geometric size parameters, ideal position, mortise and tenon structure position and connection relationship information with adjacent building blocks of each building block. For example, the model will clearly indicate the coordinates, orientation and mortise and tenon connection method of each building block in three-dimensional space with surrounding building blocks.

[0039] This step is performed after the first building block layout image is successfully acquired. The mobile terminal identifies the type, relative position and orientation of each special mark in the first building block layout image based on the mark positioning technology.

[0040] Since special markers are attached to the surface of the building blocks in a predetermined fixed position and orientation, the precise position coordinates and spatial orientation of each building block in three-dimensional space are determined using spatial transformation relationships based on the deformation, size, and orientation of the special markers in the two-dimensional image. The spatial transformation relationship refers to the mathematical relationship that converts the marker information in the two-dimensional image into three-dimensional spatial coordinates and orientation, which is usually implemented based on camera calibration parameters.

[0041] The pre-set 3D digital model is called up, and the position coordinates and orientation information of the actual blocks in the block pile are compared one by one with the corresponding blocks in the pre-set model. If a block's actual position coordinates or orientation deviates from the ideal state in the pre-set model, and the deviation exceeds a preset threshold, the block is marked as an abnormal block and added to the abnormal block set. The preset threshold is reasonably determined based on the tolerance design of the block size and the mortise and tenon structure (typically, the position error does not exceed 10% of the minimum block size, and the angular error does not exceed 15 degrees). After all blocks are compared, a set containing all abnormal blocks is generated.

[0042] It is understandable that there are multiple ways to determine abnormal blocks: In some embodiments, the captured image is first divided into multiple area grids, each grid corresponding to a specific spatial position in a preset three-dimensional digital model; based on the special mark of each building block in the first building block layout image, the grid position of each mark is recorded; based on the expected area where each building block should appear in the preset three-dimensional digital model, the actual area of ​​the special mark is compared with the expected area. If a mark appears in an unexpected area, the corresponding building block is directly marked as having a positional abnormality; for the building blocks located in the correct functional area, its relative positional relationship with the adjacent building blocks is checked; then, the directional characteristics of the special mark (such as asymmetric patterns, direction indicator lines) are used to determine the spatial orientation of each building block, and the angular difference between the actual orientation and the preset orientation is calculated; if the angular deviation exceeds a threshold, the building block is added to the abnormal building block set and marked as having an orientation error.

[0043] In other embodiments, a preset three-dimensional digital model is loaded into memory as a standard reference model. The model contains the precise position coordinates (X, Y, Z values) and orientation angles (pitch, yaw, and roll angles) of each building block in an ideal state. All building blocks identified in the first building block layout image are traversed, and for each building block, the corresponding building block instance in the preset model is searched. For each pair of corresponding relationships, the position deviation value (by calculating the spatial distance between the two points) and the orientation deviation value (by calculating the angular difference between the two orientation vectors) between the actual building block and the ideal building block are calculated. Based on a predetermined position deviation threshold (e.g., 8 mm) and orientation deviation threshold (e.g., 12 degrees), when a building block's position deviation exceeds the position threshold or the orientation deviation exceeds the angle threshold, the building block is determined to be abnormal. For each building block determined to be abnormal, the deviation type (abnormal position, abnormal orientation, or both), the specific deviation value, and the offset direction relative to the ideal state are recorded. The size change of the marker can also be used to evaluate the distance deviation of the building block, the degree of tilt of the building block can be analyzed by the degree of deformation of the marker, and the motion trajectory model of the building block can be established using a sequence of images taken multiple times to determine abnormalities. This is not limited here.

[0044] S103 : In response to a user clicking on a first abnormal block in the abnormal block set highlighted on the first block layout image, determining, based on the preset three-dimensional digital model, a removal order of associated blocks in the block stack that restrict adjustment of the first abnormal block.

[0045] The associated blocks refer to blocks in the block pile that restrict the adjustment of the first abnormal block. That is, if these blocks are not taken out first, the direction of the first abnormal block cannot be correctly adjusted.

[0046] This step is performed after the abnormal block set has been detected and highlighted on the interface, and the user has selected a specific abnormal block (the first abnormal block) by clicking.

[0047] First, a first building block layout image is displayed on the screen of the mobile terminal, and all abnormal building blocks are highlighted in the image in a highlighted color or special mark manner.

[0048] After the user clicks on a first abnormal building block, if the position coordinates of the first abnormal building block are consistent with those of a corresponding building block in a preset three-dimensional digital model but the orientation information is inconsistent, the mobile terminal will identify all associated building blocks that restrict direct adjustment of the first abnormal building block based on the spatial position and connection status of the first abnormal building block in the preset three-dimensional digital model. These associated building blocks are usually building blocks stacked above the first abnormal building block or connected to it by mortise and tenon joints; and calculate a reasonable order for removing these associated building blocks based on the connection sequence of the mortise and tenon joints.

[0049] If the position coordinates of the first abnormal block are inconsistent with those of the corresponding block in the preset three-dimensional digital model, after the user clicks on the first abnormal block, the mobile terminal will first determine whether the currently placed block is of the wrong type based on the type information of the block that should be placed at that location in the preset three-dimensional digital model. If it is the wrong type of block, the correct position where the block should be placed will be marked on the interface, and the correct type of block that should be placed at the current location will be displayed. For mortise and tenon joint blocks, due to their structural characteristics, they can only be fully in place or unable to be connected during installation. The system mainly focuses on the correctness of the block type and the consistency of the orientation information, without considering the possibility of position offset.

[0050] In some embodiments, based on a preset three-dimensional digital model, the mortise and tenon structure information of the first abnormal building block is first extracted, including a table of tenon parameters, a table of mortise and tenon joint parameters, and a table of connection pairings (recording the type and strength level of the connection). Then, based on this mortise and tenon structure information, the mortise and tenon joint table is queried to identify blocks directly connected to the first abnormal building block in the actual building block pile and mark them as preliminarily associated blocks. These preliminarily associated blocks are further analyzed. Based on their connection positions and connection strengths with the first abnormal building block, the preset connection information is spatially mapped to the actual building block pile coordinate system. Three-dimensional transformation simulation is then used to identify associated blocks that physically hinder the orientation adjustment of the first abnormal building block. Connection strength is assessed by calculating contact area and determining meshing depth: the larger the contact area, the higher the connection strength, and the greater the depth, the tighter the connection. Physical obstacles include meshing restrictions of the mortise and tenon structure, i.e., the tight connection between the tenon and tenon joint may restrict the building block's rotation, and spatial interference, i.e., the presence of other blocks may hinder the first abnormal building block's rotation path. Based on the disassembly direction and removal path of the relevant mortise and tenon connection types determined in the preset three-dimensional digital model, the removal order of the associated building blocks is determined.

[0051] In other embodiments, the building block pile is first divided into different structural levels, and the assembly tree structure of the building blocks is extracted from a preset model. With the first abnormal building block as the root node, all parent building blocks (i.e., blocks located above it or closely connected to it) are traversed upward to generate a removal list in order from farthest to nearest (the farthest building blocks are removed first).

[0052] It is understandable that the order of taking out the associated building blocks can be determined in a variety of ways, which are not limited here.

[0053] S104: Displaying an animation of correcting the first abnormal building block in the building block pile on the mobile terminal, wherein the animation includes at least a first animation image of removing the associated building blocks in the removal order and a second animation image of correcting the first abnormal building block with a rotation direction arrow indication and an angle value.

[0054] This step is performed after determining the order in which the associated blocks must be removed. The mobile terminal generates dynamic visual guidance based on the calculated removal order and target orientation. Specifically, the mobile terminal first displays the dynamic process of each associated block being separated and removed from the block pile in sequence, following the predetermined removal order, while simultaneously displaying a highlighted prompt at the corresponding position in the actual block pile. Once all associated blocks have been removed and displayed in the first animated image, the mobile terminal superimposes a rotation direction arrow and specific angle value at the current position of the first abnormal block. Throughout the entire animation display, the operation prompt text is synchronously displayed on the screen.

[0055] In some embodiments, animation display can be achieved in a variety of ways: Alternatively, augmented reality (AR) technology can be used to spatially align the virtual animation with the actual building block stack. The mobile terminal uses the real-time camera image as the background, precisely overlaying virtual elements such as the removal animation and rotation instructions onto the corresponding physical building block positions. The animation plays in a step-by-step manner, pausing after each step to confirm completion before continuing to the next step.

[0056] Optionally, the mobile terminal generates a pre-rendered 2D animation that comprehensively displays the operation process from different perspectives (e.g., top view, side view, perspective view, etc.). The user can use on-screen control buttons to perform interactive operations such as zooming in, pausing, and replaying, controlling the animation playback according to their own understanding and rhythm. This is not limited here.

[0057] In some embodiments, after the correction operation of the first abnormal building block is completed, it is necessary to verify the correction effect and continue to guide the user to complete the correction of the remaining abnormal building blocks. After the user feedbacks that the correction operation of the first abnormal building block has been completed through the confirmation completion mark on the mobile terminal interface, the mobile terminal starts the camera module. The mobile terminal first displays the viewfinder auxiliary interface. When it is detected that the building block pile has completely entered the viewfinder range and is clearly visible, the mobile terminal automatically or after the user confirms the acquisition of the second building block layout image. The same technical path as the first building block layout image analysis is adopted to identify the building block type, position and orientation corresponding to each special mark in the second building block layout image. After confirming that the first abnormal building block is correct, the mobile terminal executes a complete building block layout comparison process based on the second building block layout image, identifies all buildings whose position coordinates or orientation information are inconsistent with the preset three-dimensional digital model, and removes the corrected first abnormal building block from the abnormal building block set, and finally generates an updated abnormal building block set.

[0058] In the above embodiment, unique markings fixed on the surface of each building block serve as the basis for identification, accurately tracking the spatial state of each building block, detecting deviations in the block's position or orientation, and visually identifying abnormal blocks in the building block layout image. By analyzing all associated blocks that affect adjustment, an animation demonstrates the order in which the associated blocks should be removed and the specific adjustment methods for abnormal blocks. This instant, staged visual feedback and dynamic guidance mechanism allows users to clearly understand the correct state and adjustment method for each building step, significantly reducing the trial-and-error cost and learning threshold when building complex mortise and tenon structures.

[0059] In actual applications, due to the user's unclear understanding of the mortise and tenon structure, when independently selecting abnormal blocks for modification, the selected abnormal blocks lack intuitive priority reflection, which may cause the selected abnormal blocks to be relatively marginal in the structure. During the entire modification process, multiple selections and modifications are required, affecting the efficiency of error correction.

[0060] See also Figure 2 , is another flow chart of an intelligent assisted building method based on physical mortise and tenon building blocks in an embodiment of the present application.

[0061] S201 : In the process of using physical mortise and tenon building blocks to build a target module, a first building block layout image of a built building block stack is captured by a camera of a mobile terminal.

[0062] Step S201 is similar to step S101 and will not be described in detail here.

[0063] S202: Establish a one-to-one mapping relationship between the actual mark and the preset building block in the preset three-dimensional digital model according to the building block type corresponding to each recognized special mark.

[0064] The mobile terminal processes the captured first image of the building block layout and uses computer vision technology to identify special markings within the image. After identifying these markings, the terminal queries a pre-set database of markings and building block types to determine the specific building block type, size, and functional attributes corresponding to each marking. Subsequently, the terminal accesses a pre-set 3D digital model library to identify the pre-set building block model associated with the target module. Based on the recognition results, a one-to-one mapping relationship is established between each special marking in the actual image and the corresponding building block in the pre-set 3D digital model. This mapping relationship records the building block's unique identifier, type information, and ideal position and orientation data within the model.

[0065] In some embodiments, the special mark includes a preset color and shape combination, where the shape is a unique combination of 4 to 6 geometric figures, and each shape uses a high-contrast color to improve visual recognition efficiency.

[0066] S203 : Based on the position and deformation characteristics of the special mark in the image and in combination with the camera parameters of the mobile terminal, calculate the position coordinates and orientation information of each building block in the three-dimensional space.

[0067] The mobile terminal corrects the image based on the camera parameters to restore the true geometric features of the marker. Using this 2D image information and the camera's position and angle parameters during capture, and applying the principles of spatial geometric transformation, the specific position coordinates of each brick in 3D space, as well as the rotation angles around different coordinate axes, are determined based on the fixed positional relationship between the marker and the brick itself. This allows the brick's orientation to be determined. For example, the distance of the brick can be determined by analyzing the size of the marker, while the tilt angle can be determined by analyzing the deformation of the marker.

[0068] S204: For adjacent building blocks in the preset three-dimensional digital model, calculate the relative position relationship between special marks of the adjacent building blocks in the first building block layout image.

[0069] Based on the adjacent pairs of blocks defined in the pre-set 3D digital model, the pixel coordinates of the special markers corresponding to each pair of adjacent blocks in the established mapping relationship between the actual blocks and the model blocks are found in the first block layout image. These adjacent relationships are reflected in the presence of a mortise and tenon joint, surface contact, or a relative positional relationship within a specific distance between the two blocks. By calculating the pixel distance between the centers of the two markers and the angle between the connecting line and the image coordinate system, the relative position difference between them in the 2D image is determined.

[0070] In some embodiments, all adjacent pairs of blocks in a preset model can be traversed, the marked coordinates of each pair of blocks in the image can be obtained according to the mapping relationship, the pixel spacing can be calculated using the Euclidean distance formula, and the angle corresponding to the coordinate difference can be calculated using the inverse tangent function.

[0071] S205: If the relative position relationship between the special marks of a pair of adjacent building blocks exceeds a preset tolerance range, the pair of adjacent building blocks are simultaneously added to the abnormal building block set.

[0072] This step is performed after calculating the relative position relationship of adjacent building blocks. The mobile terminal obtains the standard relative position relationship data of each pair of adjacent building blocks under ideal conditions from the preset three-dimensional digital model, including parameters such as distance and angle, and simultaneously obtains the preset tolerance range value; the relative position relationship (such as distance and angle) of the special marks of each pair of adjacent building blocks calculated in the first building block layout image is compared with the preset tolerance range; if the deviation between the actual relative position relationship of a pair of adjacent building blocks and the standard data exceeds the preset tolerance range (such as the distance deviation is greater than 5 mm or the angle deviation is greater than 10 degrees), the pair of building blocks will be marked as abnormal and added to the abnormal building block set.

[0073] In some embodiments, the identification and addition of abnormal building blocks can be achieved in a variety of ways: Optionally, different tolerance ranges can be set for different types of adjacent relationships, for example, a smaller tolerance can be set for building blocks with mortise and tenon joints, and a larger tolerance can be set for building blocks with only surface contact relationships, and then each pair of building blocks is evaluated separately to see whether the deviation exceeds the tolerance range of the corresponding type; Optionally, deviations in multiple dimensions can be considered simultaneously, including horizontal distance deviation, vertical distance deviation, and angle deviation. Only when the deviation in a certain dimension exceeds the corresponding tolerance value, the pair of building blocks is added to the abnormal set. This is not limited here. S206: Calculate the criticality score of each abnormal building block based on the building block position deviation, orientation angle error, and the number of adjacent building blocks.

[0074] After identifying the set of abnormal blocks, each block in the set is evaluated from multiple perspectives: its position deviation (the distance between its actual and ideal positions), orientation angle error (the angular difference between its actual and ideal orientations), and the number of blocks directly connected to it are calculated. A weighted summation is then used, with weights assigned to each of the position deviation, angle error, and number of adjacent blocks (e.g., a weight of 0.4 for position deviation, 0.3 for angle error, and 0.3 for number of adjacent blocks). The weighted total score is then calculated as the criticality score. Generally, blocks with larger position deviations, larger angle errors, and more adjacent blocks receive higher criticality scores, indicating that these blocks are more critical to the overall structure and require priority adjustment.

[0075] In some embodiments, the criticality score can be calculated in a variety of ways: Optionally, a position coefficient can be assigned based on the position of the building blocks in the overall structure, with a higher position coefficient for bottom-level support blocks and a lower position coefficient for top-level decorative blocks. This coefficient is then multiplied by the base criticality score to obtain the final criticality score. Optionally, the specificity of the building block type can be taken into account, and additional importance scores can be assigned to certain key functional blocks (such as load-bearing blocks or blocks connecting special structures) to increase their criticality scores. It is understood that other methods can also be used to calculate the criticality score, and this is not limited here.

[0076] S207 , sorting the abnormal building blocks from high to low according to the criticality scores, and generating an abnormal building block set including a priority correction order.

[0077] Sort all the blocks in the abnormal block set in descending order according to their criticality scores, and based on this sorting result, generate a new abnormal block set with a clear priority correction order.

[0078] In some embodiments, given the strict assembly order requirements for mortise and tenon building blocks, the mobile terminal identifies the first incorrectly placed building block according to the standard building sequence. If only one building block is faulty, adjustment guidance is provided for that building block. If multiple building blocks are faulty, the locations of all faulty building blocks are displayed, with the first building block highlighted.

[0079] S208 : In response to a user clicking on a first abnormal block in the abnormal block set highlighted on the first block layout image, determining, based on the preset three-dimensional digital model, a removal order of associated blocks in the block stack that restrict adjustment of the first abnormal block.

[0080] Step S208 is similar to step S103 and will not be described in detail here.

[0081] S209 , based on the geometric feature data of the snap-fit ​​mortise and tenon structure extracted from the preset three-dimensional digital model, identifying associated building blocks containing special structures in the building block pile.

[0082] The mobile terminal first extracts detailed geometric feature data for all snap-on mortise and tenon joints from a pre-set 3D digital model, including information such as the snap's shape, size, connection direction, and disassembly method. Then, by comparing the first block layout image, the mobile terminal directly queries a pre-set database of special structures by block model and category. The mobile terminal then identifies and marks the blocks containing these special snap-on mortise and tenon joints among the associated blocks to be removed. These special joints may require specific manipulations for proper disassembly, such as pressing a specific location, sliding in a specific direction, or rotating, differing from standard stacking or simple mortise and tenon joints.

[0083] S210: Determine whether an associated building block containing a special structure is identified in the building block stack.

[0084] Check the list of identified associated blocks to see if there are any blocks marked as containing snap-fit ​​mortise and tenon structures.

[0085] If there are special structure blocks, execute step S211; If the special structure building block does not exist, execute step S212.

[0086] S211. Add a flashing warning sign to the first animation image based on the precise positioning of the coordinate positions and orientations of these special structures in the building block pile.

[0087] Based on the determined associated building blocks containing special structures, a flashing warning sign is superimposed at the corresponding position in the first animated image according to the three-dimensional coordinates and orientation of the building blocks with special structures. Based on the extracted detailed geometric feature data of the snap-on mortise and tenon structure, the warning sign is usually also accompanied by text prompts or voice prompts, detailing how to correctly operate these special structures to avoid damage to the building blocks or difficulties caused by improper operation during the disassembly and assembly process.

[0088] S212: Displaying an animation of correcting the first abnormal building block in the building block pile on the mobile terminal, wherein the animation includes at least a first animation image of removing the associated building blocks in the removal order and a second animation image of correcting the first abnormal building block with a rotation direction arrow indication and an angle value.

[0089] Step S212 is similar to step S104 and will not be described in detail here.

[0090] In the embodiment of the present application, due to the comprehensive consideration of multi-dimensional factors such as the position deviation of the building blocks, the orientation angle error and the number of adjacent building blocks, a criticality score is constructed to quantify the impact level of abnormal building blocks from three aspects: the degree of spatial misalignment, the accuracy of directional deviation, and the complexity of structural association. The subjective judgment is converted into an objective data-driven priority sequence, which effectively solves the blindness problem in traditional manual inspection and avoids repeated dismantling and modification caused by prioritizing low-impact errors.

[0091] The following describes the mobile terminal in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , is a schematic diagram of a physical device structure of a mobile terminal in an embodiment of the present application.

[0092] It should be noted that Figure 3 The structure of the mobile terminal shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0093] like Figure 3As shown, the mobile terminal includes a CPU 301, which can perform various appropriate actions and processes according to the programs stored in the ROM 302 or the programs loaded from the storage part 308 into the RAM 303, such as executing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An I / O interface 305 is also connected to the bus 304.

[0094] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, push button switches, and the like; an output section 307 including a liquid crystal display (LCD), an audio output device, indicator lights, and the like; a storage section 308 including a hard disk and the like; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 310 as needed, so that computer programs read from the removable media can be installed in the storage section 308 as needed.

[0095] In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, an embodiment of the present invention includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 309 and / or installed from the removable medium 311. When the computer program is executed by the CPU 301, the various functions defined in the present invention are performed.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings.

[0097] Specifically, the mobile terminal of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, an intelligent auxiliary building method based on physical mortise and tenon building blocks provided by the above embodiment is implemented.

[0098] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the mobile terminal described in the above embodiments, or may exist independently and not be incorporated into the mobile terminal. The storage medium carries one or more computer programs, and when executed by a processor of the mobile terminal, the mobile terminal implements the intelligent assisted construction method based on physical mortise and tenon building blocks provided in the above embodiments.

[0099] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0100] As used in the above embodiments, the term “when” may be interpreted to mean “if” or “after” or “in response to determining that” or “in response to detecting that”, depending on the context. Similarly, the phrases “upon determining that” or “if (stated condition or event) is detected” may be interpreted to mean “if determining that” or “in response to determining that” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

Claims

1. An intelligent auxiliary construction method based on physical mortise and tenon building blocks, characterized in that: Applied to a mobile terminal, the method includes: In the process of building a target module using physical mortise and tenon building blocks, a first building block layout image of the built building block stack is captured by a camera of a mobile terminal; a special mark different from other building blocks is fixed on the surface of each building block according to a preset orientation relationship; comparing each block in the block pile with a corresponding block in a preset three-dimensional digital model of the target module based on a special mark of each block in the first block layout image, and identifying blocks in the block pile with inconsistent position coordinates and orientation information as an abnormal block set; In response to a user clicking on a first abnormal block in the abnormal block set highlighted on the first block layout image, determining, based on the preset three-dimensional digital model, an order of removing associated blocks from the block stack that restrict adjustment of the first abnormal block; the first abnormal block and the corresponding block have consistent position coordinates but inconsistent orientation information; An animation of correcting the first abnormal building block in the building block pile is displayed on the mobile terminal, which at least includes a first animation image of removing the associated building blocks in the removal order and a second animation image of correcting the first abnormal building block with a rotation direction arrow indication and an angle value.

2. The method according to claim 1, characterized in that The step of comparing each block in the block pile with a corresponding block in a preset three-dimensional digital model of the target module based on the special mark of each block in the first block layout image, and treating blocks in the block pile with inconsistent position coordinates and orientation information as an abnormal block set specifically includes: According to the building block type corresponding to each identified special mark, a one-to-one mapping relationship between the actual mark and the preset building block in the preset three-dimensional digital model is established; Based on the position and deformation characteristics of the special markers in the image and the camera parameters of the mobile terminal, the position coordinates and orientation information of each building block in three-dimensional space are calculated; For building blocks set as adjacent in the preset three-dimensional digital model, calculating the relative position relationship between special marks of the adjacent building blocks in the first building block layout image, the relative position relationship including the distance and angle relationship between the actual marks; If the relative position relationship between the special marks of a pair of adjacent building blocks exceeds the preset tolerance range, the pair of adjacent building blocks will be added to the abnormal building block set at the same time. The preset tolerance range is a pre-set allowable position deviation threshold, which is usually determined according to the building block size and splicing accuracy requirements.

3. The method according to claim 2, characterized in that The method further comprises: Calculate the criticality score of each abnormal block based on the block position deviation, orientation angle error, and the number of adjacent blocks; Sort the abnormal building blocks from high to low according to their criticality scores to generate a set of abnormal building blocks with a priority correction order.

4. The method according to claim 1, wherein The step of determining, in response to a user clicking on a first abnormal block in the abnormal block set highlighted on the first block layout image, a removal order of associated blocks in the block stack that restrict adjustment of the first abnormal block based on the preset three-dimensional digital model, specifically includes: In response to a user clicking on a first abnormal building block in the abnormal building block set highlighted on the first building block layout image, obtaining mortise and tenon structure information of the first abnormal building block from a preset three-dimensional digital model, including a tenon position, a mortise position, and a connection method thereof; Based on the mortise and tenon structure information of the first abnormal building block, identifying a building block in the building block stack that is directly connected to the first abnormal building block as a preliminary associated building block; Determining, based on the connection position and connection strength between the preliminary associated building block and the first abnormal building block, an associated building block that physically hinders the orientation adjustment of the first abnormal building block, wherein the physical hindrance includes meshing restrictions of the mortise and tenon structure and mutual interference in spatial positions, and the connection strength includes contact area calculation and meshing depth; Based on the disassembly direction and removal path of the mortise and tenon joint determined by the preset three-dimensional digital model, the removal order of the associated building blocks is determined.

5. The method according to claim 4, characterized in that The method further comprises: After determining the order in which the associated building blocks are to be removed, calculating a rotation angle that needs to be adjusted for the first abnormal building block based on a comparison between the actual orientation of the first abnormal building block and orientation information of the first abnormal building block in a preset three-dimensional digital model, where the rotation angle is a three-dimensional spatial difference represented by Euler angles; After the removal of the associated building block is determined based on the rotation angle, the rotation direction and angle value of the first abnormal building block are corrected.

6. The method according to claim 1, characterized in that Before the step of displaying on the mobile terminal an animation of correcting the first abnormal building block in the building block pile, which at least includes a first animated image of removing the associated building blocks in the removal order and a second animated image of correcting the first abnormal building block with a rotation direction arrow indication and an angle value, the method further includes: Based on geometric feature data of the snap-fit ​​mortise and tenon structure extracted from the preset three-dimensional digital model, identifying associated building blocks containing special structures in the building block pile, the special structures including elastic snaps, rotating locking mechanisms, and sliding locking mechanisms; Based on the precise positioning of the coordinate positions and orientations of these special structures in the building block pile, a flashing warning sign is added to the first animated image.

7. The method according to claim 1, characterized in that After displaying on the mobile terminal an animation of correcting the first abnormal building block in the building block pile, which at least includes a first animated image of removing the associated building blocks in the removal order and a second animated image of correcting the first abnormal building block with a rotation direction arrow indication and an angle value, the method further includes: After the user confirms that the first abnormal building block has been corrected, a second building block layout image of the corrected building block stack is captured by a camera of the mobile terminal; comparing each building block in the building block stack with a corresponding building block in a preset three-dimensional digital model of the target module based on a special mark of each building block in the second building block layout image; After confirming that the position coordinates and orientation information of the first abnormal building block are consistent with the corresponding building block in the preset three-dimensional digital model, the abnormal building block set after removing the corrected first abnormal building block is displayed on the mobile terminal.

8. A mobile terminal, characterized in that: The mobile terminal includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the mobile terminal to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on a mobile terminal, the mobile terminal is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a mobile terminal, the mobile terminal is enabled to execute the method according to any one of claims 1 to 7.

Images