Intelligent auxiliary building method based on entity mortise and tenon block and mobile terminal

Abstract

An intelligent auxiliary building method based on entity mortise and tenon block and a mobile terminal, in the method, the mobile terminal can accurately identify the block information based on the unique special mark of each block surface, quickly locate the abnormal block with position or orientation deviation, and visually display the adjustment process in the form of animation. By implementing the present application, users can obtain stage visual feedback and dynamic building guidance with the help of real-time difference detection and visual guidance. The user operation threshold is significantly reduced, and the building efficiency and accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to home entertainment product software, and particularly relates to an intelligent auxiliary building method based on entity mortise and tenon blocks and a mobile terminal. BACKGROUND

[0002] The mortise and tenon block, as a traditional building toy with a long history, is popular due to its exquisite structure design and unique connection mode.

[0003] In the related art, the building guidance of the mortise and tenon block is mainly performed through a paper instruction manual or a video tutorial. The paper instruction manual usually includes a series of static pictures arranged according to the building steps, and the user needs to complete the block splicing by referring to the pictures. The video tutorial dynamically displays the building process, and the user can follow the video rhythm to operate. In addition, some manufacturers have also developed simple APP applications to provide digital building guidelines through real-time rendering of preset block component models or simple AR technology.

[0004] However, the mortise and tenon blocks are connected by precise concave-convex mortise and tenon structures, forming a stable three-dimensional structure. The building process often needs to be operated according to a specific order, angle and accurate positioning, which has a high technical threshold for users, especially children whose spatial cognitive ability is still in the development stage. These technologies cannot effectively identify the problems in the building process and need the user to frequently switch attention between entity operation and graphic understanding. When facing building difficulties, the user can only adjust by repeatedly trying or comparing the final product. SUMMARY

[0005] The present application provides an intelligent auxiliary building method based on entity mortise and tenon blocks and a mobile terminal, which is used to provide error operation query and dynamic building guidance.

[0006] In a first aspect, the application provides an intelligent auxiliary building method based on entity mortise and tenon blocks, applied to a mobile terminal, the method comprising: in the process of building a target module using entity mortise and tenon blocks, capturing a first block layout image of a built block pile by a camera of the mobile terminal; each block surface is fixed with a special mark different from other blocks according to a preset orientation relationship; based on the special marks of each block in the first block layout image, comparing each block in the block pile with a corresponding block in a preset three-dimensional digital model of the target module, and taking the blocks in the block pile with inconsistent position coordinates and orientation information as an abnormal block set; in response to a user clicking a first abnormal block in the abnormal block set highlighted on the first block layout image, determining the removal order of associated blocks in the block pile that restrict the adjustment of the first abnormal block based on the preset three-dimensional digital model; the position coordinates of the first abnormal block and the corresponding block are consistent, but the orientation information is inconsistent; displaying an animation of correcting the first abnormal block in the block pile on the mobile terminal, wherein at least includes a first animation image of removing the associated blocks in the removal order and a second animation image of the corrected first abnormal block displayed with a rotation direction arrow indication and an angle value.

[0007] In the above embodiment, the mobile terminal can capture the layout state during the block building process through the camera, accurately identify each block surface based on the unique special mark, quickly locate the abnormal blocks with position or orientation deviation by comparing the actually built block pile with the preset three-dimensional digital model, and intuitively show the adjustment order and specific adjustment method through animation. This instant difference detection and visual guidance scheme greatly reduces the difficulty of users in checking errors during block building, and improves the building efficiency and accuracy.

[0008] In combination with some embodiments of the first aspect, in some embodiments, a one-to-one mapping relationship between the actual mark and the preset block in the preset three-dimensional digital model is established according to the block type corresponding to each special mark identified; based on the position and deformation characteristics of the special mark in the image, combined with the camera parameters of the mobile terminal, the position coordinates and orientation information of each block in the three-dimensional space are calculated; for the blocks set as adjacent in the preset three-dimensional digital model, the relative position relationship between the special marks of adjacent blocks in the first block layout image is calculated, including the distance and angle relationship between the actual marks; if the relative position relationship between the special marks of a pair of adjacent blocks exceeds a preset tolerance range, the pair of adjacent blocks are added to the abnormal block set at the same time, and the preset tolerance range is a pre-set allowed position deviation threshold, which is usually determined according to the block size and splicing accuracy requirements.

[0009] In the above embodiment, the mobile terminal realizes accurate calculation of the spatial position of the building blocks and strict control of the relative position relationship. For building blocks preset to be adjacent, a one-to-one mapping relationship between the actual markers and the preset model is established, the relative position relationship between the special markers is calculated, including the actual distance and angle, and the adjacent building blocks that exceed the preset tolerance range are marked as abnormal at the same time. This double detection mechanism based on the preset tolerance range ensures the splicing accuracy and structural stability of the building blocks.

[0010] In combination with some embodiments of the first aspect, in some embodiments, according to the position deviation amount, the orientation angle error and the number of adjacent building blocks, a key degree score of each abnormal building block is calculated, the larger the position deviation amount and the orientation angle error are, the higher the score is; the abnormal building blocks are sorted according to the key degree score from high to low to generate an abnormal building block set containing a priority correction order.

[0011] In the above embodiment, the mobile terminal comprehensively considers multiple dimensions such as the position deviation amount, the orientation angle error and the number of adjacent building blocks, and calculates the key degree score of each abnormal building block. This sorting mechanism based on multi-dimensional scoring enables the abnormal building blocks with a large impact range and high deviation degree to be processed preferentially, ensuring the efficiency of the overall correction process.

[0012] In combination with some embodiments of the first aspect, in some embodiments, in response to the user clicking a first abnormal building block in the highlighted abnormal building block set on the first building block layout image, the mortise and tenon structure information of the first abnormal building block is obtained from the preset three-dimensional digital model, including the position of the tenon, the position of the mortise and the connection mode thereof; based on the mortise and tenon structure information of the first abnormal building block, the building blocks directly connected to the first abnormal building block in the building block stack are identified as preliminary associated building blocks; based on the connection position and the connection strength of the preliminary associated building blocks and the first abnormal building block, the associated building blocks that constitute a physical obstruction to the orientation adjustment of the first abnormal building block are determined, the physical obstruction includes the engagement limitation of the mortise and tenon structure and the mutual interference of the spatial position, the connection strength includes the contact area calculation and the engagement depth judgment, the larger the contact area is, the higher the connection strength is, the larger the depth is, the tighter the connection is; based on the disassembly direction and the taking-out path of the mortise and tenon connection determined through the preset three-dimensional digital model, the taking-out order of the associated building blocks is determined.

[0013] In the above embodiment, the mobile terminal accurately identifies the directly connected building blocks by obtaining the mortise and tenon structure information of the abnormal building blocks, and determines the associated building blocks that will cause a physical obstruction to the adjustment based on the connection position and the connection strength. Then, by analyzing the contact area and the engagement depth, the connection strength is accurately evaluated, and the optimal disassembly path is determined in combination with the preset three-dimensional digital model. This intelligent analysis mechanism based on the mortise and tenon structure ensures the feasibility and safety of the adjustment process.

[0014] In some embodiments of the first aspect, after determining the taking-out sequence of the associated blocks, the rotation angle that the first abnormal block needs to adjust is calculated based on a comparison between the actual orientation of the first abnormal block and the orientation information of the first abnormal block in the preset three-dimensional digital model, the rotation angle being a three-dimensional space difference represented by Euler angles; after determining the taking-out of the associated blocks based on the rotation angle, the rotation direction and angle value of the first abnormal block are corrected.

[0015] In the above embodiments, the mobile terminal calculates the specific rotation angle that the abnormal block needs to adjust in the three-dimensional space through Euler angles, and based on this calculation result, reasonably plans the taking-out sequence of the associated blocks and the adjustment scheme of the abnormal blocks, ensuring the accuracy of the block orientation adjustment.

[0016] In some embodiments of the first aspect, based on the geometric feature data of the buckle-type mortise-tenon structure extracted from the preset three-dimensional digital model, the associated blocks containing special structures are identified in the block pile, the special structures including elastic buckles, rotary locking mechanisms, and sliding locking mechanisms; based on the accurate positioning of the coordinate positions and orientations of these special structures in the block pile, a flashing warning mark is added in the first animation image.

[0017] In the above embodiments, the mobile terminal establishes an intelligent identification and warning mechanism for special structure blocks. Based on the geometric feature data in the preset three-dimensional digital model, the blocks containing special structures such as elastic buckles, rotary locking mechanisms, and sliding locking mechanisms can be accurately identified and positioned. By adding a flashing warning mark in the animation, the user is intuitively reminded of these structures that need special attention, effectively avoiding damage to the blocks due to improper operation.

[0018] In some embodiments of the first aspect, after the user confirms that the first abnormal block has been corrected, a second block layout image of the corrected block pile is collected by the camera of the mobile terminal;

[0019] Based on the special marks of each block in the second block layout image, the blocks in the block pile are compared with the corresponding blocks in the preset three-dimensional digital model of the target module; after confirming that the position coordinates and orientation information of the first abnormal block are consistent with those of the corresponding block in the preset three-dimensional digital model, the set of abnormal blocks after the removal of the corrected first abnormal block is displayed on the mobile terminal.

[0020] In the above embodiment, the mobile terminal implements a closed-loop verification mechanism of the block correction process, and the corrected block layout image is collected again and compared with the preset three-dimensional digital model to ensure that the abnormal blocks indeed reach the expected position and orientation requirements. The verification mechanism based on real-time feedback not only ensures the correction quality of each abnormal block, but also updates the abnormal block set to be corrected in time, and provides clear correction progress feedback.

[0021] In a second aspect, the embodiments of the present application provide a mobile terminal, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is configured to store computer program codes, the computer program codes comprising computer instructions, and the one or more processors invoke the computer instructions to enable the mobile terminal to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0022] In a third aspect, the embodiments of the present application provide a computer program product comprising instructions, which, when executed on a mobile terminal, enable the mobile terminal to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0023] In a fourth aspect, the embodiments of the present application provide a computer-readable storage medium comprising instructions, which, when executed on a mobile terminal, enable the mobile terminal to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0024] It can be understood that the mobile terminal provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiments of the present application. Therefore, the beneficial effects that can be achieved are referred to the beneficial effects in the corresponding method, which will not be described here.

[0025] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0026] 1. Since the block building state monitoring mechanism based on camera real-time capture and special mark recognition is adopted, the system can accurately identify abnormal blocks with inconsistent position coordinates and orientation information, and display the adjustment scheme in the form of animation, effectively solving the problems of unable to timely find block installation deviation and non-intuitive adjustment method in the prior art, and thereby realizing real-time monitoring and accurate guidance of the block building process.

[0027] 2. Due to the adoption of the three-dimensional space mapping based on special markers and the computer mechanism for calculating the relative position relationship of adjacent blocks, the system can accurately obtain the spatial position information of each block and strictly control the assembly accuracy between adjacent blocks, realizing high-precision positioning and quality control during the block building process.

[0028] 3. Due to the adoption of the special structure identification and flashing warning mechanism based on geometric feature data, the system can accurately identify and locate the blocks containing special connection structures and provide prominent visual reminders, effectively solving the problem that the special structure blocks are easily damaged due to improper operation in the prior art, and further realizing intelligent protection and safety guidance for special structures. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a flowchart of an intelligent auxiliary building method based on entity mortise and tenon blocks in the embodiments of the present application.

[0030] Figure 2 is another flowchart of an intelligent auxiliary building method based on entity mortise and tenon blocks in the embodiments of the present application.

[0031] Figure 3 is a schematic diagram of an entity device structure of a mobile terminal in the embodiments of the present application. DETAILED DESCRIPTION

[0032] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting to the present application. As used in the specification, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or" used in the present application, means any or all possible combinations of one or more of the listed items.

[0033] Hereinafter, the terms "first" and "second" are only for the purpose of description, and cannot be understood as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0034] Please refer to Figure 1 is a flowchart of an intelligent auxiliary building method based on entity mortise and tenon blocks in the embodiments of the present application.

[0035] S101、In the process of building a target module using physical mortise and tenon blocks, a first block layout image of the built block pile is collected by the camera of the mobile terminal; each block surface is fixed with special marks different from other blocks according to a preset orientation relationship.

[0036] Wherein, the physical mortise and tenon block refers to a physical block with tenon and mortise structures, which can be stably combined through the complementary connection of concave-convex structures; the target module refers to a complete structure or a partial structure that the user expects to build; the mobile terminal refers to a portable electronic device, such as a smart phone, a tablet computer, etc., which is used to run an application program; the preset orientation relationship refers to the fixed position and direction of each special mark on the corresponding block surface, which is used to determine the correct orientation of the block in the three-dimensional space.

[0037] This step is executed when the user uses physical mortise and tenon blocks for building and needs the mobile terminal to provide auxiliary guidance. The user starts the auxiliary building application program through the mobile terminal and aims the camera of the mobile terminal at the built block pile. The camera of the mobile terminal will collect an image containing the entire block pile from a suitable angle, which clearly captures the special marks on the surface of each block. These special marks are fixed on the surface of each block according to the predetermined position and direction when the block is manufactured, and the marks of each block are different from those of other blocks.

[0038] It can be understood that the target module can be determined in various ways:

[0039] In some embodiments, the user can independently select the target module on the mobile terminal.

[0040] In other embodiments, the special marks on the surface of each mortise and tenon block in the image can be identified based on the first block layout image collected by the camera of the mobile terminal, and the matching degree of the special marks identified in the first block layout image with a preset block layout scheme is matched, and the stage in the preset building scheme with a matching degree exceeding a predetermined threshold is determined as the target module. The target module is a building stage arranged according to the standard building sequence in the preset building scheme, and each building stage has a specific block layout state, including the type of blocks with specific marks, their spatial position coordinates and orientation angles, which are not limited here.

[0041] It can be understood that image acquisition can be achieved in various ways: in some embodiments, the user opens the building assistance software on the mobile terminal, actively triggers the shooting function, places the building blocks in the lens field of view, ensures that the light is uniformly irradiated to clearly display the marks and contours of the building blocks, and the software automatically saves the captured image and marks the shooting time; in other embodiments, the user enables the automatic acquisition mode of the device, and when it is detected that the building block pile moves or the building action pauses, the camera automatically starts shooting. The shooting angle can also be expanded by an external auxiliary device, or the image can be preprocessed to remove the edge irrelevant area, which is not limited here.

[0042] S102, based on the special marks of each building block in the first building block layout image, comparing the position coordinates and orientation information of each building block in the building block pile with the corresponding building block in the preset three-dimensional digital model of the target module, and marking the building block in the building block pile as an abnormal building block set.

[0043] Among them, the preset three-dimensional digital model represents the accurate digital expression of the target module established in advance, including the special marks, geometric size parameters, ideal position, mortise and tenon structure position and connection relationship information of each building block, for example, the model will clearly indicate the coordinates, orientation and mortise and tenon connection mode of each building block in the three-dimensional space.

[0044] This step is executed after successfully collecting the first building block layout image. The mobile terminal identifies the type, relative position and orientation of each special mark in the first building block layout image based on the mark positioning technology.

[0045] Since the special marks are attached to the surface of the building blocks at a fixed position and direction, based on the deformation, size and direction of the special marks in the two-dimensional image, the accurate position coordinates and spatial orientation of each building block in the three-dimensional space are determined by using the spatial transformation relationship, wherein the spatial transformation relationship refers to the mathematical relationship of converting the mark information in the two-dimensional image into three-dimensional space coordinates and orientation, which is usually realized based on camera calibration parameters.

[0046] The preset three-dimensional digital model is called, and the position coordinates and orientation information of the actual building blocks in the building block pile are compared with the corresponding building blocks in the preset model one by one through the correspondence relationship. When it is detected that the actual position coordinates or orientation of a building block deviate from the ideal state in the preset model, and the deviation exceeds the preset threshold value, the building block is marked as an abnormal building block and added to the abnormal building block set. Among them, the preset threshold value is reasonably determined according to the tolerance of the building block size and the mortise and tenon structure (usually the position error does not exceed 10% of the minimum size of the building block, and the angle error does not exceed 15 degrees). After the comparison of all building blocks is completed, a set containing all abnormal building blocks is generated.

[0047] It can be understood that there are various implementation methods for determining abnormal building blocks:

[0048] In some embodiments, the collected image is first divided into a plurality of area grids, each grid corresponding to a specific spatial position in the preset three-dimensional digital model; based on the special markers of each block in the first block layout image, the grid position of each marker is recorded; based on the expected area where each block should appear in the preset three-dimensional digital model, the actual area of the special marker is compared with the expected area, and if a marker appears in a non-expected area, the corresponding block is directly marked as position abnormal; for blocks located in the correct functional area, the relative position relationship with adjacent blocks is checked; then the directional features of the special markers (such as asymmetric patterns, direction indicating lines) are used to determine the spatial orientation of each block, and the angle difference between the actual orientation and the preset orientation is calculated; if the angle deviation exceeds the threshold value, the block is added to the abnormal block set and marked as orientation error.

[0049] In other embodiments, the preset three-dimensional digital model is loaded into the memory as a standard reference model, which contains the exact position coordinates (X, Y, Z values) and orientation angles (pitch angle, yaw angle, roll angle) of each block in the ideal state; all blocks identified in the first block layout image are traversed, and for each block, its corresponding block instance in the preset model is found. For each pair of correspondence, the position deviation value (by calculating the spatial distance between two points) and the orientation deviation value (by calculating the angle difference between two orientation vectors) between the actual block and the ideal block are calculated; based on the set position deviation threshold (such as 8mm) and the orientation deviation threshold (such as 12 degrees), when the position deviation of a block exceeds the position threshold or the orientation deviation exceeds the angle threshold, the block is determined as an abnormal block; for the blocks determined to be abnormal, record the deviation type (position abnormal, orientation abnormal, or both), the deviation specific value, and the offset direction relative to the ideal state. The size change of the marker can also be combined to evaluate the near-far position deviation of the block, the marker deformation degree can be analyzed to evaluate the inclination degree of the block, and the motion trajectory model of the block can be established by using multiple images to make abnormality judgment, which is not limited here.

[0050] S103, in response to the user clicking the first abnormal block in the set of abnormal blocks highlighted on the first block layout image, determining the removal order of the associated blocks in the block pile that limit the adjustment of the first abnormal block based on the preset three-dimensional digital model.

[0051] Among them, the associated blocks refer to the blocks in the block pile that limit the adjustment of the first abnormal block, that is, without removing these blocks first, the orientation of the first abnormal block cannot be adjusted correctly.

[0052] This step is performed after the abnormal block set has been detected and highlighted on the interface, and the user has selected one of the specific abnormal blocks (first abnormal block) by clicking.

[0053] First, display the first block layout image on the screen of the mobile terminal, and highlight all abnormal blocks in the image with a highlight color or special marker.

[0054] After the user clicks on the first abnormal block, if the first abnormal block is consistent with the position coordinates of the corresponding block in the preset three-dimensional digital model but inconsistent in the orientation information, the mobile terminal identifies all associated blocks that restrict the direct adjustment of the first abnormal block based on the spatial position and connection of the first abnormal block in the preset three-dimensional digital model. These associated blocks are usually blocks stacked above the first abnormal block or blocks connected with mortise and tenon joints. According to the connection order of the mortise and tenon structure, the mobile terminal calculates the reasonable order for removing these associated blocks.

[0055] If the first abnormal block is inconsistent with the position coordinates of the corresponding block in the preset three-dimensional digital model, after the user clicks on the first abnormal block, if the first abnormal block is inconsistent with the position coordinates of the corresponding block in the preset three-dimensional digital model, the mobile terminal first determines whether the currently placed block is of the wrong type based on the type information of the block that should be placed at that position in the preset three-dimensional digital model. If it is a block of the wrong type, the correct position where the block should be placed is identified on the interface, and the correct block type that should be placed at the current position is displayed. For blocks connected with mortise and tenon joints, due to their structural characteristics, they can only be completely installed or cannot be connected during installation. Therefore, the system mainly focuses on the correctness of the block type and the consistency of the orientation information, and does not need to consider the position offset.

[0056] In some embodiments, based on the preset three-dimensional digital model, the mortise and tenon structure information of the first abnormal block is first extracted, including the tenon parameter table, the mortise parameter table and the connection pairing table (record the type and strength level of the connection); then, based on these mortise and tenon structure information, the mortise and tenon pairing table is queried, and the blocks directly connected with the first abnormal block in the actual block pile are identified and marked as preliminary associated blocks; the preliminary associated blocks are further analyzed, and based on their connection position and connection strength with the first abnormal block, the preset connection information is converted to the actual block pile coordinate system through spatial mapping, and the associated blocks that constitute physical obstacles for the orientation adjustment of the first abnormal block are determined through three-dimensional transformation simulation. The connection strength is evaluated through contact area calculation and engagement depth judgment, the larger the contact area, the higher the connection strength, and the greater the depth, the tighter the connection. The physical obstacles include the engagement restriction of the mortise and tenon structure, that is, the close connection of the tenon and the mortise may restrict the rotation of the block, and the mutual interference of the spatial position, that is, the existence of other blocks may hinder the rotation path of the first abnormal block. Based on the disassembly direction and the taking-out path of the related mortise and tenon connection type determined in the preset three-dimensional digital model, the taking-out sequence of the associated blocks is determined.

[0057] In other embodiments, the block pile is first divided into different structural levels, and the assembly tree structure of the blocks is extracted from the preset model, taking the first abnormal block as the root node, traversing all parent blocks (i.e. blocks located above or connected closely) upwards, and generating a taking-out list in the order from far to near (the blocks farthest away are disassembled first).

[0058] It can be understood that the taking-out sequence of the associated blocks can be determined in various ways, which are not limited here.

[0059] S104, displaying on the mobile terminal an animation of correcting the first abnormal block in the block pile, wherein at least includes a first animation image of taking out the associated blocks in the taking-out sequence and a second animation image of the corrected first abnormal block displaying a rotation direction arrow indication and an angle value.

[0060] This step is performed after determining the taking-out sequence of the associated blocks, and the mobile terminal generates dynamic visual guidance based on the calculated taking-out sequence and the target orientation. Specifically, the mobile terminal first displays the dynamic process of each associated block being separated and removed from the block pile in the predetermined taking-out sequence, and displays a highlight prompt at the corresponding position of the actual block pile. After all the associated blocks are taken out and displayed in the first animation image, the mobile terminal superimposes a rotation direction arrow and a specific angle value on the current position of the first abnormal block. During the entire animation display process, operation prompt text is displayed on the screen.

[0061] In some embodiments, the animation display can be implemented in various ways:

[0062] Optionally, based on the implementation of augmented reality (AR) technology, the virtual animation is spatially aligned with the actual pile of blocks. The mobile terminal uses the picture captured by the camera in real time as the background, and accurately superimposes virtual elements such as the taking-out animation and the rotation instruction on the corresponding physical block position; the animation adopts a step-by-step playing mode, and is paused after each step is completed, waiting for the user to confirm that the actual operation of the step has been completed, and then continuing the next step of the animation.

[0063] Optionally, the mobile terminal generates a pre-rendered 2D animation, which comprehensively shows the operation process through different perspectives (such as top view, side view, perspective view, etc.). The user can realize interactive operations such as zooming in, pausing, and replaying through the control buttons on the screen, and control the animation playing according to his own understanding rhythm. Here is not limited.

[0064] In some embodiments, after the first abnormal block correction operation is completed, the correction effect needs to be verified and the user is guided to complete the correction of the remaining abnormal blocks. After the user confirms through the completion identifier on the interface of the mobile terminal that the correction operation of the first abnormal block has been completed, the mobile terminal starts the camera module. The mobile terminal first displays the viewfinder auxiliary interface, and when it is detected that the block pile completely enters the viewfinder range and is clearly visible, the mobile terminal automatically or after user confirmation collects a second block layout image. Using the same technical path as the first block layout image analysis, the type, position and orientation of each special marker in the second block layout image are identified. After confirming that the first abnormal block has been corrected, the mobile terminal performs a complete block layout comparison process based on the second block layout image, identifies all blocks with inconsistent position coordinates or orientation information with the preset three-dimensional digital model, removes the corrected first abnormal block from the abnormal block set, and finally generates an updated abnormal block set.

[0065] In the above embodiment, the unique marker fixed on the surface of each block is used as the basis for identification, the spatial state of each block is accurately tracked, the deviation of the position or orientation of the block is detected, and the abnormal block is intuitively identified in the block layout image; by analyzing all the related blocks that affect the adjustment, the taking-out order of the related blocks and the specific adjustment method of the abnormal block are demonstrated based on the animation. This instant and phased visual feedback and dynamic guidance mechanism enables the user to clearly understand the correct state and adjustment method of each building step, significantly reducing the trial-and-error cost and learning threshold in the building process of the complex mortise and tenon structure.

[0066] In actual application, due to the user's insufficient understanding of the mortise and tenon structure, the selected abnormal block lacks intuitive priority in the process of independently selecting the abnormal block for modification, which may lead to the selected abnormal block being relatively marginal in the structure, and the user needs to select and modify multiple times in the entire modification process, affecting the efficiency of error correction.

[0067] Referring to Figure 2 , another flowchart of an embodiment of the intelligent auxiliary building method based on the entity mortise and tenon block.

[0068] S201, in the process of building the target module using the entity mortise and tenon block, a first block layout image of the built block pile is collected by the camera of the mobile terminal.

[0069] Step S201 is similar to step S101, which will not be repeated here.

[0070] S202, according to the block type corresponding to each special mark identified, a one-to-one mapping relationship between the actual mark and the preset block in the preset three-dimensional digital model is established.

[0071] The mobile terminal processes the collected first block layout image, and identifies the special marks in the image through computer vision technology. After identifying these marks, the preset mark-block type comparison database is queried to determine the specific block type, size and functional attribute corresponding to each mark. Then, the preset three-dimensional digital model library is accessed to find the preset block model related to the target module, and according to the identification result, a one-to-one mapping relationship between each special mark in the actual image and the corresponding block in the preset three-dimensional digital model is established. This mapping relationship records the unique identifier, type information and ideal position and orientation data of the block in the model.

[0072] In some embodiments, the special mark includes a preset color and shape combination, and the shape is a unique combination of 4 to 6 geometric figures, each shape uses a high-contrast color to improve visual recognition efficiency.

[0073] S203, based on the position and deformation characteristics of the special mark in the image, combined with the camera parameters of the mobile terminal, the position coordinates and orientation information of each block in the three-dimensional space are calculated.

[0074] The mobile terminal corrects the image according to the camera parameters to restore the real geometric features of the mark; through these two-dimensional image information and the position and angle parameters when the camera is shooting, using the principle of spatial geometric transformation, according to the fixed position relationship between the mark and the block body, the specific position coordinates of each block in the three-dimensional space and the rotation angle around different coordinate axes are determined, and then the orientation of the block is determined. For example, the nearness of the block can be determined by analyzing the size change of the mark, and the inclination angle of the block can be determined by analyzing the deformation degree of the mark.

[0075] S204, for the blocks set as adjacent in the preset three-dimensional digital model, the relative position relationship between the adjacent blocks in the special marks in the first block layout image is calculated.

[0076] According to the adjacent block pairs defined in the preset three-dimensional digital model, in the mapping relationship between the established actual blocks and the model blocks, the pixel coordinates of the special marks corresponding to each pair of adjacent blocks in the first block layout image are found, and these adjacent relationships are manifested as the existence of mortise and tenon connection, surface contact or relative position relationship within a certain distance between two blocks. By calculating the pixel distance of the centers of the two marks and the angle between the connecting line and the image coordinate system, the relative position difference of the two marks in the two-dimensional image is determined.

[0077] In some embodiments, by traversing all adjacent block pairs in the preset model, the mark coordinates of each block pair in the image are obtained according to the mapping relationship, the pixel distance is calculated using the Euclidean distance formula, and the angle corresponding to the coordinate difference is calculated by the inverse tangent function.

[0078] S205, if the relative position relationship between the special marks of a pair of adjacent blocks exceeds the preset tolerance range, the pair of adjacent blocks is added to the abnormal block set at the same time.

[0079] This step is performed after the relative position relationship of the adjacent blocks is calculated. The mobile terminal obtains the standard relative position relationship data of each pair of adjacent blocks in the ideal state from the preset three-dimensional digital model, including distance, angle and other parameters, and obtains the preset tolerance range value at the same time. The relative position relationship (such as distance, angle) of the special marks of each pair of adjacent blocks in the first block layout image is compared with the preset tolerance range. If the deviation between the actual relative position relationship of a pair of adjacent blocks and the standard data exceeds the preset tolerance range (such as the distance deviation is greater than 5mm or the angle deviation is greater than 10 degrees), the pair of blocks will be marked as abnormal at the same time and added to the abnormal block set.

[0080] In some embodiments, the identification and addition of abnormal blocks can be realized in various ways: optionally, different tolerance ranges can be set for different types of adjacent relationships, for example, a smaller tolerance is set for the mortise and tenon connection block pair, and a larger tolerance is set for the block pair with only surface contact relationship, and then the deviation of each block pair is evaluated whether it exceeds the tolerance range of the corresponding type; optionally, multiple dimensional deviations can be considered at the same time, including horizontal distance deviation, vertical distance deviation and angle deviation, only when the deviation of a certain dimension exceeds the corresponding tolerance value, the pair of blocks is added to the abnormal set, which is not limited here

[0081] S206, according to the block position deviation, the orientation angle error and the number of adjacent blocks, the key score of each abnormal block is calculated.

[0082] After determining the abnormal block set, each block in the abnormal block set is evaluated from multiple dimensions: the position deviation amount of the block (the distance between the actual position and the ideal position), the angle error of the orientation (the angle difference between the actual orientation and the ideal orientation), and the number of other blocks directly connected to the block; then, a weighted sum is used to set weight coefficients for the position deviation, the angle error, and the number of adjacent blocks, such as a position deviation weight of 0.4, an angle error weight of 0.3, and a number of adjacent blocks weight of 0.3, and then a weighted total score is calculated as a key degree score. Generally, blocks with greater position deviation, greater angle error, and more adjacent blocks will have a higher key degree score, indicating that these blocks have a more critical impact on the overall structure and need to be adjusted first.

[0083] In some embodiments, the calculation of the key degree score can be implemented in various ways: optionally, a position coefficient can be set according to the position of the block in the overall structure, with a higher position coefficient for bottom support blocks and a lower position coefficient for top decorative blocks; multiplying this coefficient by the basic key degree score to obtain the final key degree score; optionally, the particularity of the block type can be considered, and additional importance points can be set for certain key function blocks (such as load-bearing blocks or blocks connecting special structures) to increase their key degree score. It can be understood that other ways of calculating the key degree score can also be used, which are not limited here.

[0084] S207, sorting the abnormal blocks by key degree score from high to low to generate an abnormal block set containing a priority correction order.

[0085] All blocks in the abnormal block set are sorted in descending order of their key degree scores, and a new abnormal block set containing an explicit priority correction order is generated based on this sorting result.

[0086] In some embodiments, given the strict assembly sequence requirement of mortise and tenon blocks, the mobile terminal determines the first error among all incorrectly placed blocks according to the standard building sequence. If only one block is problematic, the adjustment guidance is provided directly for the block; if multiple blocks are problematic, the positions of all error blocks are displayed, but the first sequence block is highlighted.

[0087] S208, in response to the user clicking on the first abnormal block in the abnormal block set highlighted on the first block layout image, determining the removal sequence of the associated blocks in the block pile that restrict the adjustment of the first abnormal block based on the preset three-dimensional digital model.

[0088] Step S208 is similar to S103, which is not repeated here.

[0089] S209, based on the geometric feature data of the buckle-type mortise-tenon structure extracted from the preset three-dimensional digital model, identifying the associated blocks containing special structures in the block stack.

[0090] The mobile terminal first extracts detailed geometric feature data of all buckle-type mortise-tenon structures from the preset three-dimensional digital model, including the shape, size, connection direction, and disassembly method of the buckle. Then, by comparing the first block layout image, the preset special structure database is directly queried through the model and category of the block, and the blocks containing these special buckle-type mortise-tenon structures are identified and marked among the associated blocks to be removed. These special structures may require specific operation methods to be disassembled correctly, such as pressing a specific position, sliding or rotating in a specific direction, etc., which is different from ordinary stacking or simple mortise-tenon connection.

[0091] S210, determining whether the associated blocks containing special structures are identified in the block stack.

[0092] Checking whether there are blocks marked as containing buckle-type mortise-tenon structures in the list of identified associated blocks.

[0093] If there are special structure blocks, step S211 is executed;

[0094] If there are no special structure blocks, step S212 is executed.

[0095] S211, based on the accurate positioning of the coordinates and orientation of these special structures in the block stack, adding a flashing warning mark in the first animation image.

[0096] Based on the determined associated blocks containing special structures, a flashing warning mark is superimposed in the corresponding position in the first animation image according to the three-dimensional coordinates and orientation of the special structure blocks. Based on the extracted detailed geometric feature data of the buckle-type mortise-tenon structure, the warning mark is usually accompanied by a text prompt or a voice prompt, which details how to correctly operate these special structures to avoid damaging the blocks or causing difficulties in the disassembly process due to improper operation.

[0097] S212, displaying an animation on the mobile terminal for correcting the first abnormal block in the block stack, including at least a first animation image of removing the associated blocks in the removal order and a second animation image of the corrected first abnormal block with a rotating direction arrow indication and an angle value.

[0098] Step S212 is similar to step S104, which will not be repeated here.

[0099] In the embodiments of the present application, since the position deviation amount, the orientation angle error and the number of adjacent blocks and other multi-dimensional factors of the blocks are comprehensively considered, the influence level of the abnormal blocks is quantified from three aspects of spatial dislocation degree, direction deviation accuracy and structure correlation complexity, the subjective judgment is converted into an objective data-driven priority sequence, and the blindness problem in the traditional manual checking is effectively solved, and repeated disassembly and modification caused by preferential processing of low-impact errors is avoided.

[0100] The mobile terminal in the embodiments of the present application is described from the perspective of hardware processing. Please refer to Figure 3 FIG. 1 is a schematic diagram of an entity device structure of the mobile terminal in the embodiments of the present application.

[0101] It should be noted that Figure 3 The structure of the mobile terminal shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0102] As Figure 3 shown, the mobile terminal includes a CPU 301, which can perform various appropriate actions and processes according to programs stored in a ROM 302 or loaded from a storage portion 308 to a RAM 303, such as performing the methods described in the above embodiments. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302 and the RAM 303 are connected to each other through a bus 304. An I / O interface 305 is also connected to the bus 304.

[0103] The following components are connected to the I / O interface 305: an input portion 306 including an audio input device, a button switch and the like; an output portion 307 including a Liquid Crystal Display (LCD) and an audio output device, an indicator and the like; a storage portion 308 including a hard disk and the like; and a communication portion 309 including a network interface card such as a LAN (Local Area Network) card, a modem and the like. The communication portion 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory and the like is mounted on the drive 310 as needed, so that a computer program read therefrom is installed in the storage portion 308 as needed.

[0104] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program according to embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising computer programs for executing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 309, and / or installed from the removable medium 311. When the computer program is executed by the CPU 301, various functions defined in the present application are executed.

[0105] The flow charts and block diagrams in the drawings are schematic illustrations of possible architectures, functions and operations of systems, methods and computer program products in accordance with various embodiments of the present application. In this regard, each block in the flow charts or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures.

[0106] In particular, the mobile terminal of the embodiment includes a processor and a memory, and the memory stores a computer program which, when executed by the processor, implements the smart auxiliary building method based on the entity mortise and tenon block provided by the above embodiment.

[0107] As another aspect, the present application also provides a computer readable storage medium, which can be included in the mobile terminal described in the above embodiments, or can exist separately without being assembled into the mobile terminal. The storage medium carries one or more computer programs, which, when executed by a processor of the mobile terminal, cause the mobile terminal to implement the smart auxiliary building method based on the entity mortise and tenon block provided by the above embodiment.

[0108] The above-described embodiments are merely intended for describing the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0109] As used in the above embodiments, the term "when" can be interpreted to mean "if" or "after" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "on determining" or "if detecting (a stated condition or event)" can be interpreted to mean "if determined" or "in response to determining" or "on detecting (a stated condition or event)" or "in response to detecting (a stated condition or event)" depending on the context.

Claims

1. An intelligent auxiliary building method based on entity mortise and tenon block, characterized in that, The method is applied to a mobile terminal and comprises the following steps: In the process of building a target module using mortise and tenon blocks, a first block layout image of a built block pile is collected by a camera of the mobile terminal; each block surface is fixed with a special mark different from other blocks according to a preset orientation relationship; Based on the special marks of each block in the first block layout image, the blocks in the block pile that are inconsistent in position coordinates and / or orientation information with corresponding blocks in a preset three-dimensional digital model of the target module are compared, and the blocks in the block pile that are inconsistent in position coordinates and / or orientation information are taken as an abnormal block set; In response to a user clicking a first abnormal block in the abnormal block set highlighted on the first block layout image, the removal sequence of associated blocks in the block pile that restrict the adjustment of the first abnormal block is determined based on the preset three-dimensional digital model, specifically comprising: In response to the user clicking the first abnormal block in the abnormal block set highlighted on the first block layout image, mortise and tenon structure information of the first abnormal block is obtained from the preset three-dimensional digital model, including tenon position, mortise position and connection mode thereof; Based on the mortise and tenon structure information of the first abnormal block, blocks in the block pile directly connected with the first abnormal block are identified as preliminary associated blocks; Based on the connection position and connection strength of the preliminary associated blocks and the first abnormal block, associated blocks that constitute a physical obstruction to the orientation adjustment of the first abnormal block are determined, the physical obstruction including engagement restriction of mortise and tenon structure, mutual interference of spatial position, and the connection strength including contact area calculation and engagement depth; Based on the disassembly direction and removal path of the mortise and tenon connection determined through the preset three-dimensional digital model, the removal sequence of the associated blocks is determined; the position coordinates of the first abnormal block and the corresponding blocks are consistent but the orientation information is inconsistent; An animation of correcting the first abnormal block in the block pile is displayed on the mobile terminal, at least including a first animation image of removing the associated blocks in the removal sequence and a second animation image of the corrected first abnormal block displayed with a rotation direction arrow indication and an angle value.

2. The method of claim 1, wherein, The step of comparing the blocks in the block pile that are inconsistent in position coordinates and / or orientation information with corresponding blocks in the preset three-dimensional digital model of the target module based on the special marks of each block in the first block layout image specifically comprises: According to the block type corresponding to each special mark identified, a one-to-one mapping relationship between actual marks and preset blocks in the preset three-dimensional digital model is established; Based on the position and deformation characteristics of the special marks in the image, the position coordinates and orientation information of each block in the three-dimensional space are calculated in combination with the camera parameters of the mobile terminal; For blocks set as adjacent in the preset three-dimensional digital model, the relative position relationship between the adjacent blocks in the special marks in the first block layout image is calculated, the relative position relationship including distance and angle relationship between actual marks; If the relative positional relationship between the special marks of the pair of adjacent blocks exceeds a preset tolerance range, the pair of adjacent blocks are simultaneously added to the set of abnormal blocks, the preset tolerance range being a pre-set allowable positional deviation threshold, and the allowable positional deviation threshold being determined according to the block size and the splicing accuracy requirement.

3. The method of claim 2, wherein, The method further comprises: calculating a key score of each abnormal block according to the block positional deviation, the orientation angle error and the number of adjacent blocks; sorting the abnormal blocks according to the key scores from high to low to generate a set of abnormal blocks containing a priority correction order.

4. The method of claim 1, wherein, The method further comprises: After determining the removal order of the associated blocks, the method further comprises: After determining the removal order of the associated blocks based on the rotation angle, the method further comprises:

5. The method of claim 1, wherein, Before the step of displaying, on the mobile terminal, the animation of correcting the first abnormal block in the block pile, which at least includes the first animation image of removing the associated blocks in the removal order and the second animation image of the corrected first abnormal block with the rotation direction arrow indication and the angle value, the method further comprises: Based on the geometric feature data of the buckle-type mortise-and-tenon structure extracted from the pre-set three-dimensional digital model, the method further comprises: Based on the accurate positioning of the coordinate positions and orientations of the special structures in the block pile, the method further comprises:

6. The method of claim 1, wherein, After the step of displaying, on the mobile terminal, the animation of correcting the first abnormal block in the block pile, which at least includes the first animation image of removing the associated blocks in the removal order and the second animation image of the corrected first abnormal block with the rotation direction arrow indication and the angle value, the method further comprises: After the user confirms that the first abnormal block has been corrected, the method further comprises: Based on the special marks of the blocks in the second block layout image, the method further comprises: After confirming that the positional coordinates and orientation information of the first abnormal block are consistent with those of the corresponding block in the pre-set three-dimensional digital model of the target module, the method further comprises:

7. A mobile terminal, characterized by The mobile terminal comprises one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code comprising computer instructions, and the one or more processors invoke the computer instructions to enable the mobile terminal to perform the method according to any one of claims 1-6.

8. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are run on a mobile terminal, the mobile terminal is caused to perform the method according to any one of claims 1-6.

9. A computer program product, characterised in that, When the computer program product is run on a mobile terminal, the mobile terminal is caused to perform the method according to any one of claims 1-6.

Images