Image recognition-based mural automatic splicing control system and method

Abstract

This invention discloses an automatic mural splicing control system and method based on image recognition, belonging to the field of image recognition technology. The method includes: performing image recognition decomposition and structured encoding on the mural design drawing to obtain an assembly code map, a target wall reference rendering, and a 3D preview model; acquiring multi-view images of the wall work area and completing camera calibration, determining the transformation relationship between the camera coordinate system and the wall coordinate system to form a wall mapping matrix, and using the wall mapping matrix to map the assembly code map into a target placement coordinate table; acquiring pixel block images of the material area, extracting the appearance contour and appearance feature descriptors, and performing nearest neighbor matching with the assembly code map to obtain the corresponding entries in the target placement coordinate table. By combining the assembly code map with the target placement coordinate table, this invention can accurately generate the grasping and placement posture sequence of the robotic arm, achieving automated assembly and significantly improving the accuracy and efficiency of the assembly process.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an automatic mural splicing control system and method based on image recognition. Background Technology

[0002] With the development of intelligent manufacturing and automation technologies, image recognition technology has been gradually introduced into the field of mural assembly to improve production efficiency and assembly quality. Traditional mural assembly methods typically rely on manual operation, where operators manually assemble mural elements onto the wall according to design drawings. This is not only time-consuming and labor-intensive but also prone to assembly errors. To solve this problem, the introduction of image recognition technology has made the automation of mural assembly possible. By structurally encoding the design drawings, the system can automatically generate assembly code maps and use cameras and robotic arms to precisely control the mural assembly process, thereby greatly improving assembly accuracy and efficiency. In recent years, with advancements in camera calibration technology, image matching algorithms, and robotic arm control technology, image recognition-based automatic mural assembly control methods have been widely applied and have demonstrated excellent results in actual production.

[0003] However, existing technologies still have some shortcomings in the mural assembly process. First, traditional methods are poorly adaptable to the wall surface work area, especially during multi-view image acquisition and camera calibration, where differences in camera angle and shooting distance may affect the accuracy of mural splicing. Second, existing image recognition technologies may suffer from misalignment or missing parts when processing large-scale mural splicing due to image matching errors during the splicing process, especially in complex designs and large-sized wall work areas, where misalignment and inaccurate splicing are more pronounced. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an image recognition-based automatic mural splicing control method to solve the problems of inaccurate splicing, misalignment, or missing parts caused by factors such as changes in viewing angle and image matching errors during the mural splicing process.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides an automatic mural splicing control method based on image recognition, comprising, The mural design drawings are decomposed by image recognition and structured coding to obtain an assembly coding map, a target wall reference rendering map, and a 3D preview model; Collect multi-view images of the wall work area and complete camera calibration, determine the transformation relationship between the camera coordinate system and the wall coordinate system, form a wall mapping matrix, and use the wall mapping matrix to map the assembly coding map into a target placement coordinate table; Collect pixel block images of the material area, extract the appearance contour and appearance feature descriptors and perform nearest neighbor matching with the assembly coding map to obtain the corresponding entries of the target placement coordinate table, generate the robotic arm grasping and placement pose sequence and complete the assembly, and generate the actual placement pose. The system acquires a global image of the assembled mural, spatially aligns the global image with the target wall reference rendering using a wall mapping matrix, verifies each mural piece by piece based on the appearance feature descriptor of the assembly coding map, generates an anomaly list and a grid heat map, and combines the actual placement pose with the 3D preview model to form an interactive delivery package.

[0007] As a preferred embodiment of the image recognition-based automatic mural splicing control method of the present invention, the step of performing image recognition decomposition and structured encoding on the mural design drawing specifically includes: Obtain the mural design drawings and associate them with the physical dimensions of the wall, target resolution, and pixel block specifications to form a set of design parameters; The mural design drawing is divided into grids according to the pixel block specifications in the design parameter set, generating a pixel block array and recording the grid coordinates and adjacent topological relationships of each pixel block; Extract the pixel block appearance feature descriptors from the pixel block array and jointly encode them with grid coordinates and adjacent topological relationships to generate an assembly coding map; Perform wall coordinate rendering on the assembly coding map to generate a target wall reference rendering map, and perform stereoscopic preview modeling on the target wall reference rendering map to generate a three-dimensional preview model.

[0008] As a preferred embodiment of the image recognition-based automatic mural splicing control method of the present invention, wherein: the formation of the wall mapping matrix specifically includes: A camera array is deployed to acquire multi-angle images of the wall work area, resulting in a multi-view image set. Camera calibration is then performed on the multi-view image set to generate camera calibration parameters. Based on the camera calibration parameters, wall feature points are extracted from the multi-view image set and cross-view feature point associations are established to generate wall feature point trajectories. Perform a geometric consistency check on the trajectory of wall feature points and output the valid set of wall feature points; The transformation relationship between the camera coordinate system and the wall coordinate system is calculated by solving the effective wall feature point set, and the wall reference coordinates are obtained. The wall reference coordinates and camera calibration parameters are mapped and solved to form the wall mapping matrix.

[0009] As a preferred embodiment of the image recognition-based automatic mural splicing control method of the present invention, wherein mapping the assembly coding map into a target placement coordinate table specifically involves: Extract grid coordinates, target orientation, and adjacent topological relationships from the assembly coding map to form an assembly element set; The wall mapping matrix is ​​parsed and the transformation relationship between the camera coordinate system and the wall coordinate system is called to form the wall coordinate transformation parameters; Perform the first logical judgment on the assembly element set and filter out the grid coordinates that exceed the wall operation area, retaining the valid assembly element set located within the wall operation area; Perform a second logical judgment on the effective assembly element set and check the connectivity consistency of adjacent topological relationships in the wall coordinate system, and output a set of effective grids with connectivity consistency; By using the wall mapping matrix and wall coordinate transformation parameters, the grid coordinates and target orientation in the effective grid set are converted into the three-dimensional wall landing point and wall orientation. The three-dimensional wall landing point and wall orientation are then arranged in grid order to obtain the target placement coordinate table.

[0010] As a preferred embodiment of the image recognition-based automatic mural splicing control method of the present invention, wherein obtaining the corresponding entry of the target placement coordinate table specifically involves: A top-mounted camera is placed in the material area to capture images of the pixel blocks to be grabbed, generating a set of pixel block images. Perform appearance contour extraction on the pixel block image set to obtain contour descriptors, and at the same time extract the corresponding pixel block appearance feature descriptors from the pixel block image set to form a pixel block feature set; Extract the appearance feature descriptors from the assembly coding map to form a map feature set; Perform nearest neighbor matching on the pixel block feature set and the spectral feature set, and output the matching results; The matching results are correlated with the target coordinate table for retrieval to obtain the corresponding entries; Perform a placeability check on the corresponding entry and filter out completed entries, then output the entries to be placed.

[0011] As a preferred embodiment of the image recognition-based automatic mural splicing control method of the present invention, the step of generating the robotic arm grasping and placing pose sequence and completing the splicing specifically includes: Extract the 3D placement points and wall orientations from the items to be placed to form placement parameters; The placement parameters are used to solve for the pose, generate the robot arm placement pose, and plan the robot arm placement trajectory to obtain the placement pose sequence; The grasping points and grasping directions are solved from the contour descriptor to generate the grasping pose of the robotic arm and the grasping trajectory of the robotic arm is planned to obtain the grasping pose sequence. The grasping pose sequence and the placement pose sequence are combined in time to form a grasping and placement pose sequence, which is then sent to the robotic arm to perform block picking, transporting and pressing attachment actions. Collect pose feedback during the execution of the robotic arm to generate the actual placement pose.

[0012] As a preferred embodiment of the image recognition-based automatic mural splicing control method of the present invention, the step of spatially aligning the global image of the mural with the reference rendering image of the target wall surface specifically involves: The completion status of the wall work area is determined based on the actual placement posture, and global imaging acquisition is triggered to perform global imaging of the wall work area and generate a global image of the mural. The wall coordinates of the global image of the mural are remapped using a wall mapping matrix to obtain a wall-aligned image. Perform same-scale rasterization on the target wall reference rendering to obtain the reference raster image; Spatially register the wall-aligned image with the reference raster image and output the alignment result.

[0013] As a preferred embodiment of the image recognition-based automatic mural splicing control method of the present invention, the generation of the anomaly list and the grid heat map specifically includes: Based on the assembly coding map, extract the map appearance feature descriptors of each grid to form a map verification set; Based on the alignment results, each grid region is located in the wall alignment image and real-shot appearance feature descriptors are extracted to form a real-shot verification set; The image verification set and the real-shot verification set are compared according to the grid coordinates, and the verification conclusion is output. The verification results are categorized and summarized according to grid coordinates to obtain an anomaly list. The anomaly list is then labeled according to grid coordinates to generate a grid heat map.

[0014] As a preferred embodiment of the image recognition-based automatic mural splicing control method of the present invention, wherein: the formation of an interactive delivery package specifically includes: Based on the assembly coding map, the target wall reference rendering map and the 3D preview model, establish the same grid coordinate index; Extract the 3D placement point and orientation of the wall in the actual placement pose, update the pose of the 3D preview model according to the grid coordinate index, and generate the assembly restoration model. The anomaly list and the grid heatmap are linked to the assembly reconstruction model by grid coordinate index to generate an interactive annotation layer; The assembly restoration model, interactive annotation layer, and target wall baseline rendering are uniformly encapsulated to generate an interactive delivery package.

[0015] Secondly, the present invention provides an automatic mural splicing control system based on image recognition, including an encoding generation module, which performs image recognition decomposition and structured encoding on the mural design drawing to obtain an assembly encoding map, a target wall reference rendering map, and a three-dimensional preview model; The wall registration module acquires multi-view images of the wall work area and completes camera calibration, determines the transformation relationship between the camera coordinate system and the wall coordinate system, forms a wall mapping matrix, and uses the wall mapping matrix to map the assembly coding map into a target placement coordinate table. The sorting and matching module collects pixel block images of the material area, extracts the appearance contour and color features, performs nearest neighbor matching with the assembly coding map, obtains the corresponding entries in the target placement coordinate table, generates the robotic arm grasping and placement pose sequence, completes the assembly, and records the actual placement pose. The assembly execution module acquires a global image of the assembled mural, spatially aligns the global image of the mural with the target wall reference rendering image through a wall mapping matrix, verifies each block according to the color characteristics of the assembly coding map, generates an anomaly list and a grid heat map, and combines the actual placement pose with the 3D preview model to form an interactive delivery package.

[0016] The beneficial effects of this invention are as follows: By performing image recognition decomposition and structured coding on the mural design drawings, an assembly coding map, a target wall reference rendering map, and a 3D preview model can be automatically generated, ensuring the consistency between the design drawings and the actual assembly; through multi-view image acquisition and camera calibration, combined with the transformation between the wall coordinate system and the camera coordinate system, the position of each pixel block on the wall can be accurately calculated, effectively avoiding splicing errors caused by differences in viewing angles; by combining the assembly coding map with the target placement coordinate table, the grasping and placement posture sequence of the robotic arm can be accurately generated, realizing automated assembly; with the help of the wall mapping matrix, global image acquisition and spatial alignment of the assembled mural can be performed, and combined with the anomaly list and grid heat map, the assembly results can be verified block by block, misaligned or missing blocks can be detected in real time, and an interactive result report can be generated, greatly improving the accuracy and efficiency of the assembly process and ensuring that the final result meets the design requirements. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an image recognition-based automatic mural splicing control method.

[0019] Figure 2 Flowchart for creating the wall mapping matrix.

[0020] Figure 3 Flowchart for placing the corresponding entries in the coordinate table to obtain the target.

[0021] Figure 4 Flowchart for generating an exception list and a grid heatmap. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides an automatic mural splicing control method based on image recognition, including the following steps: S1. Perform image recognition decomposition and structured coding on the mural design drawings to obtain the assembly coding map, the target wall reference rendering map, and the three-dimensional preview model.

[0026] S1.1. Obtain the mural design drawing and associate it with the physical dimensions of the wall, the target resolution, and the pixel block specifications. The mural design drawing is imported through the image intelligent decomposition 2D and 3D mural design and production platform. The physical dimensions of the wall are represented by the wall width and wall height. The target resolution is represented by the number of pixels in the horizontal direction and the number of pixels in the vertical direction. The pixel block specifications are represented by the pixel block width and pixel block height. The physical dimensions of the wall, the target resolution, and the pixel block specifications together with the mural design drawing constitute the design parameter set. The design parameter set is used to limit the pixel mapping relationship of the mural design drawing within the range of the physical dimensions of the wall and to limit the grid boundaries of the mesh segmentation.

[0027] S1.2. The mural design drawing is meshed according to the pixel block specifications in the design parameter set to generate a pixel block array. The meshing is performed with the pixel block width and pixel block height of the pixel block specifications as a fixed grid step. The mural design drawing forms a two-dimensional pixel coordinate system at the target resolution. The two-dimensional pixel coordinate system is divided into grid columns at equal intervals along the horizontal direction according to the pixel block width, and into grid rows at equal intervals along the vertical direction according to the pixel block height. The intersection area of ​​the grid unit corresponds to a pixel block. The pixel block array is arranged in the order of grid rows and columns to form a two-dimensional array structure. Each pixel block in the pixel block array records the raster coordinates and adjacent topological relationships. The raster coordinates are represented by the grid row number and the grid column number. The adjacent topological relationships are determined by the upper, lower, left, and right grid units that differ from the grid row number and the grid column number by one unit and are written into the adjacent relationship field of the pixel block array. The pixel block array, together with the raster coordinates and adjacent topological relationships, serves as the object for subsequent pixel block appearance feature descriptor extraction and joint encoding.

[0028] S1.3. Extract the appearance feature descriptors of the pixel blocks in the pixel block array and jointly encode them with the grid coordinates and adjacent topological relationships to generate an assembly coding map. The appearance feature descriptors of the pixel blocks are calculated from the image region corresponding to each pixel block in the pixel block array.

[0029] The pixel block appearance feature descriptor calculation process includes separating the color channels of the image region and counting the number of pixels at each gray level within a preset gray level range to form a color histogram. Gradient calculation is performed on the image region to obtain the gradient magnitude and gradient direction of each pixel. The gradient magnitude is accumulated according to a preset angle range to form a gradient direction histogram. The color histogram and the gradient direction histogram are spliced ​​together in a fixed order to form the pixel block appearance feature descriptor.

[0030] To further explain, the preset grayscale range is 0 to 255, which is divided into 8 equal-width intervals for grayscale histogram statistical intervals. 0 to 255 corresponds to the grayscale value range of commonly used 8-bit images. The equal-width division is used to ensure the consistency of statistical dimensions of different pixel blocks and facilitate subsequent joint encoding and nearest neighbor matching. The 8 intervals reduce the statistical dimensions while maintaining the color distribution differentiation to reduce storage and retrieval overhead. Therefore, the full range of values ​​from 0 to 255 is selected and equal-width binning is used instead of narrower ranges or unequal-width binning.

[0031] The preset angle range is 0 to 180 degrees and divided into 9 equal-width intervals as the gradient direction histogram statistical interval. 0 to 180 degrees corresponds to the unsigned representation range of commonly used gradient directions. The equal-width division is used to ensure the consistency of texture direction statistics and facilitate direct comparison between different pixel blocks. The 9 intervals can cover common edge directions and control the feature dimension to avoid excessive subdivision leading to noise sensitivity. Therefore, the standard direction range of 0 to 180 degrees is selected and equal-width binning is adopted.

[0032] The fixed order of "color histogram first, gradient direction histogram second, with color histograms arranged sequentially by color channel and gradient direction histograms arranged in ascending order of angle range" is because color histograms represent the global color distribution of pixel blocks and are independent of the spatial location of image regions. Arranging color histograms first allows pixels of the same color system to be clustered preferentially during nearest neighbor matching. Gradient direction histograms represent the distribution of texture and edge structure and are more sensitive to local details. Arranging gradient direction histograms later can further distinguish texture differences when color distributions are similar. The sequential arrangement of color channels ensures that different pixel blocks can be directly compared in the same channel dimension, and the ascending angle range ensures that different pixel blocks can be directly compared in the same directional dimension. This ensures that the dimensional semantics of pixel block appearance feature descriptors in the assembly coding map are fixed and supports consistent distance metrics.

[0033] The joint encoding uses grid coordinates as index keys to combine the pixel block appearance feature descriptor and the adjacent topological relationship at the same grid coordinate and write them into the assembly encoding map, so that the assembly encoding map contains the pixel block appearance feature descriptor and the adjacent topological relationship at any grid coordinate.

[0034] S1.4. Perform wall coordinate rendering on the assembly coding map to generate a target wall reference rendering map. Wall coordinate rendering maps the grid coordinates of the assembly coding map to the wall coordinate system defined by the physical dimensions of the wall. The range of the wall coordinate system is determined by the physical dimensions of the wall in the design parameter set, the pixel grid density of the target wall reference rendering map is determined by the target resolution in the design parameter set, and the area occupied by the pixel block in the target wall reference rendering map is determined by the pixel block specification in the design parameter set. The wall coordinate rendering traverses the assembly coding map in grid coordinate order, fills the pixel block image area corresponding to each grid coordinate into the corresponding rendering area of ​​the target wall reference rendering map, and maintains the adjacent grid units corresponding to adjacent topological relationships in the target wall reference rendering map.

[0035] S1.5. Perform stereoscopic preview modeling on the target wall reference rendering to generate a 3D preview model. Stereoscopic preview modeling uses the target wall reference rendering as the texture mapping object and the wall plane defined by the physical dimensions of the wall as the 3D geometric carrier. The target wall reference rendering is mapped one-to-one with the pixel coordinates to the surface texture coordinates of the wall plane to form a textured 3D preview model of the wall. The correspondence between the physical dimensions of the wall and the target wall reference rendering is preserved, and the spatial position of the raster coordinates of the assembly coding map on the wall plane is kept consistent.

[0036] S2. Acquire multi-view images of the wall work area and complete camera calibration, determine the transformation relationship between the camera coordinate system and the wall coordinate system, form a wall mapping matrix, and use the wall mapping matrix to map the assembly coding map into a target placement coordinate table.

[0037] S2.1. Determine the working area on the wall and arrange the camera array. The working area on the wall is defined by the physical dimensions of the wall, which include the width and height of the wall. The working area on the wall is represented in the wall coordinate system by the origin and axes of the wall grid coordinates. The origin of the wall grid coordinates is set at a corner of the working area. The axes of the wall grid coordinates include the axis along the width of the wall and the axis along the height of the wall. The camera array is arranged in multiple positions directly in front of and to the side of the working area to cover the entire area. After the camera array is arranged, the camera array position record is output.

[0038] S2.2. Multi-angle image sets are obtained by acquiring images of the wall work area from multiple angles. The multi-angle acquisition is completed by the camera array triggering the shooting sequentially according to the camera array position record. The multi-angle image set contains the wall work area image corresponding to each camera. The multi-angle image set is associated with the camera array position record and stored to form the acquisition record.

[0039] S2.3. Perform camera calibration on the multi-view image set to generate camera calibration parameters. The camera calibration performs calibration pattern recognition on the multi-view image set and extracts the calibration pattern corner points. The calibration pattern corner points are used to solve the camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters include focal length parameters and principal point coordinate parameters, and the camera extrinsic parameters include camera attitude parameters and camera position parameters.

[0040] Based on camera calibration parameters, wall feature points are extracted from the multi-view image set, and cross-view feature point associations are established to generate wall feature point trajectories. Wall feature point extraction adopts corner detection and scale-invariant feature extraction to obtain a set of wall feature points from each image. Cross-view feature point association adopts feature descriptor matching to establish the correspondence between wall feature points in images from different viewpoints. The correspondence between wall feature points is connected according to the viewpoint order in the acquisition record to form the wall feature point trajectory.

[0041] S2.4. Perform a geometric consistency check on the wall feature point trajectories and output a valid set of wall feature points. The geometric consistency check includes performing geometric constraint verification on the wall feature point trajectories and removing those that do not meet the geometric constraints. The geometric constraint verification is completed using a joint verification method of two-view geometric consistency and reprojection consistency. Two-view geometric consistency is used to verify the consistency of the imaging geometric relationship between cross-view feature points under different viewpoints. Reprojection consistency is used to verify the consistency between the predicted projection position of the wall feature points under the constraints of camera calibration parameters and the actual position of the wall feature points. The wall feature point trajectories that pass the geometric consistency check are retained to form a valid set of wall feature points. When the number of valid wall feature points is insufficient to solve the transformation relationship between the camera coordinate system and the wall coordinate system, return to S2.2 to re-execute multi-angle acquisition and camera calibration and update the multi-view image set and camera calibration parameters until the valid set of wall feature points meets the solution requirements.

[0042] The wall reference coordinates are obtained by solving the transformation relationship between the camera coordinate system and the wall coordinate system using the effective wall feature point set. The transformation relationship between the camera coordinate system and the wall coordinate system is solved by the pose calculation method. The pose calculation method combines the cross-view correspondence of the effective wall feature point set with the camera calibration parameters to obtain the pose representation of the wall plane in the camera coordinate system. The pose representation of the wall plane is converted into the pose relationship of the wall coordinate system in the camera coordinate system and the transformation relationship between the camera coordinate system and the wall coordinate system is output. At the same time, the wall reference coordinates are determined according to the origin of the wall grid coordinates and the axis of the wall grid coordinates.

[0043] S2.5. Perform coordinate mapping calculations between the wall reference coordinates and camera calibration parameters to form a wall mapping matrix. The coordinate mapping calculation establishes a correspondence between the planar coordinate points in the wall coordinate system and the pixel coordinate points in the multi-view image set. The pixel coordinate points are provided by the effective wall feature point set, and the planar coordinate points are provided by the wall reference coordinates. The coordinate mapping calculation solves the planar mapping relationship for the corresponding point set and outputs the wall mapping matrix.

[0044] The grid coordinates, target orientation, and adjacent topological relationships in the assembly coding map are extracted to form an assembly feature set. The assembly feature set organizes the field content of the assembly coding map using the grid coordinates as an index.

[0045] The wall mapping matrix is ​​parsed and the transformation relationship between the camera coordinate system and the wall coordinate system is called to form the wall coordinate transformation parameters. The wall coordinate transformation parameters include the pose transformation expression between the wall coordinate system and the camera coordinate system, as well as the mapping expression of the wall mapping matrix.

[0046] S2.6. Perform a first logical judgment on the assembly element set and filter out grid coordinates that exceed the wall operation area. The first logical judgment performs an accessibility judgment on the grid coordinates in the assembly element set according to the wall coordinate range corresponding to the physical dimensions of the wall. The wall coordinate range limited by the physical dimensions of the wall is determined by the wall width and the wall height. The grid coordinate accessibility judgment includes mapping the grid coordinates to the wall plane coordinates and detecting whether the wall plane coordinates fall within the wall coordinate range. Grid coordinates that fall within the wall coordinate range are retained, and a valid assembly element set is output.

[0047] The second logical judgment is performed on the effective assembly element set and the connectivity consistency of adjacent topological relationships in the wall coordinate system is detected. The set of effective grids with connectivity consistency is output. The second logical judgment reads the adjacent topological relationships for each grid coordinate in the effective assembly element set and generates a grid adjacency relationship graph. When the grid coordinate pointed to by the adjacent topological relationship of any grid coordinate in the grid adjacency relationship graph is not included in the effective assembly element set, it is determined to be inconsistent in connectivity, and the corresponding adjacent edge is removed from the grid adjacency relationship graph. The connectivity consistency detection performs a connected component traversal on the grid adjacency relationship graph and retains the connected component containing the most grid coordinates to form a set of effective grids with connectivity consistency.

[0048] S2.7. Convert the grid coordinates and target orientation in the effective grid set into the three-dimensional landing point and wall orientation of the wall using the wall mapping matrix and wall coordinate transformation parameters, and arrange them according to the grid order of the effective grid set to obtain the target placement coordinate table.

[0049] To further explain, the grid order adopts the row and column scanning order based on the grid row number of the grid coordinates from small to large, and within the same grid row number, based on the grid column number from small to large. The grid order is obtained by sorting the grid coordinates in the effective grid set according to the grid row number and the grid column number, and is used for the arrangement of entries in the target placement coordinate table.

[0050] The conversion from raster coordinates to 3D wall landing point involves mapping the raster coordinates to wall plane coordinates and combining them with wall coordinate transformation parameters to obtain a 3D coordinate representation in the wall coordinate system.

[0051] To further explain, the mapping from grid coordinates to wall plane coordinates is calculated using the wall coordinate conversion formula for the grid center point, specifically: ; in, Grid row number indicating raster coordinates; The grid column number representing the raster coordinates; The width of the pixel block, representing the pixel block specification; The height of the pixel block, representing the pixel block size; This represents the distance between the center points of adjacent horizontal pixels in the wall coordinate system. This represents the distance between the center points of adjacent vertical pixels in the wall coordinate system. The width direction coordinate of the wall surface represents the planar coordinates of the wall surface. The wall's height coordinates represent the wall's planar coordinates; wall planar coordinates. It is used to locate the grid coordinates in the effective grid set to the spatial position corresponding to the physical size of the wall and to calculate the three-dimensional landing point of the wall.

[0052] The conversion from target orientation to wall orientation includes expressing the target orientation of the assembly element set as the wall orientation in the wall coordinate system. The three-dimensional placement points of the wall and the wall orientation are arranged in the grid order of the effective grid set to form a target placement coordinate table.

[0053] S3. Collect pixel block images of the material area, extract the appearance contour and appearance feature descriptors, and perform nearest neighbor matching with the assembly coding map to obtain the corresponding entries in the target placement coordinate table, generate the robotic arm grasping and placement pose sequence, complete the assembly, and generate the actual placement pose.

[0054] S3.1. A top-mounted camera is set up in the material area and the illumination status of the three-color indicator light is set. The pixel blocks to be captured are placed in the imaging range of the material area in a disordered manner. The top-mounted camera continuously images the material area to obtain a set of pixel block images. The set of pixel block images is associated with the imaging time of the top-mounted camera to form an acquisition sequence.

[0055] Based on the acquisition sequence, the appearance contour extraction of the pixel block image set is performed to obtain contour descriptors. The appearance contour extraction includes grayscale processing of the pixel block image set and smoothing filtering to suppress noise. Gradient calculation is performed on the smoothed grayscale image to obtain an edge response map. The edge response map is binarized to obtain an edge binary map. The edge pixel set is obtained by searching the edge binary map according to the connected component. Contour tracking is performed along the edge pixel set according to the eight-neighbor connectivity rule to obtain the closed contour of the pixel block. The closed contour of the pixel block is composed of the sequence of edge pixels connected end to end. The contour descriptor is composed of the contour point sequence of the closed contour of the pixel block, the minimum bounding rectangle, and the direction angle of the minimum bounding rectangle. The appearance feature descriptor of the pixel block image set is extracted. The appearance feature descriptor of the pixel block is calculated by concatenating the color histogram and the gradient direction histogram. The contour descriptor and the appearance feature descriptor of the pixel block are combined according to the imaging order of the pixel block image set to form the pixel block feature set.

[0056] S3.2. Extract appearance feature descriptors from the assembly coding map to form a map feature set. The map feature set organizes the appearance feature descriptors according to the grid coordinate index of the assembly coding map. The map feature set and the target placement coordinate table maintain a one-to-one correspondence according to the grid coordinate.

[0057] Nearest neighbor matching is performed on the pixel block feature set and the map feature set, and the matching results are output. Nearest neighbor matching calculates the feature distance between each pixel block appearance feature descriptor and each map appearance feature descriptor in the map feature set, and selects the map appearance feature descriptor with the smallest feature distance as the matching object. The matching results include the correspondence between the pixel block image set index and the assembly coding map grid coordinate index.

[0058] To further explain, the expression for calculating the feature distance is: ; in, Indicates the first The appearance feature descriptor of the pixel block and the first pixel block The feature distance of each graph appearance feature descriptor is used to measure the similarity between the pixel block appearance feature descriptor and the graph appearance feature descriptor and is used for nearest neighbor matching determination. Represents the first pixel in the feature set of pixel blocks A pixel block appearance feature descriptor vector; Represents the first in the spectral feature set Each graph appearance feature descriptor vector; This represents the L2 norm operation.

[0059] S3.3. The matching results are associated with the target placement coordinate table to obtain the corresponding entries. The associated search locates the entry position in the target placement coordinate table based on the assembly coding map grid coordinate index provided by the matching results and reads the three-dimensional landing point of the wall and the wall orientation to form the corresponding entries. After the corresponding entries are output, they are bound with the pixel block image set index to form the set of entries to be checked.

[0060] The placement availability test is performed on the set of items to be inspected and the items to be placed are output. The placement availability test determines the placement availability based on the status of the items in the target placement coordinate table. The status of the items is determined by the items in the target placement coordinate table corresponding to the actual placement pose. The status of the items includes completed items and incomplete items. The corresponding items that are completed items in the set of items to be inspected are removed from the set of items to be inspected and the incomplete items are retained to form the items to be placed.

[0061] It should be noted that by associating the matching results with the target placement coordinate table, obtaining the corresponding entries, and performing placement checks, it is possible to ensure that the position of each pixel block accurately matches its target placement position and effectively avoid re-execution of completed entries. This process not only improves the efficiency of the assembly process but also ensures that no repetitive work is generated during the assembly process, significantly reducing the execution time and workload of the robotic arm, thereby improving the overall accuracy and efficiency of mural assembly.

[0062] S3.4. Extract the 3D landing point and orientation of the wall from the items to be placed to form placement parameters and complete pose solving and trajectory planning. The pose solving determines the robot arm placement pose based on the 3D landing point and orientation of the wall and generates a placement pose sequence. The robot arm grasping pose is determined by the contour descriptor. The center point of the minimum bounding rectangle of the contour descriptor is used as the grasping position, and the orientation angle of the minimum bounding rectangle of the contour descriptor is used as the grasping direction. The robot arm grasping pose is determined based on the grasping position and grasping direction and generates a grasping pose sequence. The trajectory planning generates transport trajectories for the grasping pose sequence and the placement pose sequence respectively, and combines them in time sequence to form grasping and placement pose sequences. When the number of robot arms is greater than 1, the target placement coordinate table is divided into non-overlapping sub-tables according to the wall area, and grasping and placement pose sequences are generated according to the sub-tables respectively.

[0063] The grasping and placement pose sequence is sent to the robotic arm to complete the actions of picking up, transporting and pressing the block. During the execution of the actions of picking up, transporting and pressing the block, the robotic arm collects pose feedback and generates the actual placement pose. The actual placement pose is then correlated with the position of the item to be placed in the target placement coordinate table.

[0064] S4. Collect global images of the assembled mural, spatially align the global mural images with the target wall reference rendering image through the wall mapping matrix, and verify each block according to the appearance feature descriptor of the assembly code map, generate an anomaly list and grid heat map, and combine the actual placement pose and 3D preview model to form an interactive delivery package.

[0065] S4.1. Determine the completion status of the wall work area based on the actual placement posture and trigger global imaging acquisition. The completion status determination uses the entry position of the target placement coordinate table as a statistical index. The target placement coordinate table is searched one by one and the number of entries that correspond to the actual placement posture is counted. When the number of entries covers all the entry positions of the target placement coordinate table, the completion status of the wall work area is determined and the camera array is triggered to capture images. The global image of the mural is formed by recording the wall work area image of the camera corresponding to the position directly in front of the camera array and associating it with the acquisition record.

[0066] A wall-aligned image is obtained by remapping the global image of the mural using a wall mapping matrix. The wall coordinate remapping process maps the pixel coordinates of the global image of the mural to the planar coordinates of the wall coordinate system through the wall mapping matrix and generates a wall coordinate system sampling grid. The wall coordinate system sampling grid is then backsampled in the global image of the mural and bilinear interpolation is used to obtain the pixel values ​​in the wall coordinate system. The pixel values ​​in the wall coordinate system are arranged according to the wall grid coordinate origin and the wall grid coordinate axis, forming a wall-aligned image.

[0067] S4.2. Perform same-scale rasterization on the target wall reference rendering to obtain a reference raster image. The same-scale rasterization process uses the target resolution in the design parameter set to determine the pixel size of the reference raster image and uses the pixel block specification to determine the grid unit boundary. The reference raster image is superimposed with grid lines on the target wall reference rendering image and the grid row number and grid column number of the grid coordinates are consistent with the assembly coding map. The reference raster image and the target wall reference rendering image maintain a one-to-one correspondence of pixel coordinates.

[0068] The wall alignment image is spatially registered with the reference raster image to output the alignment result. Spatial registration includes extracting the pixel coordinates of raster intersection points in the reference raster image and extracting the pixel coordinates of candidate intersection points obtained from corner detection in the wall alignment image. The nearest neighbor correspondence is performed on the pixel coordinates of raster intersection points and candidate intersection points, and inconsistent corresponding point pairs are eliminated by using geometric consistency test. The planar mapping relationship is solved for the corresponding point pairs that pass the test, and the planar mapping relationship is applied to the wall alignment image to obtain the aligned wall alignment image. The planar mapping relationship of the reference raster image and the aligned wall alignment image together constitute the alignment result.

[0069] It should be noted that by performing the same-scale rasterization process on the target wall reference rendering to obtain the reference raster image, and then spatially registering the wall-aligned image with the reference raster image, the spatial alignment of images from different perspectives is effectively achieved. Through this process, the assembled image can be accurately aligned with the target design drawing, thereby improving the assembly accuracy and ensuring that the placement of each image in the mural meets the design requirements, avoiding visual errors and misalignment problems.

[0070] S4.3. Based on the alignment results, locate each grid region in the wall alignment image and extract the actual appearance feature descriptors to form an actual verification set. The grid region positioning uses the grid cell boundary of the reference grid image to map the alignment results to the aligned wall alignment image, obtaining the pixel range of the grid region. The actual appearance feature descriptors are selected within the pixel range of the grid region according to the preset gray level range of 0 to 255 and the pixel count is calculated according to 8 equal width intervals to form a color histogram. The gradient direction histogram is obtained by accumulating the gradient magnitude according to the preset angle range of 0 to 180 degrees and 9 equal width intervals. The color histogram and the gradient direction histogram are spliced ​​together in the fixed order of S1.3 to form the actual appearance feature descriptors. The actual appearance feature descriptors are organized according to the grid coordinate index to form the actual verification set.

[0071] Based on the assembly coding map, the appearance feature descriptors of each grid are extracted to form a map verification set. This set is then compared with the actual shooting verification set block by block to output the verification conclusion. The map verification set extracts the appearance feature descriptors corresponding to each grid coordinate in the assembly coding map according to the grid coordinate index, and maintains consistency with the reference grid. Figure 1The consistent raster coordinate index is verified block by block. This includes calculating the feature distance between the real-shot appearance feature descriptor and the map appearance feature descriptor for each raster coordinate index and performing a nearest neighbor consistency determination. The nearest neighbor consistency determination retrieves the nearest neighbor raster coordinate index for the real-shot appearance feature descriptor in the map verification set, and retrieves the nearest neighbor raster coordinate index for the map appearance feature descriptor in the real-shot verification set.

[0072] The nearest neighbor consistency determination includes outputting a consistent verification conclusion when the two nearest neighbor raster coordinate indices are consistent, and outputting a misaligned verification conclusion when the two nearest neighbor raster coordinate indices are inconsistent. When the actual shooting verification set does not generate an actual shooting appearance feature descriptor with the corresponding raster coordinate index, it outputs a missing verification conclusion. The consistent verification conclusion, misaligned verification conclusion, and missing verification conclusion are organized according to the raster coordinate index to form a verification conclusion set.

[0073] S4.4. Classify and summarize the verification conclusion set according to the raster coordinates to obtain an anomaly list and generate a raster heat map. The anomaly list summarizes the raster coordinate indices corresponding to misaligned verification conclusions and missing verification conclusions to form an anomaly entry set and retains the corresponding verification conclusion type field. The raster heat map maps the anomaly entry set to the raster cell position of the baseline raster map and generates a raster heat map according to the verification conclusion type field.

[0074] An interactive delivery package is formed by combining the actual placement pose with the 3D preview model. The grid coordinate index is kept consistent by the assembly coding map, the target wall reference rendering map, and the 3D preview model. The 3D landing point and wall orientation of the wall are extracted by the actual placement pose and associated with the grid coordinate index according to the position of the target placement coordinate table entry. The 3D preview model is updated with pose according to the grid coordinate index to generate the assembly restoration model. The anomaly list and grid heat map are associated with the assembly restoration model with the grid coordinate index to generate an interactive annotation layer. The assembly restoration model, interactive annotation layer, and target wall reference rendering map are uniformly encapsulated to generate an interactive delivery package.

[0075] This embodiment also provides an image recognition-based automatic mural splicing control system, including: The encoding generation module performs image recognition decomposition and structured encoding on the mural design drawings to obtain an assembly encoding map, a target wall reference rendering map, and a 3D preview model; The wall registration module acquires multi-view images of the wall work area and completes camera calibration, determines the transformation relationship between the camera coordinate system and the wall coordinate system, forms a wall mapping matrix, and uses the wall mapping matrix to map the assembly coding map into a target placement coordinate table. The sorting and matching module collects pixel block images of the material area, extracts the appearance contour and color features, performs nearest neighbor matching with the assembly coding map, obtains the corresponding entries in the target placement coordinate table, generates the robotic arm grasping and placement pose sequence, completes the assembly, and records the actual placement pose. The assembly execution module acquires a global image of the assembled mural, spatially aligns the global image of the mural with the target wall reference rendering image through a wall mapping matrix, verifies each block according to the color characteristics of the assembly coding map, generates an anomaly list and a grid heat map, and combines the actual placement pose with the 3D preview model to form an interactive delivery package.

[0076] In summary, this invention, through image recognition decomposition and structured encoding of mural design drawings, can automatically generate assembly coding maps, target wall reference rendering maps, and 3D preview models, ensuring consistency between the design drawings and the actual assembly. Through multi-view image acquisition and camera calibration, combined with the transformation between the wall coordinate system and the camera coordinate system, the position of each pixel block on the wall can be accurately calculated, effectively avoiding splicing errors caused by differences in viewing angles. By combining the assembly coding map with the target placement coordinate table, the grasping and placement posture sequence of the robotic arm can be accurately generated, achieving automated assembly. With the help of the wall mapping matrix, global image acquisition and spatial alignment of the assembled mural can be performed. Combined with an anomaly list and grid heatmap, the assembly results can be verified block by block, identifying misaligned or missing blocks in real time and generating an interactive result report, significantly improving the accuracy and efficiency of the assembly process and ensuring that the final result meets design requirements.

[0077] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An automatic mural splicing control method based on image recognition, characterized in that: include, The mural design drawings are decomposed by image recognition and structured coding to obtain an assembly coding map, a target wall reference rendering map, and a 3D preview model; Collect multi-view images of the wall work area and complete camera calibration, determine the transformation relationship between the camera coordinate system and the wall coordinate system, form a wall mapping matrix, and use the wall mapping matrix to map the assembly coding map into a target placement coordinate table; Collect pixel block images of the material area, extract the appearance contour and appearance feature descriptors and perform nearest neighbor matching with the assembly coding map to obtain the corresponding entries of the target placement coordinate table, generate the robotic arm grasping and placement pose sequence and complete the assembly, and generate the actual placement pose. The system acquires a global image of the assembled mural, spatially aligns the global image with the target wall reference rendering using a wall mapping matrix, verifies each mural piece by piece based on the appearance feature descriptor of the assembly coding map, generates an anomaly list and a grid heat map, and combines the actual placement pose with the 3D preview model to form an interactive delivery package.

2. The automatic mural splicing control method based on image recognition as described in claim 1, characterized in that: The image recognition decomposition and structured encoding of the mural design drawing specifically involves: Obtain the mural design drawings and associate them with the physical dimensions of the wall, target resolution, and pixel block specifications to form a set of design parameters; The mural design drawing is divided into grids according to the pixel block specifications in the design parameter set, generating a pixel block array and recording the grid coordinates and adjacent topological relationships of each pixel block; Extract the pixel block appearance feature descriptors from the pixel block array and jointly encode them with grid coordinates and adjacent topological relationships to generate an assembly coding map; Perform wall coordinate rendering on the assembly coding map to generate a target wall reference rendering map, and perform stereoscopic preview modeling on the target wall reference rendering map to generate a three-dimensional preview model.

3. The automatic mural splicing control method based on image recognition as described in claim 1, characterized in that: The formation of the wall mapping matrix is ​​specifically as follows: A camera array is deployed to acquire multi-angle images of the wall work area, resulting in a multi-view image set. Camera calibration is then performed on the multi-view image set to generate camera calibration parameters. Based on the camera calibration parameters, wall feature points are extracted from the multi-view image set and cross-view feature point associations are established to generate wall feature point trajectories. Perform a geometric consistency check on the trajectory of wall feature points and output the valid set of wall feature points; The transformation relationship between the camera coordinate system and the wall coordinate system is calculated by solving the effective wall feature point set, and the wall reference coordinates are obtained. The wall reference coordinates and camera calibration parameters are mapped and solved to form the wall mapping matrix.

4. The automatic mural splicing control method based on image recognition as described in claim 1, characterized in that: The process of mapping the assembly coding map to a target placement coordinate table specifically involves: Extract grid coordinates, target orientation, and adjacent topological relationships from the assembly coding map to form an assembly element set; The wall mapping matrix is ​​parsed and the transformation relationship between the camera coordinate system and the wall coordinate system is called to form the wall coordinate transformation parameters; Perform the first logical judgment on the assembly element set and filter out the grid coordinates that exceed the wall operation area, retaining the valid assembly element set located within the wall operation area; Perform a second logical judgment on the effective assembly element set and check the connectivity consistency of adjacent topological relationships in the wall coordinate system, and output a set of effective grids with connectivity consistency; By using the wall mapping matrix and wall coordinate transformation parameters, the grid coordinates and target orientation in the effective grid set are converted into the three-dimensional wall landing point and wall orientation. The three-dimensional wall landing point and wall orientation are then arranged in grid order to obtain the target placement coordinate table.

5. The automatic mural splicing control method based on image recognition as described in claim 1, characterized in that: The specific steps for obtaining the corresponding entry in the target placement coordinate table are as follows: A top-mounted camera is placed in the material area to capture images of the pixel blocks to be grabbed, generating a set of pixel block images. Perform appearance contour extraction on the pixel block image set to obtain contour descriptors, and at the same time extract the corresponding pixel block appearance feature descriptors from the pixel block image set to form a pixel block feature set; Extract the appearance feature descriptors from the assembly coding map to form a map feature set; Perform nearest neighbor matching on the pixel block feature set and the spectral feature set, and output the matching results; The matching results are correlated with the target coordinate table for retrieval to obtain the corresponding entries; Perform a placeability check on the corresponding entry and filter out completed entries, then output the entries to be placed.

6. The automatic mural splicing control method based on image recognition as described in claim 1, characterized in that: The process of generating the robotic arm's grasping and placement pose sequence and completing the assembly is as follows: Extract the 3D placement points and wall orientations from the items to be placed to form placement parameters; The placement parameters are used to solve for the pose, generate the robot arm placement pose, and plan the robot arm placement trajectory to obtain the placement pose sequence; The grasping points and grasping directions are solved from the contour descriptor to generate the grasping pose of the robotic arm and the grasping trajectory of the robotic arm is planned to obtain the grasping pose sequence. The grasping pose sequence and the placement pose sequence are combined in time to form a grasping and placement pose sequence, which is then sent to the robotic arm to perform block picking, transporting and pressing attachment actions. Collect pose feedback during the execution of the robotic arm to generate the actual placement pose.

7. The automatic mural splicing control method based on image recognition as described in claim 1, characterized in that: The spatial alignment of the global image of the mural with the reference rendering image of the target wall surface specifically involves: The completion status of the wall work area is determined based on the actual placement posture, and global imaging acquisition is triggered to perform global imaging of the wall work area and generate a global image of the mural. The wall coordinates of the global image of the mural are remapped using a wall mapping matrix to obtain a wall-aligned image. Perform same-scale rasterization on the target wall reference rendering to obtain the reference raster image; Spatially register the wall-aligned image with the reference raster image and output the alignment result.

8. The automatic mural splicing control method based on image recognition as described in claim 1, characterized in that: The generation of the anomaly list and grid heatmap specifically involves: Based on the assembly coding map, extract the map appearance feature descriptors of each grid to form a map verification set; Based on the alignment results, each grid region is located in the wall alignment image and real-shot appearance feature descriptors are extracted to form a real-shot verification set; The image verification set and the real-shot verification set are compared according to the grid coordinates, and the verification conclusion is output. The verification results are categorized and summarized according to grid coordinates to obtain an anomaly list. The anomaly list is then labeled according to grid coordinates to generate a grid heat map.

9. The automatic mural splicing control method based on image recognition as described in claim 1, characterized in that: The formation of the interactive delivery package specifically includes: Based on the assembly coding map, the target wall reference rendering map and the 3D preview model, establish the same grid coordinate index; Extract the 3D placement point and orientation of the wall in the actual placement pose, update the pose of the 3D preview model according to the grid coordinate index, and generate the assembly restoration model. The anomaly list and the grid heatmap are linked to the assembly reconstruction model by grid coordinate index to generate an interactive annotation layer; The assembly restoration model, interactive annotation layer, and target wall baseline rendering are uniformly encapsulated to generate an interactive delivery package.

10. An image recognition-based automatic mural splicing control system, based on the image recognition-based automatic mural splicing control method according to any one of claims 1 to 9, characterized in that: include, The encoding generation module performs image recognition decomposition and structured encoding on the mural design drawings to obtain an assembly encoding map, a target wall reference rendering map, and a 3D preview model; The wall registration module acquires multi-view images of the wall work area and completes camera calibration, determines the transformation relationship between the camera coordinate system and the wall coordinate system, forms a wall mapping matrix, and uses the wall mapping matrix to map the assembly coding map into a target placement coordinate table. The sorting and matching module collects pixel block images of the material area, extracts the appearance contour and color features, performs nearest neighbor matching with the assembly coding map, obtains the corresponding entries in the target placement coordinate table, generates the robotic arm grasping and placement pose sequence, completes the assembly, and records the actual placement pose. The assembly execution module acquires a global image of the assembled mural, spatially aligns the global image of the mural with the target wall reference rendering image through a wall mapping matrix, verifies each block according to the color characteristics of the assembly coding map, generates an anomaly list and a grid heat map, and combines the actual placement pose with the 3D preview model to form an interactive delivery package.

Images