A book automatic detection and positioning method based on multi-modal fusion

Abstract

The present application relates to the technical field of book detection, and more particularly to a book automatic detection and positioning method based on multi-modal fusion, which comprises: color image, depth image, inertia data acquisition and alignment, modeling bookshelf topology; book spine semantic segmentation, extracting three-dimensional point cloud candidate set; propagation node joint optimization estimating spatial position and orientation; introducing plane, normal, principal axis constraint and attitude disturbance information; extracting visual features, matching feature library to bind identity; outputting book spatial position and attitude information. Through the spatial alignment of color image and depth image, the present application models the geometric features such as bookshelf vertical plate plane, layer plate height and layer spacing into the spatial topology structure of bookshelf, effectively solving the problem of low accuracy of book instance boundary division and positioning result precision caused by light change, book spine occlusion, close arrangement of books and complex background.

Description

Technical Field

[0001] This invention relates to the field of book inspection technology, and in particular to an automatic book inspection and localization method based on multimodal fusion. Background Technology

[0002] With the rapid increase in the number of books in libraries, bookstores, and various book storage environments, traditional manual search methods are inefficient and prone to errors, making automated book location increasingly urgent. However, due to factors such as changes in lighting, spine occlusion, dense book arrangement, and complex backgrounds, existing computer vision-based book location methods are prone to decreased accuracy and location errors in book detection, instance segmentation, and title recognition. This makes it difficult to achieve stable and efficient rapid and accurate location of target books in real-world environments, posing a significant challenge to highly reliable intelligent book management.

[0003] Chinese Patent Application Publication No. CN114863458A discloses a computer vision-based book location method. The method includes: capturing and storing images of target book regions using a camera; constructing and training a text detection model to label all text in the image with text region boxes; segmenting the image into book instances based on the text region boxes to obtain book instances and their location information within the image; constructing and training a text recognition model to recognize the text region boxes and merging the text recognition results within the same book instance to obtain the book title recognition result for each book instance; and matching the book title recognition result with the name of a book the reader inputs to search for on the bookshelf, outputting the location result of the target book in the image.

[0004] Therefore, the computer vision-based book localization method has the following problems: the method relies on text detection and text recognition models to determine the book title, but the accuracy of text detection and recognition can easily drop significantly under conditions such as changes in lighting, spine occlusion, complex or damaged fonts; the method segments book instances by text region boxes, but if the books on the bookshelf are close to each other, partially occluded, or the background is complex, the boundaries of the book instances can easily be inaccurately defined. Summary of the Invention

[0005] To address this, the present invention provides an automatic book detection and localization method based on multimodal fusion, which overcomes the problems of low accuracy in book instance boundary delineation and localization results caused by changes in lighting, spine occlusion, close arrangement of books, and complex backgrounds in the prior art by fusing image features, text features, and spatial layout information.

[0006] To achieve the above objectives, this invention provides a method for automatic book detection and localization based on multimodal fusion, comprising:

[0007] Acquire color images, depth images, and inertial measurement data of the bookshelf area, spatially align the color images and depth images, and model the bookshelf upright plane, shelf height sequence, and spacing between adjacent shelves based on the depth images to construct the bookshelf spatial topology.

[0008] Based on the color image, at least one candidate spine region is extracted using a preset spine semantic segmentation model, and a corresponding candidate set of three-dimensional point clouds of the spine is extracted according to the correspondence between the candidate spine regions in the depth image.

[0009] In the bookshelf spatial topology, each candidate set of three-dimensional point clouds of the book spine is used as a propagation node. Under the prior constraints of the device attitude provided by the inertial measurement data, the spatial position and orientation of each propagation node are jointly optimized and estimated based on the parallel relationship, adjacent spacing relationship and depth continuity relationship between the propagation nodes.

[0010] In the process of joint optimization estimation, constraints such as coplanarity of the spine plane, consistency of the spine normal, and collinearity of the spine principal axes are introduced. Combined with the instantaneous attitude perturbation information represented by the inertial measurement data, the pose coordination of each propagation node is driven in the joint optimization estimation to obtain a converged set of candidate spine poses.

[0011] Extract the visual feature vector of the spine from the color image region corresponding to each candidate spine pose, and match it with the template features in the preset book feature library to bind the spine visual identity with the corresponding three-dimensional pose to obtain the spine pose.

[0012] Output the spatial position and posture information of the target book based on the spine pose.

[0013] Furthermore, the process of acquiring color images, depth images, and inertial measurement data of the bookshelf area, and spatially aligning the color and depth images includes:

[0014] The acquisition of the color image, the depth image, and the inertial measurement data is triggered synchronously by a unified timestamp, and the coordinate transformation of the depth image is performed using the extrinsic parameter matrix between the depth camera and the color camera to achieve spatial alignment with the color image.

[0015] Furthermore, the process of modeling the bookshelf upright plane, shelf height sequence, and spacing between adjacent shelves based on depth images to construct the bookshelf spatial topology includes:

[0016] The planar equation of the bookshelf uprights is determined by plane fitting, the height sequence of the shelves is determined by layered clustering of the projection distribution of the depth point cloud in the vertical direction, and the spacing between adjacent shelves is calculated based on the height difference between adjacent shelves to construct the spatial topology of the bookshelf.

[0017] Furthermore, the process of extracting at least one candidate spine region using a pre-defined spine semantic segmentation model includes:

[0018] The color image is input into the preset spine semantic segmentation model, which outputs a spine pixel-level segmentation mask and generates at least one spine candidate region based on the segmentation mask.

[0019] Furthermore, the process of extracting the corresponding candidate set of 3D point clouds of the spine based on the correspondence between the candidate spine regions in the depth image includes:

[0020] Based on the pixel coordinates of the candidate spine region in the color image, the corresponding depth value is obtained in the depth image, and the candidate set of the three-dimensional point cloud of the spine is generated by back projection calculation.

[0021] Furthermore, under the prior constraints of the device attitude provided by the inertial measurement data, the process of jointly optimizing and estimating the spatial position and orientation of each propagation node based on the parallel relationship, adjacent spacing relationship, and depth continuity relationship between propagation nodes includes:

[0022] Each of the propagation nodes is constructed as a vertex of the pose graph model;

[0023] Based on the bookshelf space topology and the spatial proximity of the propagation nodes, the connections between vertices are determined to construct the edges of the pose graph;

[0024] Based on the parallelism, spacing, and depth continuity between adjacent vertices connected by the edges, multiple types of geometric constraint residuals are defined.

[0025] By integrating the residuals of the various geometric constraints with the prior constraints of the device attitude, a nonlinear optimization objective function is constructed.

[0026] By iteratively solving the objective function, the position and orientation parameters of all vertices are updated synchronously to perform joint optimization estimation.

[0027] Furthermore, after obtaining the candidate spine pose set, the method also includes:

[0028] Based on the residuals of each propagation node during the iterative solution process, calculate its corresponding pose confidence.

[0029] Based on the pose confidence level, the poses in the candidate spine pose set are filtered.

[0030] Furthermore, the process of calculating the pose confidence of each propagation node based on the residuals during the iterative solution process includes:

[0031] Based on the final constraint residuals of each propagation node in the nonlinear optimization objective function, a residual statistical distribution is constructed.

[0032] Based on the residual statistical distribution and the preset confidence interval, calculate the pose confidence score of each propagation node;

[0033] The pose confidence score is normalized to obtain the corresponding pose confidence score.

[0034] Furthermore, by introducing constraints such as coplanarity of the spine plane, uniformity of the spine normal, and collinearity of the spine principal axes, and combining the instantaneous attitude perturbation information represented by the inertial measurement data, the process of driving the pose coordination convergence of each propagation node in the joint optimization estimation includes:

[0035] The coplanar constraint of the spine plane is constructed based on the point-to-plane distance between the spine planes corresponding to adjacent propagation nodes;

[0036] The consistent spine normal constraint is constructed based on the angle between the spine plane normal vectors corresponding to adjacent propagation nodes;

[0037] The collinear constraint of the spine main axis is constructed based on the included angle between the spine main axis directions corresponding to adjacent propagation nodes;

[0038] Under the instantaneous attitude disturbance constraints provided by the inertial measurement data, the coplanar constraint of the spine plane, the consistent constraint of the spine normal, and the collinear constraint of the spine main axis are all incorporated into the nonlinear optimization objective function for solution.

[0039] Furthermore, the process of outputting the spatial position and orientation information of the target book based on the spine pose includes:

[0040] Transform the spine pose into a unified coordinate representation in the world coordinate system;

[0041] Based on the three-dimensional coordinates of the converted spine pose in the world coordinate system, and combined with the height sequence of the shelves in the bookshelf space topology, the shelf to which the target book belongs is determined.

[0042] Based on the horizontal coordinates of the spine positions of each book within the same shelf, determine the horizontal arrangement order of the target books on the shelf;

[0043] The three-dimensional coordinates in the world coordinate system, the corresponding shelf, the horizontal arrangement order, and the attitude components in the corresponding spine pose are formatted and encapsulated to output the spatial position and attitude information of the target book.

[0044] Compared with existing technologies, the beneficial effects of this invention are as follows: by spatially aligning color images and depth images, geometric features such as the bookshelf upright plane, shelf height, and interlayer spacing are modeled as the spatial topology of the bookshelf, enabling the two-dimensional pixel information of the candidate spine region to be accurately mapped to the three-dimensional point cloud; under the device attitude constraints provided by inertial measurement data, joint optimization is performed by combining the parallel relationship between spines, interlayer spacing, and depth continuity, so that the spatial position and orientation of the spines can converge collaboratively; the introduction of plane coplanarity, normal consistency, and principal axis collinearity constraints further strengthens the geometric relationship between nodes, so that the pose estimation results are consistent with the actual arrangement and relative spacing; finally, by binding identity through visual features and template matching, the precise positioning and attitude determination of each book in three-dimensional space are achieved, thereby improving the accuracy, robustness, and automated management efficiency of detection. Candidate poses converge rapidly to the same stable pose pattern, significantly reducing isolated false detections and jump errors. Then, through high-dimensional feature matching between the visual features of the spine and the templates in the book feature library, the stable 3D pose is bound to the unique book identity, ultimately achieving synchronous determination of the target book in terms of shelf height, horizontal order, and 3D coordinates. Thus, even under complex occlusion, multi-view jitter, and lighting changes, stable and consistent output between spine detection, identity recognition, and spatial positioning results can still be achieved. This effectively solves the problems of low accuracy in book instance boundary division and positioning results caused by lighting changes, spine occlusion, close book arrangement, and complex backgrounds.

[0045] Furthermore, by synchronously triggering the acquisition of color images, depth images, and inertial measurement data using a unified timestamp, the problems of target position drift and attitude misalignment caused by inconsistent acquisition times are avoided. A stable extrinsic parameter matrix between the color and depth cameras is used to transform the coordinates of the depth images, ensuring that depth information and color texture information correspond precisely in the same pixel coordinate system. This guarantees that each spine pixel has a unique spatial distance constraint. Simultaneously, the instantaneous attitude changes represented by the inertial measurement data can establish a mapping relationship with the aligned image data under the same time reference, providing a consistent spatiotemporal reference for subsequent spine 3D modeling, pose propagation, and error correction processes, significantly reducing systematic errors introduced by motion jitter and sensor asynchrony.

[0046] Furthermore, by performing planar fitting on the depth point cloud to extract the planar equations of the bookshelf uprights, a stable and consistent spatial reference benchmark is obtained for the entire bookshelf structure. Then, by performing layered clustering on the vertical projection distribution of the depth point cloud, the discrete depth data in continuous space can be automatically decomposed into multiple physical shelf height sequences, effectively suppressing the interference of bookshelf tilt, acquisition jitter, and local occlusion on shelf identification. Furthermore, based on the height difference between adjacent shelves, the spacing between adjacent shelves is calculated, transforming the vertical constraint relationship between each shelf layer from empirical judgment into quantifiable structural parameters, thereby providing stable and reliable spatial topology support for subsequent 3D positioning of book spines, shelf assignment determination, and multi-node propagation updates.

[0047] Furthermore, a pre-defined spine semantic segmentation model is used to extract the spine region at the pixel level from the color image. This transforms the acquisition of candidate spine regions from the traditional coarse localization method based on edges or color thresholds to a precise localization method based on deep semantic features. This method can stably separate the real spine region in complex lighting, multi-textured backgrounds, and bookshelf environments with closely arranged books. It reduces the probability of non-spine regions being mistakenly selected as candidate targets from the source, providing more accurate and clear initial constraints for subsequent 3D point cloud extraction and pose propagation updates of the spine, thereby improving the reliability and consistency of the overall detection and localization results.

[0048] Furthermore, by accurately locating the pixel coordinates of the candidate spine region in the color image and simultaneously acquiring the corresponding depth information in the depth image, and then reconstructing the 3D point cloud candidate set of the spine through back projection, a reliable mapping of the spine from a 2D image to a 3D space is achieved. This allows the spatial position, scale, and hierarchical relationship of the spine to be fully expressed, thus providing a stable data foundation for subsequent spine pose propagation, spatial constraint modeling, and precise positioning. This avoids the problems of occlusion misjudgment, distance distortion, and spatial ambiguity caused by relying solely on 2D images, and significantly improves the integrity and accuracy of the spine spatial positioning.

[0049] Furthermore, by constructing a pose graph optimization model that integrates multiple geometric constraints and device attitude priors, high-precision and consistent joint estimation of the spatial poses of multiple book spines in a bookshelf is achieved. This method not only significantly improves the overall accuracy and geometric consistency of pose estimation, but also enhances the algorithm's robustness to environmental disturbances such as depth noise and occlusion by introducing physical constraints. Its structured graph optimization framework also makes solving complex problems more efficient and reliable.

[0050] Furthermore, by calculating and filtering pose confidence based on residuals, the system can effectively quantify the stability and accuracy of each propagation node during the iterative optimization process, achieving a comprehensive evaluation of the spatial position, orientation, spacing between adjacent layers, parallelism, and depth continuity of the spine. This not only eliminates abnormal poses that may be caused by acquisition noise or pose disturbances, but also ensures that the retained candidate poses are consistent in parameters such as shelf height, spine normal, main axis direction, and spatial arrangement. This results in a stable and coherent structure in the final spine pose set in three-dimensional space, further improving the accuracy and reliability of book detection and positioning, while naturally reflecting the inherent relationship between the bookshelf geometry and spine arrangement.

[0051] Furthermore, by utilizing the residual statistics inherent in the optimization process, an adaptive quality assessment system was constructed, transforming the abstract optimization results into quantifiable confidence indices. This enables the system to automatically identify and eliminate low-quality pose estimates caused by noise, occlusion, or poor fitting. By employing a dynamic confidence interval based on the overall statistical distribution, the system can adapt to different error levels in different scenarios, enhancing its robustness to environmental changes. Ultimately, this mechanism significantly improves the reliability and stability of the entire positioning system's output.

[0052] Furthermore, by strengthening the inherent strong geometric priors and physical motion consistency of book arrangement in a bookshelf scenario, the originally isolated and noise-sensitive single-point estimation is transformed into a mutually correcting and globally consistent collaborative perception. This solution not only significantly improves the estimation accuracy of the 3D pose of the book spine and the pose consistency among multiple spines by utilizing fine geometric constraints, but also effectively suppresses errors introduced by device jitter and motion blur during the acquisition process by integrating the physical motion constraints of the IMU. This significantly enhances the overall robustness of the system and the reliability of the output results in dynamic and complex lighting environments, thus providing directly reliable and accurate spatial information input for subsequent automated operations such as robot grasping and inventory.

[0053] Furthermore, by combining a pre-built bookshelf space topology, each precise six-DOF spine pose is assigned semantic labels for "belonging shelf" and "lateral arrangement order," enabling robots or management systems not only to know the precise location and orientation of a book, but also to understand its logical position within the overall bookshelf layout. The final formatted and encapsulated output eliminates data ambiguity and provides a plug-and-play standardized interface, greatly improving the integration efficiency with subsequent modules such as robot grasping planning, inventory database updates, or visual monitoring, as well as the reliability of the entire automated workflow. Attached Figure Description

[0054] Figure 1 This is a flowchart of the automatic book detection and localization method based on multimodal fusion in this embodiment;

[0055] Figure 2 This is a flowchart illustrating the joint optimization estimation process in this embodiment;

[0056] Figure 3 This is a flowchart for calculating the pose confidence of each propagation node in this embodiment;

[0057] Figure 4 This is a flowchart for outputting the spatial location and orientation information of the target book in this embodiment. Detailed Implementation

[0058] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0059] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0060] Please see Figure 1 The diagram shown is a flowchart of the automatic book detection and localization method based on multimodal fusion in this embodiment. This embodiment provides an automatic book detection and localization method based on multimodal fusion, including:

[0061] Acquire color images, depth images, and inertial measurement data of the bookshelf area. Spatially align the color and depth images, and model the bookshelf upright planes, shelf height sequences, and adjacent shelf spacing based on the depth images to construct the bookshelf spatial topology. Based on the color images, extract at least one spine candidate region using a preset spine semantic segmentation model, and extract corresponding spine 3D point cloud candidate sets according to the correspondence between the spine candidate regions in the depth images. In the bookshelf spatial topology, use each spine 3D point cloud candidate set as a propagation node. Under the prior constraints of device attitude provided by the inertial measurement data, based on the parallel relationship and adjacent spacing relationship between the propagation nodes... Based on the relationship of depth continuity, the spatial position and orientation of each propagation node are jointly optimized and estimated. In the process of joint optimization estimation, constraints such as coplanarity of the spine plane, consistency of spine normal, and collinearity of spine principal axes are introduced. Combined with the instantaneous attitude perturbation information represented by the inertial measurement data, the pose of each propagation node is driven to converge in the joint optimization estimation to obtain a converged set of candidate spine poses. The visual feature vector of the spine is extracted from the color image region corresponding to each candidate spine pose and matched with the template features in the preset book feature library to bind the visual identity of the spine with the corresponding three-dimensional pose to obtain the spine pose. The spatial position and attitude information of the target book are output according to the spine pose.

[0062] In this embodiment, color images, depth images, and inertial measurement data are synchronously acquired by a mobile acquisition terminal deployed in a library or archive bookshelf inspection scenario. The mobile acquisition terminal includes an integrated color camera, depth camera, and inertial measurement unit (IMU). The color camera is used to acquire RGB images of the bookshelf area with a resolution of 1280×720, providing information on the texture of the book spine. The depth camera is used to synchronously acquire depth images that are time-aligned with the color images, with a depth range of 0.3–5 m, outputting distance information of the book spine and bookshelf structure. The inertial measurement unit is used to output three-axis angular velocity, three-axis acceleration, and heading angle in real time, characterizing the instantaneous attitude changes of the acquisition terminal as it moves along the bookshelf, thus providing prior constraints on the device attitude for subsequent 3D modeling and pose propagation of the book spine.

[0063] In this embodiment, the preset spine semantic segmentation model adopts a pixel-level semantic segmentation model based on a deep convolutional neural network. Its network structure includes a feature encoding network and a feature decoding network. The feature encoding network consists of multiple convolutional layers and downsampling layers, which are used to extract multi-scale semantic features from the input color image. The feature decoding network consists of upsampling layers and skip connection structures, which are used to restore the multi-scale semantic features to the same spatial resolution as the original image and output the spine pixel-level probability segmentation result.

[0064] In its implementation, the semantic segmentation model takes a 512×512 RGB image as input and outputs a single-channel or multi-channel semantic probability map of the spine, matching the input size. The model uses a 3×3 kernel with a stride of 1, and the number of feature channels increases progressively to 64, 128, 256, and 512. The downsampling ratio is 2x, and upsampling is achieved using bilinear interpolation or deconvolution. The encoder backbone uses a ResNet-50 pre-trained on ImageNet, and a channel attention mechanism is introduced in the high-level feature fusion of the decoder to enhance feature extraction from the narrow spine region.

[0065] The semantic segmentation model is trained offline under supervised supervision using a pre-built spine annotation dataset. It employs a weighted sum of cross-entropy loss and Dice loss as the optimization objective function, and uses either Adam or stochastic gradient descent as the optimization algorithm. The initial learning rate is set to 1×10⁻⁶. -3 ~1×10 -4 The training iterations are no less than 50 epochs to ensure the model converges and achieves stable spine segmentation capabilities. During training, data augmentation strategies such as random brightness and contrast adjustments and slight perspective transformations are employed to improve the model's robustness.

[0066] In this embodiment, the preset book feature library is used to store visual feature templates of standard book spines and their corresponding book identity information. The book identity information includes at least one or more of the following: book title, ISBN number, and classification number.

[0067] In the specific construction process, color images of the spines of multiple standard books are pre-collected under different lighting conditions, shooting angles, and distances. These spine images undergo size normalization, brightness equalization, and noise suppression preprocessing before being input into a spine feature extraction network to obtain corresponding spine visual feature vectors. These spine visual feature vectors are then bound to the corresponding book identity information and stored to form the book feature library. The spine visual feature vectors are 128-dimensional or 256-dimensional floating-point vectors. The spine feature extraction network employs an embedded feature extraction model based on a convolutional neural network, with its output layer using a fully connected layer or a global average pooling layer for feature dimensionality reduction. The feature extraction network is trained using the ArcFace loss function to increase inter-class differences and reduce intra-class differences. When updating the feature library, the process of spine image acquisition, feature vector extraction, and identity information binding is repeated for newly added books, and the book feature library is incrementally maintained using version numbers or timestamps.

[0068] In this embodiment, after extracting the visual feature vector of the spine from the color image region corresponding to the candidate spine pose to be identified, the book identity corresponding to the template feature with the highest similarity is determined by calculating the cosine similarity or Euclidean distance between the visual feature vector of the spine and the template feature vector in the preset book feature library, thus completing the binding of the spine visual identity and the three-dimensional pose.

[0069] By spatially aligning color and depth images, geometric features such as the shelf uprights, shelf heights, and interlayer spacing are modeled as the spatial topology of the bookshelf, enabling precise mapping of 2D pixel information of candidate spine regions to 3D point clouds. Under the device attitude constraints provided by inertial measurement data, joint optimization is performed by combining the parallel relationships between spines, interlayer spacing, and depth continuity, allowing the spatial position and orientation of spines to converge collaboratively. The introduction of plane coplanarity, normal consistency, and principal axis collinearity constraints further strengthens the geometric association between nodes, ensuring that the pose estimation results are consistent with the actual arrangement and relative spacing. Finally, by binding identity through visual features and template matching, the precise positioning and attitude determination of each book in 3D space are achieved, thereby improving the accuracy, robustness, and efficiency of automated management. Candidate poses converge rapidly to the same stable pose pattern, significantly reducing isolated false detections and jump errors. Then, through high-dimensional feature matching between the visual features of the spine and the templates in the book feature library, the stable 3D pose is bound to the unique book identity, ultimately achieving synchronous determination of the target book in terms of shelf height, horizontal order, and 3D coordinates. Thus, even under complex occlusion, multi-view jitter, and lighting changes, stable and consistent output between spine detection, identity recognition, and spatial positioning results can still be achieved. This effectively solves the problems of low accuracy in book instance boundary division and positioning results caused by lighting changes, spine occlusion, close book arrangement, and complex backgrounds.

[0070] Specifically, the process of acquiring color images, depth images, and inertial measurement data of the bookshelf area and spatially aligning the color images and depth images includes: synchronously triggering the acquisition of the color images, depth images, and inertial measurement data through a unified timestamp, and using the extrinsic parameter matrix between the depth camera and the color camera to perform coordinate transformation on the depth image to complete the spatial alignment with the color image.

[0071] By synchronously triggering the acquisition of color images, depth images, and inertial measurement data using a unified timestamp, the problems of target position drift and attitude misalignment caused by inconsistent acquisition times are avoided. Furthermore, the stable extrinsic parameter matrix between the color camera and the depth camera is used to perform coordinate transformation on the depth image, so that the depth information and color texture information correspond precisely in the same pixel coordinate system, thereby ensuring that each spine pixel has a unique spatial distance constraint. At the same time, the instantaneous attitude change represented by the inertial measurement data can establish a mapping relationship with the aligned image data under the same time reference, so that the subsequent spine 3D modeling, pose propagation, and error correction processes have a consistent spatiotemporal reference basis, significantly reducing the systematic errors introduced by motion jitter and sensor asynchrony.

[0072] Specifically, the process of modeling the bookshelf upright plane, shelf height sequence, and spacing between adjacent shelves based on depth images to construct the bookshelf spatial topology includes: determining the bookshelf upright plane equation through plane fitting, determining the shelf height sequence by performing layered clustering on the projection distribution of depth point cloud in the vertical direction, and calculating the spacing between adjacent shelves based on the height difference between adjacent shelves to construct the bookshelf spatial topology.

[0073] In the specific implementation, the 3D point cloud generated from the spatially aligned depth image is used as the modeling data source. First, a set of point clouds with normals close to the vertical direction is selected within the area where the bookshelf is located. A robust plane fitting algorithm is used, with 1000 iterations and a distance threshold of 0.01 meters, to fit the spatial plane equation of the bookshelf uprights. Then, the point cloud is projected vertically, and the density distribution of the point cloud within each height interval is statistically analyzed. A density clustering algorithm is used to divide the height distribution into layers, with a neighborhood search radius of 0.02 meters and a minimum number of points of 100, thereby obtaining the height center value corresponding to each shelf, forming the shelf height sequence. Further, the difference between the height center values ​​of adjacent shelves is calculated to obtain the layer spacing parameters between adjacent layers, ultimately forming the bookshelf spatial topology structure used to constrain the spatial positioning of the book spine.

[0074] By performing planar fitting on the depth point cloud to extract the planar equations of the bookshelf uprights, a stable and consistent spatial reference benchmark is obtained for the entire bookshelf structure. Then, by performing layered clustering on the vertical projection distribution of the depth point cloud, discrete depth data in continuous space can be automatically decomposed into multiple physical shelf height sequences, effectively suppressing the interference of bookshelf tilt, acquisition jitter, and local occlusion on shelf identification. Furthermore, based on the height difference between adjacent shelves, the spacing between adjacent shelves is calculated, transforming the vertical constraint relationship between each shelf layer from empirical judgment into quantifiable structural parameters. This provides stable and reliable spatial topology support for subsequent 3D positioning of book spines, shelf assignment determination, and multi-node propagation updates.

[0075] Specifically, the process of extracting at least one candidate spine region through a preset spine semantic segmentation model includes: inputting the color image into the preset spine semantic segmentation model, outputting a spine pixel-level segmentation mask, and generating at least one candidate spine region based on the segmentation mask.

[0076] A pre-defined spine semantic segmentation model is used to extract the spine region at the pixel level from color images. This transforms the acquisition of spine candidate regions from the traditional coarse localization method based on edges or color thresholds to a precise localization method based on deep semantic features. It can stably separate the real spine region in complex lighting, multi-textured backgrounds, and bookshelf environments with closely arranged books. This reduces the probability of non-spine regions being mistakenly selected as candidate targets from the source, and provides more accurate and clear initial constraints for subsequent 3D point cloud extraction and pose propagation updates of the spine, thereby improving the reliability and consistency of the overall detection and localization results.

[0077] Specifically, the process of extracting the corresponding three-dimensional point cloud candidate set of the spine based on the correspondence of the spine candidate regions in the depth image includes: obtaining the corresponding depth value in the depth image based on the pixel coordinates of the spine candidate regions in the color image, and generating the three-dimensional point cloud candidate set of the spine through back projection calculation.

[0078] By accurately locating the pixel coordinates of candidate spine regions in color images and simultaneously acquiring the corresponding depth information in depth images, and then reconstructing a candidate set of 3D point clouds of the spine through back projection, a reliable mapping of the spine from 2D images to 3D space is achieved. This allows for a complete expression of the spine's spatial position, scale, and hierarchical relationship, thus providing a stable data foundation for subsequent spine pose propagation, spatial constraint modeling, and precise positioning. It avoids the problems of occlusion misjudgment, distance distortion, and spatial ambiguity caused by relying solely on 2D images, and significantly improves the completeness and accuracy of spine spatial positioning.

[0079] Please see Figure 2 The diagram shows a flowchart of the joint optimization estimation process in this embodiment. In this embodiment, under the prior constraints of the device attitude provided by the inertial measurement data, the process of jointly optimizing the spatial position and orientation of each propagation node based on the parallelism, adjacent spacing, and depth continuity relationships between propagation nodes includes: constructing each propagation node as a vertex of a pose graph model; determining the connections between vertices to construct edges of the pose graph based on the bookshelf space topology and the spatial proximity relationships of the propagation nodes; defining multiple types of geometric constraint residuals based on the parallelism, adjacent spacing, and depth continuity relationships between adjacent vertices connected by the edges; fusing the multiple types of geometric constraint residuals with the prior constraints of the device attitude to construct a nonlinear optimization objective function; and iteratively solving the objective function while synchronously updating the position and orientation parameters of all vertices to perform joint optimization estimation.

[0080] In this embodiment, based on the candidate sets of 3D point clouds of each spine, an initial pose, including position, is estimated for each point cloud through principal component analysis (PCA) and plane fitting (such as the RANSAC algorithm). and posture (This can be represented using quaternions or rotation matrices). The initial pose state of each spine is defined as a vertex to be optimized. Based on the bookshelf spatial topology (especially shelf affiliation) and the lateral proximity of vertices projected onto the image's two-dimensional plane (e.g., pixel center distance within 50 pixels), the connectivity between vertices is determined, thereby constructing an undirected pose graph. Where V is the set of vertices, Let be the set of edges, where each edge represents a spatial constraint between a pair of adjacent spines. For each edge in the pose graph... According to the vertices it connects and Current pose estimation, calculation of residuals with various geometric constraints: parallel constraint residuals Assuming through the vertex The corresponding point cloud fitting yields the unit normal vector of the spine plane. Then the parallel constraint residual is defined as the norm of the cross product of the two normal vectors, that is: ,in, For propagation nodes The unit normal vector of the spine plane; the residual of the adjacent spacing constraint.

[0081] Set vertex The center point of the corresponding spine plane is The spacing constraint residual is defined as the deviation of the distance on the horizontal plane (XY plane) from the expected standard spacing dref (which can be set according to common spine thickness, such as 20mm), that is: ,in, The center point of the spine plane of propagation node j; depth continuity constraint residual. This constraint penalizes abrupt changes in discontinuity between adjacent spines in the depth direction (Z-axis), and its residual can be defined as: ,in The depth difference threshold is used to determine the continuity threshold. When the depth difference is less than the depth difference threshold, the residual is zero.

[0082] The depth difference threshold is an allowable distance parameter used to determine whether there is an abnormal abrupt change in the depth direction between adjacent spines. It depends on the measurement noise characteristics of the depth camera used and the thickness variation range of the spine itself. It is usually set between 5 mm and 30 mm. In this embodiment, it is set to 15 mm, which can effectively distinguish between normal spine thickness changes and depth jumps caused by occlusion or measurement errors. This penalizes discontinuous spatial pose estimation during the optimization process and enhances the geometric rationality of the results.

[0083] Simultaneously, by pre-integrating the IMU data, the relative pose transformation of the sensor itself at adjacent time points and its information matrix are obtained. This allows us to construct prior constraint residuals for device posture, thereby constraining the optimal pose of the spine to be near a physically feasible motion trajectory.

[0084] By combining the residuals of all the above constraints in a weighted manner, a global nonlinear least squares optimization objective function is constructed:

[0085] ,

[0086] in, To balance the weighting coefficients of different constraint strengths, Using Mahalanobis distance, this embodiment employs the Levenberg-Marquardt algorithm to iteratively solve the objective function. In each iteration, the residual at the current pose is linearized, the linear equations are solved to obtain the increment of the poses of all vertices, and the result is updated until the overall pose change is less than a preset convergence threshold. Finally, the optimized set of consistent spine candidate poses is output.

[0087] Regarding weighting coefficients These are adjustable parameters used to balance the relative importance of the adjacent spacing constraint residuals and the depth continuity constraint residuals in the overall objective function. They depend on the specific sensor noise level, the confidence level of the prior information on the spine size, and the preset priority of different constraint terms, and are typically set between 0.1 and 10.0. In this embodiment, based on experimental calibration, they are set to 1.5 and 0.8 respectively, which can effectively suppress abnormal jumps in the depth direction while ensuring the reasonableness of the adjacent spine spacing, making the optimization process both geometrically consistent and robust. Regarding the preset convergence threshold, it is the accuracy standard for determining whether the nonlinear optimization iteration process terminates. It depends on the accuracy requirements of the pose estimation in the application scenario, the sensor measurement noise level, and the system real-time requirements. The convergence threshold of the position component is usually set to... Between meters, the convergence threshold of the attitude components is usually set at... In this embodiment, the position change is set to be less than 0.001 meters and the attitude angle change is less than 0.005 radians. This ensures that the pose output achieves millimeter and sub-degree level accuracy while avoiding unnecessary iterative calculations and guaranteeing the overall efficiency of the algorithm.

[0088] The continuity threshold is the maximum allowable distance difference between adjacent propagation nodes in the depth direction. It depends on the ranging accuracy of the depth camera and the actual thickness range of the spine. It is usually set between 5mm and 30mm. In this embodiment, it is set to 15mm, which can effectively suppress the impact of depth abrupt changes caused by noise points or occlusion on the stability of neighborhood propagation updates.

[0089] In this embodiment, the prior device attitude constraints provided by the inertial measurement data refer to using the three-axis angular velocity and three-axis acceleration information output by the IMU in real time to calculate the relative pose transformation between adjacent image frames through a pre-integration algorithm, and then fusing it with the pose change estimated by visual depth to provide a globally consistent attitude reference framework for the pose optimization of the spine node, thereby constraining the overall pose drift.

[0090] By constructing a pose graph optimization model that integrates multiple geometric constraints and device attitude priors, high-precision and consistent joint estimation of the spatial poses of multiple book spines in a bookshelf is achieved. This method not only significantly improves the overall accuracy and geometric consistency of pose estimation, but also enhances the algorithm's robustness to environmental disturbances such as depth noise and occlusion by introducing physical constraints. Its structured graph optimization framework also makes solving complex problems more efficient and reliable.

[0091] Specifically, after obtaining the candidate spine pose set, the method further includes: calculating the pose confidence of each propagation node based on the residual in the iterative solution process; and filtering the poses in the candidate spine pose set based on the pose confidence.

[0092] In this embodiment, after obtaining the candidate spine pose set, the residuals generated by various geometric constraints and parallelism, spacing, and depth continuity constraints during the nonlinear optimization iterative solution process of each propagation node are recorded. The pose confidence score of each propagation node is calculated by statistical analysis of these residuals. Subsequently, the candidate pose set is screened using the pose confidence score, and poses with low confidence or instability are eliminated. Only candidate poses that are highly consistent with the spatial topology, the geometric relationship between adjacent spines, and the pose perturbation information are retained, thereby ensuring that the finally selected spine pose has higher reliability and accuracy in three-dimensional space.

[0093] By calculating and filtering pose confidence based on residuals, the system can effectively quantify the stability and accuracy of each propagation node during the iterative optimization process, achieving a comprehensive evaluation of the spatial position, orientation, spacing between adjacent layers, parallelism, and depth continuity of the spine. This not only eliminates abnormal poses that may be caused by acquisition noise or pose disturbances, but also ensures that the retained candidate poses are consistent in parameters such as shelf height, spine normal, main axis direction, and spatial arrangement. This results in a stable and coherent structure in the final spine pose set in three-dimensional space, further improving the accuracy and reliability of book detection and positioning. At the same time, it can naturally reflect the inherent relationship between the bookshelf geometry and the spine arrangement.

[0094] Please see Figure 3 The flowchart shown is a process for calculating the pose confidence of each propagation node in this embodiment. In this embodiment, the process of calculating the pose confidence of each propagation node based on the residuals in the iterative solution process includes: constructing a residual statistical distribution based on the final constraint residuals of each propagation node in the nonlinear optimization objective function; calculating the pose confidence score of each propagation node according to the residual statistical distribution and the preset confidence interval; and normalizing the pose confidence score to obtain the corresponding pose confidence.

[0095] The preset confidence interval refers to the acceptable range used to evaluate the constraint residuals of each propagation node. Its function is to determine whether the node pose deviation is within a reasonable range. Specifically, the upper and lower limits of this interval can be determined based on the depth camera ranging accuracy, the range of spine thickness variation, and the tolerance of the neighborhood propagation constraints. For example, it can usually be set between ±5mm and ±30mm. In this embodiment, the mean of the residual distribution ±3 times the standard deviation is taken as the dynamic confidence interval, which can reasonably weight or eliminate abnormal residuals when calculating the pose confidence.

[0096] In this embodiment, the pose confidence calculation process includes: collecting the final constraint residuals of each propagation node in the nonlinear optimization objective function. Each node corresponds to a residual vector or scalar, which directly characterizes the degree to which its current pose estimation satisfies multiple geometric constraints and prior device attitude constraints; constructing a global residual statistical distribution based on the final constraint residuals of all nodes; calculating the key statistical moments of this distribution, including its arithmetic mean and standard deviation. These statistics characterize the overall error level and dispersion of pose estimation in the current scene; calculating a pose confidence score for each propagation node according to the constructed statistical distribution; setting a dynamic confidence interval based on the above statistics, and considering a node to be relatively reliable if its residual falls into this interval. The core principle of the score calculation is that the smaller the node's residual and the closer it is to the center of the distribution, the higher its initial score should be. Specifically, the relative position of the node's residual can be mapped to a non-negative initial score value using a function based on exponential decay or reciprocal relationship; and normalizing the initial scores of all nodes. The score of each node is divided by the sum of the scores of all nodes, so that the sum equals one, thus normalizing the score into a pose confidence level between zero and one. This confidence level is used as a quantitative indicator and is directly used for subsequent screening of candidate spine poses.

[0097] By utilizing the residual statistics inherent in the optimization process, an adaptive quality assessment system is constructed, transforming abstract optimization results into quantifiable confidence indices. This enables the system to automatically identify and eliminate low-quality pose estimates caused by noise, occlusion, or poor fitting. By employing a dynamic confidence interval based on the overall statistical distribution, the system can adapt to error levels in different scenarios, enhancing its robustness to environmental changes. Ultimately, this mechanism significantly improves the reliability and stability of the entire positioning system's output.

[0098] Specifically, the process of introducing coplanar constraints on the spine plane, consistent constraints on the spine normal, and collinear constraints on the spine principal axes, combined with the instantaneous attitude perturbation information represented by the inertial measurement data, drives the pose coordination and convergence of each propagation node in the joint optimization estimation. This process includes: the coplanar constraints on the spine plane are constructed based on the point-to-plane distance between the spine planes corresponding to adjacent propagation nodes; the consistent constraints on the spine normal are constructed based on the angle between the normal vectors of the spine planes corresponding to adjacent propagation nodes; the collinear constraints on the spine principal axes are constructed based on the angle between the directions of the spine principal axes corresponding to adjacent propagation nodes; under the instantaneous attitude perturbation constraints provided by the inertial measurement data, the coplanar constraints on the spine plane, the consistent constraints on the spine normal, and the collinear constraints on the spine principal axes are jointly incorporated into the nonlinear optimization objective function for solution.

[0099] In this embodiment, for each pair of adjacent propagation nodes connected by pose graph edges, a geometric analysis is first performed on the corresponding 3D point cloud of the spine. Principal component analysis is used to extract the first principal component direction of the point cloud, which is taken as the principal axis direction of the spine. Simultaneously, a random sampling consensus algorithm is used to robustly fit the plane containing the spine, obtaining the unit normal vector and the center point of the plane. Based on these geometric properties, three types of constraint residuals are constructed. First, the spine plane coplanarity constraint residual is constructed by calculating the average distance from multiple sampling points in the spine point cloud of node one to the fitted spine plane of node two. Second, the spine normal consistency constraint residual is constructed by calculating the angle between the unit normal vectors of the fitted spine planes of the two nodes. Third, the spine principal axis collinearity constraint residual is constructed by calculating the angle between the extracted spine principal axis direction vectors of the two nodes. At the same time, the angular velocity and acceleration data collected by the inertial measurement unit are processed. By pre-integrating the IMU data between adjacent image frames, the relative rotation and translation changes of the sensor during this time are calculated, and its covariance matrix is ​​estimated, thereby constructing constraint residuals characterizing the instantaneous motion disturbance of the device itself. Finally, the coplanar constraint residuals based on point-to-surface distance, the normal consistency constraint residuals based on the angle between normal vectors, the collinear constraint residuals based on the angle between principal axes, and the attitude disturbance constraint residuals based on IMU pre-integration are all introduced as new optimization terms into the original nonlinear optimization objective function. A nonlinear least squares optimization algorithm, such as the Levenberg-Marquardt algorithm, is used to iteratively solve the extended objective function, which includes all geometric and physical constraints, thereby synchronously and collaboratively optimizing the poses of all propagation nodes and driving them to converge to a globally consistent and physically reliable solution.

[0100] By strengthening the inherent strong geometric priors and physical motion consistency of book arrangement in a bookshelf scenario, the originally isolated and noise-sensitive single-point estimation is transformed into a mutually correcting and globally consistent collaborative perception. This solution not only significantly improves the estimation accuracy of the 3D pose of the book spine and the pose consistency among multiple spines by utilizing fine geometric constraints, but also effectively suppresses errors introduced by device jitter and motion blur during the acquisition process by integrating the physical motion constraints of the IMU. This significantly enhances the overall robustness of the system and the reliability of the output results in dynamic and complex lighting environments, thus providing directly reliable and accurate spatial information input for subsequent automated operations such as robot grasping and inventory.

[0101] Please see Figure 4The flowchart shown is a process for outputting the spatial position and posture information of the target book in this embodiment. In this embodiment, the process of outputting the spatial position and posture information of the target book based on the spine pose includes: converting the spine pose to a unified coordinate expression in the world coordinate system; determining the shelf to which the target book belongs based on the three-dimensional coordinates of the converted spine pose in the world coordinate system and the height sequence of the shelves in the bookshelf space topology; determining the horizontal arrangement order of the target book on the shelf based on the horizontal coordinates of each spine pose within the same shelf; and formatting and encapsulating the three-dimensional coordinates in the world coordinate system, the shelf to which the target book belongs, the horizontal arrangement order, and the posture components in the corresponding spine pose as the output spatial position and posture information of the target book.

[0102] In this embodiment, firstly, the spine poses output by the joint optimization estimation process are obtained, initially expressed based on the sensor coordinate system. Using a pre-calibrated fixed transformation matrix from the sensor coordinate system to the world coordinate system, the coordinates of each spine pose are transformed, unifying its position and orientation to a world coordinate system referenced to the bookshelf upright plane or the room floor. Next, based on the constructed bookshelf spatial topology, the shelf height sequence, composed of the center height values ​​of each shelf, is read. The vertical coordinates of each spine in the world coordinate system are matched with the shelf height sequence, and the specific shelf number to which the spine belongs is determined by calculating the minimum height difference. Then, the spine poses belonging to the same shelf are processed. The two-dimensional coordinates of these spines in the horizontal plane of the world coordinate system are extracted, typically arranged in ascending or descending order according to their direction coordinates along the bookshelf, thus determining the horizontal arrangement order index of each spine on that shelf. Finally, the above information is encapsulated in a structured manner. The world coordinate system three-dimensional coordinates, rotational attitude parameters, shelf number, and horizontal sequence index within each spine are organized according to a predefined field format to form a complete record of the book's spatial position and attitude information. This record serves as the final output of the system and can be directly used for robot path planning or visualization.

[0103] By combining a pre-built bookshelf space topology, each precise six-DOF spine pose is assigned a semantic label with "shelf to which it belongs" and "lateral arrangement order," enabling robots or management systems not only to know the precise location and posture of a book, but also to understand its logical position within the overall bookshelf layout. The final formatted and packaged output eliminates data ambiguity and provides a plug-and-play standardized interface, greatly improving the integration efficiency with subsequent modules such as robot grasping planning, inventory database updates, or visual monitoring, as well as the reliability of the entire automated workflow.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatic book detection and localization based on multimodal fusion, characterized in that, include: Acquire color images, depth images, and inertial measurement data of the bookshelf area, spatially align the color images and depth images, and model the bookshelf upright plane, shelf height sequence, and spacing between adjacent shelves based on the depth images to construct the bookshelf spatial topology. Based on the color image, at least one candidate spine region is extracted using a preset spine semantic segmentation model, and a corresponding candidate set of three-dimensional point clouds of the spine is extracted according to the correspondence between the candidate spine regions in the depth image. In the bookshelf spatial topology, each candidate set of three-dimensional point clouds of the book spine is used as a propagation node. Under the prior constraints of the device attitude provided by the inertial measurement data, the spatial position and orientation of each propagation node are jointly optimized and estimated based on the parallel relationship, adjacent spacing relationship and depth continuity relationship between the propagation nodes. In the process of joint optimization estimation, constraints such as coplanarity of the spine plane, consistency of the spine normal, and collinearity of the spine principal axes are introduced. Combined with the instantaneous attitude perturbation information represented by the inertial measurement data, the pose coordination of each propagation node is driven in the joint optimization estimation to obtain a converged set of candidate spine poses. Extract the visual feature vector of the spine from the color image region corresponding to each candidate spine pose, and match it with the template features in the preset book feature library to bind the spine visual identity with the corresponding three-dimensional pose to obtain the spine pose. Output the spatial position and posture information of the target book based on the spine pose.

2. The method for automatic book detection and localization based on multimodal fusion according to claim 1, characterized in that, The process of acquiring color images, depth images, and inertial measurement data of the bookshelf area, and spatially aligning the color and depth images includes: The acquisition of the color image, the depth image, and the inertial measurement data is triggered synchronously by a unified timestamp, and the coordinate transformation of the depth image is performed using the extrinsic parameter matrix between the depth camera and the color camera to achieve spatial alignment with the color image.

3. The method for automatic book detection and localization based on multimodal fusion according to claim 2, characterized in that, The process of modeling the bookshelf upright plane, shelf height sequence, and spacing between adjacent shelves based on depth images to construct the bookshelf spatial topology includes: The planar equation of the bookshelf uprights is determined by plane fitting, the height sequence of the shelves is determined by layered clustering of the projection distribution of the depth point cloud in the vertical direction, and the spacing between adjacent shelves is calculated based on the height difference between adjacent shelves to construct the spatial topology of the bookshelf.

4. The method for automatic book detection and localization based on multimodal fusion according to claim 3, characterized in that, The process of extracting at least one candidate spine region using a pre-defined spine semantic segmentation model includes: The color image is input into the preset spine semantic segmentation model, which outputs a spine pixel-level segmentation mask and generates at least one spine candidate region based on the segmentation mask.

5. The method for automatic book detection and localization based on multimodal fusion according to claim 4, characterized in that, The process of extracting the corresponding 3D point cloud candidate set of the book spine based on the correspondence between the candidate regions of the book spine in the depth image includes: Based on the pixel coordinates of the candidate spine region in the color image, the corresponding depth value is obtained in the depth image, and the candidate set of the three-dimensional point cloud of the spine is generated by back projection calculation.

6. The method for automatic book detection and localization based on multimodal fusion according to claim 5, characterized in that, Under the prior constraints of device attitude provided by the inertial measurement data, the process of jointly optimizing and estimating the spatial position and orientation of each propagation node based on the parallel relationship, adjacent spacing relationship, and depth continuity relationship between propagation nodes includes: Each of the propagation nodes is constructed as a vertex of the pose graph model; Based on the bookshelf space topology and the spatial proximity of the propagation nodes, the connections between vertices are determined to construct the edges of the pose graph; Based on the parallelism, spacing, and depth continuity between adjacent vertices connected by the edges, multiple types of geometric constraint residuals are defined. By integrating the residuals of the various geometric constraints with the prior constraints of the device attitude, a nonlinear optimization objective function is constructed. By iteratively solving the objective function, the position and orientation parameters of all vertices are updated synchronously to perform joint optimization estimation.

7. The method for automatic book detection and localization based on multimodal fusion according to claim 6, characterized in that, After obtaining the candidate pose set of the spine, the method further includes: Based on the residuals of each propagation node during the iterative solution process, calculate its corresponding pose confidence. Based on the pose confidence level, the poses in the candidate spine pose set are filtered.

8. The method for automatic book detection and localization based on multimodal fusion according to claim 7, characterized in that, The process of calculating the pose confidence of each propagation node based on its residual during the iterative solution includes: Based on the final constraint residuals of each propagation node in the nonlinear optimization objective function, a residual statistical distribution is constructed. Based on the residual statistical distribution and the preset confidence interval, calculate the pose confidence score of each propagation node; The pose confidence score is normalized to obtain the corresponding pose confidence score.

9. The method for automatic book detection and localization based on multimodal fusion according to claim 8, characterized in that, The process of introducing coplanar constraints on the spine plane, consistent constraints on the spine normal, and collinear constraints on the spine principal axes, and combining the instantaneous attitude perturbation information represented by the inertial measurement data, to drive the pose coordination convergence of each propagation node in the joint optimization estimation includes: The coplanar constraint of the spine plane is constructed based on the point-to-plane distance between the spine planes corresponding to adjacent propagation nodes; The consistent spine normal constraint is constructed based on the angle between the spine plane normal vectors corresponding to adjacent propagation nodes; The collinear constraint of the spine main axis is constructed based on the included angle between the spine main axis directions corresponding to adjacent propagation nodes; Under the instantaneous attitude disturbance constraints provided by the inertial measurement data, the coplanar constraint of the spine plane, the consistent constraint of the spine normal, and the collinear constraint of the spine main axis are all incorporated into the nonlinear optimization objective function for solution.

10. The method for automatic book detection and localization based on multimodal fusion according to claim 9, characterized in that, The process of outputting the spatial position and posture information of the target book based on the spine pose includes: Transform the spine pose into a unified coordinate representation in the world coordinate system; Based on the three-dimensional coordinates of the converted spine pose in the world coordinate system, and combined with the height sequence of the shelves in the bookshelf space topology, the shelf to which the target book belongs is determined. Based on the horizontal coordinates of the spine positions of each book within the same shelf, determine the horizontal arrangement order of the target books on the shelf; The three-dimensional coordinates in the world coordinate system, the corresponding shelf, the horizontal arrangement order, and the attitude components in the corresponding spine pose are formatted and encapsulated to output the spatial position and attitude information of the target book.

Images