Computer vision-based book inventory method and system

Abstract

The application relates to the technical field of computer vision, and discloses a book inventory method and system based on computer vision. The method comprises the following steps: acquiring bookshelf multi-source sensing data through a three-dimensional scanning device, and generating standardized point cloud data through processing; using a multi-modal fusion network to jointly extract book surface features and identifier information, and outputting an identification result containing a unique identifier; combining a bookshelf topological model to solve a book orientation angle and a coordinate offset, and obtaining accurate pose data; using a probabilistic roadmap algorithm to generate a robot motion trajectory based on the pose data, and correcting the trajectory online to obtain a robust inventory path; tracking a book loss or misplacement event in real time according to the path, and generating an incremental inventory report by associating a time sequence log. The application realizes high-precision book identification and state diagnosis, and improves the inventory automation level.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a book inventory method and system based on computer vision. Background Technology

[0002] The inventory work of existing libraries or large bookshelves is gradually being automated to replace purely manual operations. Typically, color images and point cloud data of the bookshelves are first acquired using depth cameras, followed by a serial processing flow: first, optical character recognition or barcode scanning is attempted to obtain book identification; if recognition fails, simple visual features are used for book matching. For localization, most methods simply use object detection algorithms to capture the position of the book spine in the image and then simply map it to 3D space to obtain a rough center point coordinate.

[0003] The aforementioned existing technical solutions have shortcomings. Recognition methods based on serial processing suffer from poor stability when faced with complex lighting conditions, blurred or worn spine text, or partial occlusion. When the initial identifier recognition step fails, subsequent matching methods based on simple visual features are prone to mismatches or recognition failures due to insufficient feature discrimination, making it difficult to achieve high-accuracy fully automated inventory in real library environments. Regarding book status assessment, the coarse three-dimensional coordinate information provided by existing methods cannot accurately reflect the specific placement posture of books. Due to the lack of modeling of the bookshelf's physical structure, the system cannot understand the spatial constraints between books and shelf panels, or between books themselves. Therefore, it is difficult to effectively detect abnormal placement states such as tilting, protrusion, or denting of books, which are crucial for determining whether books are misplaced or require repositioning.

[0004] The current technological bottleneck lies in the fact that the goal of book inventory is not only to identify books, but also to diagnose their physical condition on the shelves. Existing methods separate identification from condition assessment. The identification process does not fully utilize multi-source information, while the condition assessment process lacks the necessary environmental context information as a basis for reasoning. This results in the final inventory result only providing a list of whether books are present or not, failing to generate a practically informative incremental inventory report. Summary of the Invention

[0005] The purpose of this invention is to provide a computer vision-based book inventory method and system to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides a computer vision-based book inventory method, the method comprising:

[0007] Multi-source sensing data of the bookshelf is captured by a 3D scanning device, and spatial alignment and noise filtering are performed on the multi-source sensing data to generate a standardized point cloud dataset.

[0008] A multimodal fusion network is used to jointly extract and parse the book surface features and identifier information in the standardized point cloud dataset, and output a set of recognition results containing the unique identifier of the book.

[0009] By integrating the bookshelf topology model and the recognition result set, the orientation angle and coordinate offset of the books are calculated to obtain the book pose dataset;

[0010] Based on the book pose dataset, a probabilistic route map algorithm is applied to generate the motion trajectory of the inventory robot, and sensor observation data is inserted to correct the trajectory online, outputting a robust inventory path.

[0011] Based on the robust inventory path, the anomaly detection unit is triggered to track book loss or misplacement events during the inventory process and generate an incremental inventory report by associating it with the time-series inventory log.

[0012] Preferably, the step of capturing multi-source sensing data of the bookshelf using a 3D scanning device, and performing spatial alignment and noise filtering on the multi-source sensing data to generate a standardized point cloud dataset includes:

[0013] Acquire synchronous data streams from depth cameras and RGB cameras, perform timestamp matching and coordinate system transformation operations, and generate initial fused data;

[0014] The initial fused data is subjected to a voxel grid downsampling algorithm to reduce the point cloud density, and a statistical outlier removal method is used to filter out noise points to generate a denoised point cloud.

[0015] The denoised point cloud is registered with the baseline model by using the iterative nearest point algorithm, the transformation matrix is ​​calculated, and a standardized point cloud dataset is output.

[0016] Preferably, the step of using a multimodal fusion network to jointly extract and parse the book surface features and identifier information in the standardized point cloud dataset, and outputting a recognition result set containing the unique identifier of the book, includes:

[0017] A graph convolutional network model is constructed to aggregate node features in point cloud data and extract the geometric contours and texture features of the book.

[0018] An attention mechanism is used to weight and fuse geometric contour and texture features to generate an enhanced feature vector;

[0019] The book identifier character stream is identified by predicting the sequence labels in the enhanced feature vector using a connectionist temporal classification decoder.

[0020] By merging geometric contour features and identifier recognition results, a set of unique book identifiers is generated using feature splicing technology.

[0021] Preferably, the construction of the graph convolutional network model aggregates the node features in the point cloud data and extracts the geometric contours and texture features of the book, including:

[0022] The point cloud data is divided into local regions, and a k-nearest neighbor graph structure is constructed for each region.

[0023] The node features are iteratively updated through graph convolutional layers, neighbor node information is aggregated, and a local feature map is output.

[0024] A global max pooling layer is applied to reduce the dimensionality of the local feature map, generating the geometric contour and texture feature vectors of the book.

[0025] Preferably, the integrated bookshelf topology model and the recognition result set are used to calculate the orientation angle and coordinate offset of the books to obtain a book pose dataset, including:

[0026] Load the 3D mesh model of the bookshelf and map the recognition result set to the mesh vertex space;

[0027] Calculate the Euclidean distance between the centroid of the book and the grid vertex, and solve for the translation vector of the book relative to the reference position;

[0028] The principal component analysis algorithm is used to estimate the principal orientation of the book point cloud, calculate the orientation angle offset, and output the book pose dataset.

[0029] Preferably, the step of generating the motion trajectory of the inventory robot based on the book pose dataset using a probabilistic route mapping algorithm, and inserting sensor observation data to perform online correction of the trajectory, outputting a robust inventory path, includes:

[0030] Discretize the book pose dataset into a configuration space, randomly sample nodes and connect feasible edges to construct a probabilistic route graph;

[0031] The Dijkstra algorithm is used to search for the shortest path on the probabilistic route map to generate the initial motion trajectory.

[0032] Real-time pose data is acquired using lidar and inertial measurement unit, and the initial motion trajectory is corrected using an extended Kalman filter.

[0033] By combining the dynamic obstacle prediction results, the model predictive control algorithm is applied to optimize the trajectory curvature and output a robust inventory path.

[0034] Preferably, the step of using the Dijkstra algorithm to search for the shortest path on the probabilistic route map and generate the initial trajectory includes:

[0035] Assign distance weights to each node in the probabilistic roadmap, and set the start and end nodes;

[0036] Iteratively expand neighboring nodes and update the shortest path tree;

[0037] Traverse the path node sequence to generate the initial motion trajectory.

[0038] Preferably, the step of triggering the anomaly detection unit to track book loss or misplacement events during the inventory process based on the robust inventory path, and generating an incremental inventory report by associating it with the time-series inventory log, includes:

[0039] Real-time capture of 3D image data of inventory scenes, and feature matching with the recognition result set;

[0040] The optical flow method is used to calculate the motion vector of books and detect abnormal position events.

[0041] Record the spatial coordinates and time stamps of abnormal events;

[0042] Integrate historical status changes from time-series inventory logs and use a differential algorithm to generate incremental inventory reports.

[0043] Preferably, the application of optical flow to calculate the book's motion vector and detect positional anomalies includes:

[0044] Corner features are extracted from consecutive image frames, and the motion displacement of the feature points is calculated using the Lucas-Cornard method.

[0045] Compare the motion displacement with the threshold range to identify abnormal motion areas;

[0046] Clustering outlier regions generates a list of outlier events.

[0047] Preferably, the present invention also includes a computer vision-based book inventory system, the system including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the computer program, implements the steps of the computer vision-based book inventory method described above.

[0048] Compared with the prior art, the beneficial effects of the present invention are:

[0049] A multimodal fusion network is employed to jointly extract and parse book surface features and identifier information from a standardized point cloud dataset. This network performs deep fusion at the feature level, allowing geometric point cloud data to provide spatial context for visual feature localization, while high-resolution texture information provides a basis for distinguishing books with similar geometric features. This processing mechanism can leverage the overall visual appearance of the spine for complementary reasoning when identifier information is incomplete or of poor quality; conversely, when visual feature similarity is high, accurate identifier information can serve as the decisive basis for judgment. This deep cross-modal interaction improves the recognition success rate and stability in real-world scenarios such as complex lighting, partial occlusion, and spine wear, avoiding the overall recognition interruption caused by the failure of a single modality in traditional serial processes, thus providing a technical foundation for achieving fully automatic high-precision recognition.

[0050] This system integrates a bookshelf topology model and a set of book recognition results to calculate the orientation angle and coordinate offset of books. The bookshelf topology model introduces spatial constraints that books should follow, evaluating each detected book within a continuous physical context. By comparing the actual point cloud of the book with the ideal position predicted by the model, the precise orientation angle of the book relative to the shelf plane and its coordinate offset relative to adjacent book rows can be calculated. This calculation method elevates simple object detection to the level of spatial relationship diagnosis, and the generated pose dataset directly quantifies the degree of abnormal book placement. Based on this, the system can perceive subtle pose changes such as slight forward tilting, backward tilting, or protruding from the row. This information is crucial for determining whether a book has been misplaced or needs to be reset, exceeding the ability of traditional methods that can only provide existence checks. This represents a leap from inventory lists to operational status reports. Attached Figure Description

[0051] Figure 1 This is a schematic diagram illustrating the working principle of the computer vision-based book inventory method described in this invention.

[0052] Figure 2 A flowchart for generating a standardized point cloud dataset;

[0053] Figure 3 A flowchart for identifying unique book identifiers using a multimodal fusion network;

[0054] Figure 4 Analysis diagram of book pose calculation results;

[0055] Figure 5 This is a diagram showing the analysis of the robot path planning results. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please see Figure 1 This invention provides a computer vision-based book inventory method, comprising: capturing multi-source perception data of a bookshelf using a 3D scanning device; performing spatial alignment and noise filtering on the multi-source perception data to generate a standardized point cloud dataset; employing a multimodal fusion network to jointly extract and parse the book surface features and identifier information in the standardized point cloud dataset, outputting a recognition result set containing unique book identifiers; integrating the bookshelf topology model and the recognition result set to calculate the orientation angle and coordinate offset of the books, obtaining a book pose dataset; based on the book pose dataset, applying a probabilistic route map algorithm to generate the motion trajectory of the inventory robot, and inserting sensor observation data to perform online correction of the trajectory, outputting a robust inventory path; and, based on the robust inventory path, triggering an anomaly perception unit to track book loss or misplacement events during the inventory process, and generating an incremental inventory report by associating it with a time-series inventory log.

[0058] Example 1: See Figure 2 In practical implementation, the synchronous data stream acquisition process of the depth camera and RGB camera relies on hardware trigger signals and a precise time synchronization module. The depth camera uses active stereo vision or time-of-flight principles to acquire the depth information of the scene, generating a depth map containing distance information for each pixel. The RGB camera synchronously captures high-resolution color images of the same scene. The timestamp matching operation is driven by a high-precision clock on the embedded system, assigning a globally unique and strictly increasing timestamp identifier to each frame of depth and RGB data. The coordinate system transformation operation is based on the intrinsic and extrinsic parameter matrices obtained from camera calibration. The intrinsic parameter matrix contains internal geometric parameters of the camera, such as focal length and principal point, while the extrinsic parameter matrix defines the rotation and translation relationship between the depth camera coordinate system and the RGB camera coordinate system. By multiplying the 3D point cloud data in the depth camera coordinate system by the extrinsic parameter matrix and transforming it to the RGB camera coordinate system, and then mapping it to the 2D pixel coordinate system of the RGB image through perspective projection transformation, pixel-level alignment of depth and color information is achieved, thereby generating initial fused data. The initial fused data is a point cloud data structure containing 3D spatial coordinates and corresponding RGB color values.

[0059] In practice, a voxel grid downsampling algorithm is applied to the initial fused data. This algorithm divides the 3D point cloud space into a series of uniform cubic grids, or voxels. All points within each voxel are replaced by a single representative point, which is typically the geometric center of all points within that voxel or obtained by calculating the average coordinates of all points within the voxel. The voxel size in the voxel grid downsampling algorithm is a configurable parameter. The choice of voxel size requires a trade-off between reducing data volume and preserving necessary geometric details. A larger voxel size leads to more aggressive data compression but may lose subtle features, while a smaller voxel size preserves more details but is computationally more demanding. In practice, the voxel size is dynamically set based on the average point spacing of the original point cloud and the accuracy requirements of subsequent processing tasks. A statistical outlier removal method is used to filter out noisy points. This method analyzes the average distance between each point and its k nearest neighbors, assuming these distances follow a normal distribution. The average distance and standard deviation of the entire point cloud are calculated. Any point whose average distance is greater than the global average distance plus a threshold of n times the standard deviation is considered an outlier and removed from the point cloud. In statistical outlier removal methods, the number of neighboring points *k* and the multiplier factor *n* are key parameters that need to be adjusted based on the point cloud density and noise level. The point cloud data generated after voxel grid downsampling and statistical outlier removal is called a denoised point cloud, which has lower data density and a higher signal-to-noise ratio.

[0060] In practical implementation, the denoised point cloud is registered with a reference model using the Iterative Closest Point Algorithm (TLA). The reference model is a precise 3D digital model of the bookshelf unit, which can be a computer-aided design model or a reference point cloud pre-acquired by a high-precision 3D scanner. The TLA is an iterative optimization algorithm used to calculate the optimal rigid body transformation, i.e., rotation matrix and translation vector, between two point cloud datasets. The basic steps of the TLA include: in each iteration, finding the nearest neighbor point in the reference model point cloud for each point in the denoised point cloud as a corresponding point pair; then calculating the rotation matrix and translation vector that minimizes the average distance between all corresponding point pairs; updating the position of the denoised point cloud using the calculated transformation matrix; repeating the above process until the average distance change is less than a preset convergence threshold or the maximum number of iterations is reached. The TLA is sensitive to the initial position, so in practical applications, it is usually combined with coarse registration methods such as feature descriptor-based matching to provide a good initial pose estimate. The transformation matrix output by the TLA after successful registration contains the rotation and translation parameters required to align the denoised point cloud to the reference model coordinate system. This transformation matrix is ​​applied to perform coordinate transformation on the original denoised point cloud, ultimately outputting a standardized point cloud dataset. All points in the standardized point cloud dataset lie in a unified coordinate system consistent with the baseline model, with consistent scale, orientation, and origin.

[0061] In some embodiments, synchronization between the depth camera and the RGB camera can be achieved through a dedicated synchronization signal line. Specifically, the depth camera generates a hardware trigger signal during exposure and sends it to the RGB camera. Upon receiving the trigger signal, the RGB camera immediately begins image acquisition. In other embodiments, synchronization can also be achieved through software. The host computer simultaneously sends acquisition commands to both the depth camera and the RGB camera, recording the timestamp of the command issuance as the timestamp of the data frame. The intrinsic and extrinsic parameter matrices involved in the coordinate system transformation operation need to be obtained in advance through the camera calibration process. Camera calibration uses a calibration board with a known geometric pattern. Multiple images of the calibration board at different positions and orientations are captured, and algorithms such as the Zhang Zhengyou calibration method are used to calculate the camera's intrinsic parameters and relative pose.

[0062] It's understandable that voxel mesh downsampling algorithms not only reduce the amount of point cloud data but also smooth the point cloud surface to some extent, because each point within a voxel is represented by a center point, which helps eliminate some minor noise. Statistical outlier removal methods can effectively filter isolated noise points caused by sensor errors, environmental dust, or light interference; these noise points are usually far from the main point cloud clusters. The registration accuracy of the iterative nearest-point algorithm directly affects the accuracy of subsequent book recognition and pose estimation. Therefore, selecting an appropriate convergence threshold and maximum number of iterations is crucial, while also considering the normal vector and curvature information of the point cloud to improve the search and matching quality of corresponding point pairs.

[0063] Optionally, a radius filtering step can be added before voxel mesh downsampling. Radial filtering removes points whose number of neighboring points within a specified radius is less than a threshold. This filtering method is particularly effective for removing sparse, floating noise points. Optionally, for the iterative nearest point algorithm, a point-to-plane error metric can be used instead of the standard point-to-point error metric. The point-to-plane error metric can accelerate convergence and improve registration accuracy when the point cloud surface is relatively smooth. It minimizes the distance from a point in the source point cloud to the local tangent plane of the target point cloud.

[0064] It is understandable that generating a standardized point cloud dataset is the foundation of data preprocessing for the entire book inventory process, and the quality of the standardized point cloud dataset directly determines the performance of subsequent feature extraction and recognition. Spatial alignment of multi-source sensing data ensures that information from different modalities can be correctly correlated, and noise filtering improves the purity of the data, enabling subsequent algorithms to run on a more reliable data basis. The design of the entire processing flow fully considers the sensor errors and environmental interference that may exist in the actual application environment. Through a series of rigorous mathematical operations and algorithmic processing, the original, noisy multi-source sensing data is transformed into a clean, coordinate-uniform standardized point cloud dataset.

[0065] Example 2: See Figure 3In practical implementation, the process of constructing a graph convolutional network model begins by representing a standardized point cloud dataset as graph-structured data. Each 3D point in the point cloud is defined as a node in the graph, and edges between nodes are established based on spatial proximity. A k-nearest neighbor graph structure is constructed for each node. This structure calculates the Euclidean distance between each point in the point cloud and all other points, selects the k nearest neighbors as their neighbors, and establishes connecting edges. Each node's initial feature vector can contain attributes such as the point's 3D coordinates, color information, normal vector, and local curvature. The graph convolutional network model consists of multiple stacked graph convolutional layers. Each layer transforms and aggregates the input node features. The graph convolution operation linearly transforms the features of the central node and its neighbors using a learnable weight matrix. Then, pooling functions such as max pooling or average pooling are used to aggregate the transformed features of the neighbors. Finally, the aggregated neighbor features are combined with the transformed features of the central node itself, typically through summation or concatenation operations, to generate a new feature representation of the central node. Through iterative updates of multiple graph convolutional layers, the feature vector of each node can capture the geometric and textural information of its local neighborhood and even the global context. Finally, a global max pooling layer is applied to reduce the dimensionality of the feature vectors of all nodes. The global max pooling layer takes the maximum value of each feature channel along the node dimension and outputs a global feature vector of fixed dimension. This global feature vector serves as the overall representation of the geometric contour and texture features of the book.

[0066] In practical implementation, an attention mechanism is used to weightedly fuse geometric contour and texture features. This attention mechanism is typically implemented using a self-attention module. The self-attention module takes the geometric contour and texture feature vectors extracted by the graph convolutional network model as input and generates a query vector, key vector, and value vector through three learnable linear transformation matrices. The dot product of the query vector and all key vectors is calculated, and then a scaling operation and a softmax function are applied to obtain the attention weight distribution, which represents the importance of different feature dimensions. The value vector is then weighted and summed using the attention weights to generate a weighted feature representation. This self-attention mechanism can dynamically capture the dependencies between different parts within the feature vector, thus adaptively weighting and fusing geometric contour and texture features. This highlights feature channels with strong discriminative power for book recognition tasks, suppresses redundant or noisy features, and ultimately generates an enhanced feature vector. The enhanced feature vector integrates the macroscopic shape information and microscopic surface details of the book, resulting in stronger representational capabilities.

[0067] In practical implementation, a connectionist temporal classification decoder is used to predict sequence labels in the enhanced feature vector. This decoder is specifically designed to handle sequence learning problems where the lengths of the input and output sequences are inconsistent. The enhanced feature vector is first mapped to a higher-dimensional feature space through one or more fully connected layers, and then input into a recurrent neural network layer, such as a long short-term memory network or a gated recurrent unit. The recurrent neural network layer processes the input sequence step by step, capturing long-term dependencies within the sequence. The output of the recurrent neural network layer at each time step is passed through a linear transformation layer and a softmax activation function, generating a probability distribution on a predefined character set. The loss function of the connectionist temporal classification decoder directly calculates the conditional probability between the input sequence features and the output character sequence during training, without requiring forced alignment of the input and output sequences. During the inference phase, a beam search algorithm is used to search for the most probable character sequence from the probability distribution output by the connectionist temporal classification decoder. The beam search algorithm maintains multiple candidate sequences and iteratively expands and prunes them, ultimately selecting the sequence with the highest overall probability as the identified book identifier character stream. The book identifier character stream can be an ISBN, book code, or book title text, etc.

[0068] In practice, the geometric contour features and identifier recognition results are combined using a feature concatenation technique. This involves concatenating the geometric contour feature vector extracted from the graph convolutional network model with the feature vector of the identifier character stream identified by the connectionist temporal classification decoder after being transformed by an embedding layer. The geometric contour feature vector represents the global features characteristic of the book's physical form, while the identifier feature vector represents the semantic features characteristic of the book's identity information. The feature concatenation operation connects these two feature vectors, which are from different sources and have complementary semantics, along the feature dimension to form a comprehensive feature representation. This concatenated high-dimensional feature vector is then dimensionality-reduced and fused using a fully connected layer, ultimately generating a recognition result containing the book's unique identifier. The recognition result set is stored in a structured format, with each record containing the book's unique identifier, spatial location confidence in the point cloud, and other information.

[0069] In some embodiments, graph convolutional network models can incorporate edge convolution operations. Edge convolution not only aggregates features from neighboring nodes but also explicitly considers the relative spatial relationships between the central node and each neighboring node, such as the direction vector and length of the edges, encoding these geometric relationships into the feature learning process. In some embodiments, the attention mechanism can employ a multi-head attention mechanism. This mechanism segments the feature vector into multiple subspaces, independently computes attention weights in each subspace, and finally concatenates the outputs of the multiple heads to capture diverse feature relationships in different subspaces.

[0070] It is understandable that graph convolutional network models can directly process non-Euclidean data such as point clouds. By effectively modeling the local geometric relationships between points through graph structures, the learned features exhibit good robustness to transformations such as rotation and translation of the point cloud. Connectionist temporal classification decoders avoid the step of precise character segmentation required in traditional optical character recognition processes, making them particularly suitable for text recognition in complex situations such as font variations, lighting conditions, and occlusion in natural scenes. Feature concatenation technology simply and effectively fuses features from different modalities and levels of abstraction, providing a comprehensive and discriminative representation for books. Skip connections can be added to graph convolutional network models. Skip connections fuse features from shallow and deep graph convolutional layers, helping to alleviate the gradient vanishing problem in deep network training while preserving multi-scale feature information. Optionally, for connectionist temporal classification decoders, a convolutional neural network layer can be added before the recurrent neural network layer. The convolutional neural network layer can extract local correlations of input features and has a certain degree of translation invariance, which helps improve the stability of sequence features.

[0071] It is understandable that multimodal fusion networks, through end-to-end training, unify and optimize multiple steps such as point cloud feature extraction, identifier recognition, and feature fusion within a single framework, enabling each module to collaborate and jointly optimize the final book recognition target. The combination of graph convolutional network models' ability to model the local structure of point clouds and connectionist temporal classification decoders' ability to process sequential information effectively allows for the simultaneous extraction of both the physical form and identity information of books from point cloud data. The introduction of an attention mechanism enables the network to adaptively focus on the feature regions most important to the current recognition task, improving the efficiency and robustness of feature representation. The entire recognition process design fully considers the characteristics of point cloud data and the actual needs of book recognition, achieving high-precision extraction of unique book identifiers.

[0072] In practice, dividing point cloud data into local regions is typically based on spatial clustering algorithms, such as Euclidean clustering. Euclidean clustering clusters points based on the Euclidean distance between them, grouping points with a distance less than a set threshold into the same local region. Each local region contains a set of spatially continuous and neighboring point clouds, which may correspond to the spine, cover, or a local surface of a book. A k-nearest neighbor graph structure is constructed for each local region. This structure uses each point within the local region as a graph node, calculates the distance from that node to all other points within the region, selects the k nearest neighbors as its neighbors, and establishes connecting edges. The k-nearest neighbor graph structure effectively captures the spatial proximity relationships and local geometry within the local region.

[0073] In practice, node features are iteratively updated through graph convolutional layers, with each layer executing a message passing mechanism. In each graph convolutional layer, each node receives feature information from its neighbors and aggregates these neighbors' features with its own, thereby updating its feature representation. The specific operations of the graph convolutional layer include linear transformations of the features of the central node and each neighbor's features, followed by the aggregation of the transformed features of all neighbor nodes using a symmetric aggregation function such as max pooling, average pooling, or summation pooling. Finally, the aggregated neighbor features are combined with the transformed features of the central node itself, for example, through addition or concatenation, and then passed through a non-linear activation function such as ReLU to output the new features of the node updated by the graph convolutional layer. By stacking multiple graph convolutional layers, the receptive field of each node gradually expands, enabling the aggregation of information from more distant neighbors, thus learning point cloud features from local to global perspectives. After iterative updates through multiple graph convolutional layers, a local feature map corresponding to each local region is output, containing the depth feature representation of each node.

[0074] In practice, a global max pooling layer is applied to reduce the dimensionality of the local feature map. The global max pooling layer maximizes the value of each feature channel of all nodes in the local feature map along the node dimension. Specifically, assuming the local feature map contains N nodes, and the feature vector of each node has a dimension of D, the global max pooling layer outputs a D-dimensional vector. The value in each dimension of this D-dimensional vector is the maximum value of the feature values ​​of all N nodes in that dimension. The global max pooling operation makes the output feature vector invariant to the order of points in the input point cloud and captures the most salient feature activations in the point cloud. The final generated book geometric contour and texture feature vector is a fixed-length global descriptor that summarizes the key shape and texture information of the entire point cloud or a local region, suitable for subsequent classification, recognition, or retrieval tasks.

[0075] Example 3: In specific implementation, the process of loading the 3D mesh model of the bookshelf involves reading a pre-built model file from a persistent storage device. The 3D mesh model typically uses a triangular mesh representation, consisting of a vertex set, an edge set, and a face set. The vertex set contains the 3D spatial coordinates of each vertex, the edge set defines the connection relationships between vertices, and the face set is formed by three vertex indices constituting a triangular face. The 3D mesh model is obtained by accurately modeling with computer-aided design software or by scanning and reconstructing the physical bookshelf using a high-precision 3D scanner. The 3D mesh model fully expresses the structural dimensions of the bookshelf, the shelf positions, and the spatial range of each book placement compartment. Mapping the recognition result set to the mesh vertex space is a spatial projection process. The recognition result set contains the 3D position information of each recognized book in the point cloud coordinate system. The 3D coordinates of the books are transformed to the coordinate system of the 3D mesh model using a coordinate transformation matrix. After conversion, for each book, it is necessary to find one or more mesh vertices in the vertex space of the 3D mesh model that are closest to its projected position, and establish an association mapping relationship between the book and the mesh vertices. This mapping relationship provides a spatial reference framework for subsequent calculation of the offset of the book relative to the bookshelf reference position.

[0076] In practice, calculating the Euclidean distance between the book's centroid and the grid vertices is the core step in solving the translation vector problem. The book's centroid is obtained by calculating the average coordinates of all point cloud data points belonging to that book, using the following formula:

[0077] ;

[0078] in: A three-dimensional coordinate vector representing the center of gravity of the book. This represents the total number of points in the point cloud belonging to this book. Indicates the first The calculated centroid of the book represents its center of mass in space. The book's reference position corresponds to the theoretical position of its centroid on the 3D mesh model when the book is correctly placed on the bookshelf. This theoretical position can be predefined at a specific vertex of the 3D mesh model. The translation vector of the book relative to the reference position is calculated directly through vector subtraction:

[0079] ;

[0080] in: It is the calculated translation vector. These are the coordinates of the book's center of gravity. These are the coordinates of the corresponding reference grid vertex. Translation vector. It includes the offset of the book relative to its correct position in the X, Y, and Z coordinate axes.

[0081] In practical implementation, the principal orientation of the book point cloud is estimated using Principal Component Analysis (PCA), a statistical method used to extract the main feature orientation from multidimensional data. PCA first calculates the covariance matrix of the book point cloud data, reflecting the variance of the point cloud in each dimension and the covariance relationship between different dimensions. Eigenvalue decomposition is then performed on the covariance matrix, yielding three eigenvalues ​​and three corresponding eigenvectors. These three eigenvectors are pairwise orthogonal and sorted in descending order of their corresponding eigenvalues. The eigenvector corresponding to the largest eigenvalue is the first principal component orientation of the point cloud; this orientation represents the direction with the largest variance in the point cloud distribution. For a regularly shaped book, the first principal component orientation is typically along the spine. Calculating the orientation angle offset requires comparing the estimated principal orientation of the book point cloud with a reference orientation. The reference orientation is the theoretical direction that the book's principal orientation should point to when it is correctly placed on a bookshelf; this reference orientation can also be predefined in the 3D mesh model. The orientation angle offset is obtained by calculating the angle between the principal orientation vector and the reference orientation vector, reflecting the angle of rotation of the book around its normal axis. The final output book pose dataset is a structured collection of data. Each record corresponds to a recognized book and contains a unique identifier for the book and the calculated translation vector. and orientation angle offset It fully describes the current position and orientation of each book on the bookshelf, and the deviation from its standard position.

[0082] In some embodiments, the vertices of the 3D mesh model can be appended with attribute information, such as vertex color, texture coordinates, or normal vectors. This attribute information can be used for more refined visualization rendering or to assist in calculations. In some embodiments, the determination of the reference position mesh vertices can be based on the book's ISBN or classification number. By querying a predefined mapping table, the book identifier is mapped to specific vertex coordinates on the 3D mesh model. In some embodiments, for books with irregular shapes or special bindings, a more refined 3D model can be pre-built for them, and weighting factors can be introduced into the principal component analysis algorithm to weight the calculation of the covariance matrix according to the local curvature or density of the points, so as to estimate the principal direction more accurately.

[0083] Understanding this approach involves mapping the recognition result set to the vertex space of a high-precision 3D mesh model, establishing a precise, digital spatial reference system that allows for accurate comparison between the book's actual and theoretical positions. The translation vector is calculated by determining the Euclidean distance between the centroid and the reference point; this method is intuitive and computationally efficient, accurately reflecting the book's linear displacement deviation. Principal component analysis (PCA) robustly extracts the main orientation from the geometric distribution of the point cloud, exhibiting robustness to density variations and local noise, resulting in stable and reliable orientation angle estimations.

[0084] Optionally, before calculating the translation vector, the book point cloud can be preprocessed. For example, a radius-based outlier removal algorithm can be applied to filter out isolated noise points that may be caused by recognition errors, ensuring that the calculated centroid is more representative. It is understandable that generating the book pose dataset is a crucial step in the entire book inventory system. The pose dataset is not only used to determine whether a book is misplaced, but also provides accurate target point coordinates and posture information for subsequent robot path planning. The translation vector and orientation angle offset together constitute the six-degree-of-freedom pose description of the book, completely depicting the actual state of the book in the bookshelf space. By combining the visual recognition results with a precise bookshelf topology model, the pose calculation process transforms abstract image or point cloud information into spatial metric data with clear physical meaning, providing a quantitative basis for automated inventory decisions and execution.

[0085] See Figure 4 This visualization showcases the book pose calculation results based on a 3D mesh model and point cloud data. The image clearly presents the 3D structural framework of the bookshelf and the spatial distribution of the books. Green dots mark the theoretical reference positions of the books on the shelf, representing the ideal coordinates where the books should be correctly placed. Red triangles show the actual positions of the books obtained through point cloud analysis, intuitively reflecting their true arrangement on the shelf. Blue directional arrows indicate the main orientations of the books calculated using principal component analysis, representing the actual angles of the books in space. Purple dashed lines connect the theoretical and actual positions, clearly showing the offset vector of each book relative to its standard position. This visualization verifies the effectiveness of the pose calculation algorithm, accurately identifying book positional deviations and directional offsets, providing precise spatial reference data for subsequent mis-shelf detection and robot path planning. This intuitive display of 3D spatial relationships allows for rapid assessment of the book arrangement status and guides subsequent automated operations.

[0086] Example 4: In specific implementation, discretizing the book pose dataset into a configuration space is a fundamental step in path planning. The configuration space is a mathematical space whose dimension is determined by the number of degrees of freedom of the inventory robot. For a robot moving in three-dimensional space, the configuration space typically includes position coordinates and attitude angles. The book pose dataset provides the position and attitude information of each book in the bookshelf coordinate system. This information is quantized into a set of discrete points in the configuration space, each point representing a specific configuration that the robot needs to achieve when inventorying that book. The random sampling node process generates a large number of random points within the safe area of ​​the configuration space. The safe area is determined by the bookshelf geometry model and the robot body model through a collision detection algorithm, ensuring that the robot pose corresponding to the sampling point will not interfere with the bookshelf. The feasible edge connection operation checks whether there is a collision-free path segment between every two random sampling nodes. The feasibility of the edge is determined by performing dense collision detection along the connection line. Finally, a network structure composed of nodes and feasible edges is constructed, namely the probabilistic route graph. The probabilistic route graph covers all possible safe movement areas of the robot from the starting point to the ending point.

[0087] In practical implementation, the Dijkstra algorithm is used to search for the shortest path on the probabilistic path graph. The Dijkstra algorithm is a classic single-source shortest path algorithm. Distance weights are assigned to each node in the probabilistic path graph, based on Euclidean distance between nodes, motion time cost, or other optimization metrics. A start and end node are set; the start node corresponds to the robot's initial pose at the start of inventory, and the end node corresponds to the final pose after completing the inventory task. The Dijkstra algorithm initializes the distance to the start node to 0, and the distances to other nodes to infinity. It then iteratively expands neighboring nodes, selecting the unvisited node with the smallest current distance at each step and updating the distance estimates of all its neighboring nodes to the start node via that node. This process continues until the end node is visited, ultimately constructing a shortest path tree from the start node to all nodes. The backtracking path sequence starts from the end node and traces backward along the shortest path tree back to the start node, connecting the traversed node sequences to generate the initial motion trajectory. The initial motion trajectory is a series of discrete pose points describing the expected path of the robot's motion.

[0088] In practical implementation, real-time pose data is acquired through LiDAR and an Inertial Measurement Unit (IMU). The LiDAR measures distance information of the surrounding environment by emitting a laser beam and receiving reflected signals, generating point cloud data. The IMU, containing a gyroscope and accelerometer, measures the robot's angular velocity and linear acceleration. An Extended Kalman Filter (EKF) is used to correct the initial motion trajectory. The EKF is a nonlinear state estimator that fuses multi-sensor data through two steps: prediction and update. The prediction step uses IMU data and the robot's motion model to predict the pose state at the next moment. The update step uses the difference between the actual observation data from the LiDAR and the predicted value to correct the state estimate. Through continuous iteration of the prediction and update process, the EKF outputs a more accurate real-time robot pose estimate. This estimate is used to correct the initial motion trajectory online, compensating for accumulated errors caused by wheel slippage and uneven ground during trajectory execution.

[0089] In practical implementation, referring to Table 1 and combining the dynamic obstacle prediction results, dynamic obstacle prediction estimates the future trajectory of moving objects in the environment by analyzing continuous multi-frame LiDAR point cloud data and using target tracking algorithms such as Kalman filtering or multi-hypothesis tracking. Model predictive control (MMC) algorithms are applied to optimize trajectory curvature. MMC is a model-based control strategy that predicts the future behavior of the system within a finite time frame and continuously optimizes the control input. The MMC algorithm establishes a dynamic model of the robot, using the current state as initial conditions, and predicts the robot's trajectory under different control inputs over a future period. The optimal control sequence is calculated by optimizing the objective function, which typically includes terms such as trajectory tracking error, control magnitude, distance constraints to obstacles, and trajectory smoothness. The optimized trajectory has smoother curvature changes, avoiding sharp turns and abrupt accelerations, while actively avoiding predicted dynamic obstacles, ultimately outputting a robust inventory path. The robust inventory path is a series of timestamped pose points that guide the robot to complete the inventory task safely and smoothly.

[0090] Table 1: Key parameter settings in the probabilistic route map construction and path search process

[0091]

[0092] In some embodiments, the construction of probabilistic roadmaps can employ different sampling strategies, such as a bridge-test-based sampling strategy. This strategy tends to generate more sampling points in narrow passage regions, improving the connectivity of the roadmap in complex environments. When searching for the shortest path using the Dijkstra algorithm, distance weights can be combined with a heuristic function, and the A algorithm can be used for the search. The A algorithm, by introducing the estimated cost to the destination as heuristic information, can find the optimal path more quickly. Optionally, for sensor data fusion, in addition to the extended Kalman filter, an unscented Kalman filter or a particle filter can also be used. The unscented Kalman filter handles nonlinear problems through unscented transformation and may have higher estimation accuracy in some strongly nonlinear scenarios.

[0093] It is understandable that the probabilistic path graph algorithm transforms the complex path planning problem into a search problem on a discrete graph structure by discretizing and sampling a continuous high-dimensional configuration space, greatly reducing computational complexity. The Dijkstra algorithm guarantees finding the global shortest path from the starting point to the destination on the probabilistic path graph, providing an optimal baseline trajectory for robot motion. The extended Kalman filter achieves robust pose tracking by fusing the advantages of LiDAR and inertial measurement unit (IMU). LiDAR provides accurate absolute position references but has a low update frequency, while IMU provides high-frequency relative motion increments but suffers from drift errors. The model predictive control algorithm, through online rolling optimization, not only considers the current tracking error but also predicts future system states and environmental changes, enabling the robot to proactively adapt to the dynamic environment and generate smooth, safe, and executable robust inventory paths.

[0094] See Figure 5 This diagram showcases the path planning and trajectory optimization results of the inventory robot in its workspace. The figure fully presents the boundary range of the robot's working environment, the distribution of obstacles, and the planned optimal motion path. The black boundary line defines the robot's movable working area, and the red polygonal areas represent obstacles to be avoided. The blue scatter dots represent feasible nodes randomly sampled in the configuration space, generated using a probabilistic route graph algorithm. The gray lines show feasible connections between these nodes. The thick green solid line represents the shortest path found using the Dijkstra algorithm, which avoids all obstacles from start to finish while ensuring efficient movement. Orange star markers indicate key turning points on the path, providing precise navigation references for robot motion control. The cyan dashed line shows the smooth trajectory optimized by the model predictive control algorithm. This trajectory ensures safety while providing smoother curvature changes, avoiding sharp turns and drastic acceleration changes. This multi-level path planning method ensures that the robot can safely and efficiently complete the book inventory task in complex environments.

[0095] In Example 5, real-time capture of stereoscopic image data of the inventory scene is accomplished through a binocular camera system. This system consists of two parallel RGB cameras that synchronously acquire left and right view images of the scene at a fixed frame rate. A depth map is generated, and a stereo matching algorithm is used to calculate disparity and convert it into distance information. The feature matching process with the recognition result set uses scale-invariant feature transform or directional fast rotation brief descriptor algorithms to extract key point features from the current stereoscopic image. These features are then compared with the book feature descriptors stored in the recognition result set for similarity. Feature matching employs a nearest neighbor search strategy, setting a distance ratio threshold to filter out incorrect matches, ensuring that only high-confidence corresponding points are retained for state change detection.

[0096] In the specific implementation, optical flow is applied to calculate the book's motion vector. Based on the assumptions of constant brightness and spatial consistency, optical flow estimates the motion displacement of pixels between consecutive image frames. Corner features are extracted from consecutive image frames using the Harris corner detector or the FeaturesfromAcceleratedSegmentTest algorithm. Corner features refer to points in the image that exhibit significant grayscale changes in two vertical directions, and these points have good invariance to rotation and illumination changes. The Lucas-Cornard method is used to calculate the motion displacement of feature points. The Lucas-Cornard method assumes that the motion displacement of all points within a small spatial neighborhood is consistent, and the motion vector of each feature point is solved using the least squares method. Specifically, for each corner feature, all pixels within a one-pixel window surrounding it are taken, and a system of linear equations is constructed. The displacement vector of this feature point in adjacent frames is calculated through iterative optimization, and the displacement vector includes both horizontal and vertical components.

[0097] In practice, abnormal movement regions are identified by comparing the displacement with a threshold range. The threshold range is derived from historical inventory data and includes the maximum allowable displacement for normal book movement. For each feature point's displacement vector, its amplitude is calculated and compared with a preset threshold. If the displacement amplitude exceeds the threshold, the feature point is marked as an anomaly. Abnormal movement regions are formed by connecting neighboring anomaly points. Morphological operations such as dilation and erosion are used to fill the voids within the region, generating a connected anomaly region mask. Clustering anomaly regions generates an anomaly event list. The clustering algorithm uses density-based spatial clustering combined with noise algorithms or K-means clustering to merge spatially similar anomaly regions into one anomaly event. Each anomaly event records its bounding box coordinates, region area, and average motion vector.

[0098] In practice, the spatial coordinates and timestamps of abnormal events are recorded. Spatial coordinates are extracted from the depth map of the stereo image data, and the image pixel coordinates are transformed to three-dimensional coordinates in the world coordinate system using camera calibration parameters. Timestamps are assigned to each abnormal event using the system's high-precision clock, with the timestamp format being International Standard Time (USST) and accurate to the millisecond level. Relevant information for abnormal events includes the event type (e.g., lost or misplaced books), associated book identifiers, confidence scores, and spatial locations. This data is stored in a structured format in a temporary cache. Historical state changes from the time-series inventory log are integrated. The time-series inventory log is a time-series database that records the complete results of each inventory task, including the book list, location information, and timestamps for each inventory. An incremental inventory report is generated using a differential algorithm. This algorithm compares the current inventory result with the most recent historical inventory result, including differences such as added books, deleted books, and books with changed locations. The incremental inventory report generation process first aligns the data points on the time series by matching them using the unique identifiers of the books. Then, it calculates the symmetry difference and positional changes of the sets, and finally outputs a report file containing a summary of changes, a detailed list of changes, and anomaly analysis.

[0099] In some embodiments, stereo image data can be captured using an active stereo vision system that combines an infrared projector and an infrared camera to improve the accuracy and robustness of stereo matching by projecting structured light patterns. In some embodiments, feature matching can employ deep learning-based feature descriptors, such as superpoints or local feature descriptors, which are trained through neural networks and are better adapted to changes in viewpoint and occlusion.

[0100] Optionally, the optical flow method can be combined with the pyramid optical flow method. The pyramid optical flow method calculates optical flow by constructing an image pyramid at different scales, first estimating coarse motion at a large scale and then refining it at a small scale, improving the estimation accuracy for large displacement motions. Optionally, a motion direction consistency constraint can be introduced during anomaly clustering, clustering only anomaly regions with similar motion directions together to avoid erroneous merging of regions with different motion patterns. Optionally, blockchain technology can be used to store the time-series inventory logs, ensuring the immutability and traceability of log data and enhancing the credibility of inventory reports. The anomaly detection unit achieves continuous monitoring of changes in book status through real-time image analysis and historical data comparison. The optical flow method provides quantitative measurement of motion information, enabling anomaly detection to be independent of static feature matching and capable of capturing transient movement events. The differential algorithm efficiently identifies incremental changes, avoiding the computational overhead of full comparison, and the incremental inventory report provides timely status updates for library management.

[0101] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0102] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A book inventory method based on computer vision, characterized in that, include: Multi-source sensing data of the bookshelf is captured by a 3D scanning device, and spatial alignment and noise filtering are performed on the multi-source sensing data to generate a standardized point cloud dataset. A multimodal fusion network is used to jointly extract and parse the book surface features and identifier information in the standardized point cloud dataset, and output a set of recognition results containing the unique identifier of the book. By integrating the bookshelf topology model and the recognition result set, the orientation angle and coordinate offset of the books are calculated to obtain the book pose dataset; Based on the book pose dataset, a probabilistic route map algorithm is applied to generate the motion trajectory of the inventory robot, and sensor observation data is inserted to correct the trajectory online, outputting a robust inventory path. Based on the robust inventory path, the anomaly detection unit is triggered to track book loss or misplacement events during the inventory process and generate an incremental inventory report by associating it with the time-series inventory log.

2. The computer vision-based book inventory method according to claim 1, characterized in that, The process involves capturing multi-source sensory data of the bookshelf using a 3D scanning device, performing spatial alignment and noise filtering on the multi-source sensory data, and generating a standardized point cloud dataset, including: Acquire synchronous data streams from depth cameras and RGB cameras, perform timestamp matching and coordinate system transformation operations, and generate initial fused data; The initial fused data is subjected to a voxel grid downsampling algorithm to reduce the point cloud density, and a statistical outlier removal method is used to filter out noise points to generate a denoised point cloud. The denoised point cloud is registered with the baseline model by using the iterative nearest point algorithm, the transformation matrix is ​​calculated, and a standardized point cloud dataset is output.

3. The computer vision-based book inventory method according to claim 1, characterized in that, The method employs a multimodal fusion network to jointly extract and parse the book surface features and identifier information from the standardized point cloud dataset, outputting a recognition result set containing unique book identifiers, including: A graph convolutional network model is constructed to aggregate node features in point cloud data and extract the geometric contours and texture features of the book. An attention mechanism is used to weight and fuse geometric contour and texture features to generate an enhanced feature vector; The book identifier character stream is identified by predicting the sequence labels in the enhanced feature vector using a connectionist temporal classification decoder. By merging geometric contour features and identifier recognition results, a set of unique book identifiers is generated using feature splicing technology.

4. The computer vision-based book inventory method according to claim 3, characterized in that, The constructed graph convolutional network model aggregates node features from point cloud data to extract the geometric contours and texture features of the book, including: The point cloud data is divided into local regions, and a k-nearest neighbor graph structure is constructed for each region. The node features are iteratively updated through graph convolutional layers, neighbor node information is aggregated, and a local feature map is output. A global max pooling layer is applied to reduce the dimensionality of the local feature map, generating the geometric contour and texture feature vectors of the book.

5. The computer vision-based book inventory method according to claim 1, characterized in that, The integrated bookshelf topology model and the recognition result set are used to calculate the orientation angle and coordinate offset of the books to obtain a book pose dataset, including: Load the 3D mesh model of the bookshelf and map the recognition result set to the mesh vertex space; Calculate the Euclidean distance between the centroid of the book and the grid vertex, and solve for the translation vector of the book relative to the reference position; The principal component analysis algorithm is used to estimate the principal orientation of the book point cloud, calculate the orientation angle offset, and output the book pose dataset.

6. The computer vision-based book inventory method according to claim 1, characterized in that, Based on the book pose dataset, a probabilistic route mapping algorithm is applied to generate the motion trajectory of the inventory robot, and sensor observation data is inserted to correct the trajectory online, outputting a robust inventory path, including: Discretize the book pose dataset into a configuration space, randomly sample nodes and connect feasible edges to construct a probabilistic route graph; The Dijkstra algorithm is used to search for the shortest path on the probabilistic route map to generate the initial motion trajectory. Real-time pose data is acquired using lidar and inertial measurement unit, and the initial motion trajectory is corrected using an extended Kalman filter. By combining the dynamic obstacle prediction results, the model predictive control algorithm is applied to optimize the trajectory curvature and output a robust inventory path.

7. The computer vision-based book inventory method according to claim 6, characterized in that, The process of using Dijkstra's algorithm to search for the shortest path on a probabilistic route graph and generate an initial trajectory includes: Assign distance weights to each node in the probabilistic roadmap, and set the start and end nodes; Iteratively expand neighboring nodes and update the shortest path tree; Traverse the path node sequence to generate the initial motion trajectory.

8. The computer vision-based book inventory method according to claim 1, characterized in that, Based on the robust inventory path, the anomaly detection unit is triggered to track book loss or misplacement events during the inventory process, and an incremental inventory report is generated by associating it with the time-series inventory log, including: Real-time capture of 3D image data of inventory scenes, and feature matching with the recognition result set; The optical flow method is used to calculate the motion vector of books and detect abnormal position events. Record the spatial coordinates and time stamps of abnormal events; Integrate historical status changes from time-series inventory logs and use a differential algorithm to generate incremental inventory reports.

9. The computer vision-based book inventory method according to claim 8, characterized in that, The application of optical flow to calculate book motion vectors and detect positional anomalies includes: Corner features are extracted from consecutive image frames, and the motion displacement of the feature points is calculated using the Lucas-Cornard method. Compare the motion displacement with the threshold range to identify abnormal motion areas; Clustering outlier regions generates a list of outlier events.

10. A computer vision-based book inventory system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the computer vision-based book inventory method according to any one of claims 1 to 9.

Images