Book group area detection and positioning system and robot based on YOLO

Abstract

The invention belongs to the technical field of computers, and particularly relates to a book group area detecting and positioning system and robot based on YOLO. A set of complete end-to-end technical scheme is provided, the problem of the whole process from image acquisition, model training, edge deployment to result fine screening is systematically solved, and the method is particularly suitable for automatically and accurately positioning the book area of the target layer in a library inventory scene. The book group area detection and positioning system based on YOLO is used for automatically and accurately positioning a book group area of a target checking layer from a bookshelf layer position image which is shot by the library visual checking robot and comprises a plurality of layer positions when being operated by the library visual checking robot. Therefore, the interference of the non-target horizon image on the subsequent automatic checking process is eliminated.

Description

A YOLO-based system for detecting and locating groups of books and a robot Technical Field

[0001] This application belongs to the field of computer technology, and in particular relates to a YOLO-based system for detecting and locating groups of books and a robot. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, automated book management and inventory systems in scenarios such as libraries and smart bookstores are becoming a hot research topic in the industry. Traditional book identification methods mainly rely on physical tag technologies such as barcodes and RFID. While these technologies have improved management efficiency to some extent, they still have problems such as high deployment costs, easily damaged tags, and limited coverage. The limitations of traditional methods are even more apparent, especially in large libraries or places with huge collections.

[0003] In current practical book visual inventory systems, books typically appear in groups within bookshelf images, presenting various complexities: varying tilt angles, overlapping of books, uneven lighting, blurred spine text, and images that are too large and include books from other shelves. Traditional horizontal bounding boxes (HBB) struggle to accurately capture the inaccurate extraction of book groups caused by image tilt and cannot accurately locate the boundaries of tilted books. This leads to ineffective extraction of book groups and inaccurate OCR recognition, severely impacting book title recognition. While oriented bounding boxes (OBB) can better handle tilted targets, practical deployment faces challenges such as high model complexity and slow inference speeds on edge devices. Summary of the Invention

[0004] To address the technical problem of inaccurate book group area detection results in complex and ever-changing library environments in existing library visual inventory systems, this application provides a YOLO-based book group area detection and localization system and robot. This system can comprehensively solve the shortcomings of existing technologies in tilt detection, model compression, edge device deployment, book group area and localization, and can effectively extract the book group area at this level, preventing inventory data contamination.

[0005] In a first aspect, embodiments of this application provide a YOLO-based book cluster region detection and localization system, comprising: a data preprocessing module for performing data augmentation and annotation operations on bookshelf layer images captured by a robot to generate a training dataset; a model training module for receiving the training dataset and training an improved YOLOv8 model to output oriented bounding boxes containing rotation angles for locating book cluster regions in the bookshelf layer images; a model conversion and optimization module for converting the trained model into a target format suitable for deployment on edge computing devices; an edge inference module for loading and running the model in the target format on the edge computing device to perform inference on the bookshelf layer images; and a false detection filtering module for filtering multiple candidate oriented bounding boxes output by the edge inference module and outputting a target bookshelf layer oriented bounding box.

[0006] In this application embodiment, a complete end-to-end technical solution is provided, which systematically solves the problems of the entire process from image acquisition, model training, edge deployment to result screening. It is especially suitable for the automated and accurate positioning of target layer book areas in library inventory scenarios.

[0007] In one embodiment of the first aspect, the training objective of the model training module is to learn to regress oriented bounding boxes for defining a group of books from the input bookshelf layer image based on the training dataset; wherein the oriented bounding boxes are used to crop the image of the target layer.

[0008] In this embodiment, the core learning task and output format of the model are clearly defined, and the direct purpose of model training (output OBB) is closely linked to the final application (cropping target region), ensuring the consistency and goal orientation of the technical solution.

[0009] In one embodiment of the first aspect, the data augmentation operation in the data preprocessing module includes at least random rotation, brightness adjustment, and blurring of the bookshelf shelf images to simulate a complex library environment; in the annotation operation of the data preprocessing module, for the left and right shelf images to be stitched together, the annotation box of the left shelf image completely includes the left edge of the bookshelf, and the annotation box of the right shelf image completely includes the right edge of the bookshelf; wherein, both the left and right shelf images are bookshelf shelf images captured by the robot.

[0010] In this embodiment, the data preprocessing module simulates the complex lighting, angles, and blurred environments of different libraries through data augmentation operations such as random rotation, brightness adjustment, and blurring, thereby improving the model's adaptability to diverse scenes. Targeted augmentation enhances the model's adaptability to real-world, variable environments; standardized annotation ensures that the "book cluster region" learned by the model contains complete physical boundaries, providing an accurate geometric basis for subsequent image stitching.

[0011] In one embodiment of the first aspect, the improved YOLOv8 model integrates a CSPDarknet53 backbone network with an SE attention mechanism to extract and focus on the features of the book cluster region in the bookshelf layer image; a neck network with a weighted bidirectional feature pyramid network is used to fuse the multi-scale features output by the backbone network; the improved YOLOv8 model is also configured to output a detection head that outputs oriented bounding boxes with center points of horizontal and vertical coordinates, width, height, and rotation angle; wherein, during the process of the model training module training the improved YOLOv8 model using the training dataset, a circular smoothing labeling technique is used to handle the periodicity problem of angle regression for the angle output of the detection head.

[0012] In this embodiment, the SE module enhances the model's feature selection capability for key regions; the weighted bidirectional feature pyramid network improves the model's robustness to detecting targets at different scales; the improved five parameters output by the YOLOv8 model enable accurate description of areas where books are tilted; and the circular smoothing labeling technique stabilizes the angle learning process. These improvements work synergistically to achieve a deeper understanding of the preprocessed image data.

[0013] In one embodiment of the first aspect, during the training process, the model training module performs the following loss calculation for each training sample: based on the predicted oriented bounding box parameters output by the improved YOLOv8 model and the labeled true oriented bounding box parameters, calculates the CIoU loss for optimizing the center point coordinates, width, and height of the bounding box; based on the rotation angle of the predicted oriented bounding box and the true rotation angle, uses the circular smoothing label technique to handle the periodicity of the angle and calculates the corresponding angle loss; and combines the CIoU loss and the angle loss to form part of the total regression loss of the model training module.

[0014] In this embodiment, CIoU loss ensures the accuracy of bounding box geometric position regression; the angle loss, combined with ring smoothing labeling technology, specifically addresses the periodicity of angle prediction and boundary abrupt changes. Together, they guide the model to more accurately regress the five-parameter oriented bounding box.

[0015] In one embodiment of the first aspect, the model conversion optimization module is used to: convert the trained model into a neural network model exchange format, and optimize it into a target format using a neural network toolkit, the target format representing a format that can run efficiently on the edge computing device.

[0016] In this embodiment, framework independence is achieved through standardized format conversion, and hardware-level optimization is performed through a dedicated toolchain, enabling the trained high-precision model to adapt to and run efficiently on resource-constrained edge computing devices.

[0017] In one embodiment of the first aspect, the edge inference module performs the following during runtime: compressing the bookshelf shelf image to a preset size to generate compressed image data; when inputting the compressed image into the target format model for inference, employing zero-copy memory management technology to enable the neural network processing unit (NPU) of the edge computing device to read the compressed image data; after obtaining the oriented bounding box coordinates of the model inference output in the target format, restoring them to the original bookshelf shelf image's size coordinate system according to the preset size to output coordinates for positioning and cropping.

[0018] In this embodiment, image compression directly reduces the computational load; zero-copy technology reduces memory bandwidth pressure; these strategies together ensure the real-time inference performance of the system on edge devices.

[0019] In one embodiment of the first aspect, the false detection filtering module is further configured to perform: calculating the area of ​​each candidate oriented bounding box based on the horizontal coordinate center point, vertical coordinate center point, width, height, and rotation angle output by the improved YOLOv8 model, and obtaining the area result of each candidate oriented bounding box; sorting the area results of each candidate oriented bounding box according to the size of the area, and selecting the oriented bounding box with the largest area; if there are multiple oriented bounding boxes with the largest area, calculating the Euclidean distance between the center point of each oriented bounding box with the largest area and the center point of the bookshelf layer image, and selecting the oriented bounding box with the smallest Euclidean distance to the center point of the bookshelf layer image as the target bookshelf layer oriented bounding box.

[0020] In this embodiment, the spatial prior of the target layer typically occupying a major area and being relatively centrally located in the image is utilized. Through simple and efficient geometric calculations, the true target layer bounding box is accurately selected from multiple possible false detection boxes, ensuring the uniqueness and correctness of the output results.

[0021] Secondly, this application also proposes a robot, the robot including a camera, a processor, equipped with the YOLO-based book cluster region detection and localization system as described in the first aspect above, and an edge computing device; wherein, the camera is configured to capture images of the bookshelf shelves; the edge computing device is used to run the YOLOv8 model in the YOLO-based book cluster region detection and localization system and to perform inference on the bookshelf shelf images.

[0022] In one embodiment of the second aspect, the robot is a mobile library visual inventory robot; wherein, when the YOLO-based book group area detection and localization system is run by the library visual inventory robot, it is used to automatically and accurately locate the book group area of ​​the target inventory layer from the multi-layer bookshelf layer images captured by the library visual inventory robot, so as to eliminate the interference of non-target layer images on the subsequent automated inventory process. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 is a simplified structural diagram of the robot provided in an embodiment of this application; Figure 2 is a simplified structural diagram of a YOLO-based book group area detection and localization system provided in an embodiment of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] First, as shown in Figure 1, this application embodiment provides a robot, which includes a camera 13, a processor 10, a YOLO-based book group region detection and localization system as shown in Figure 2, and an edge computing device 15; wherein, the camera is configured to capture images of the bookshelf shelves; the edge computing device is used to run the YOLOv8 model in the YOLO-based book group region detection and localization system and to perform inference on the bookshelf shelf images.

[0032] For example, the robot is a mobile library visual inventory robot; wherein, when the YOLO-based book group area detection and localization system is run by the library visual inventory robot, it is used to automatically and accurately locate the book group area of ​​the target inventory layer from the multi-layer bookshelf layer images captured by the library visual inventory robot, so as to eliminate the interference of non-target layer images on the subsequent automated inventory process.

[0033] Camera 13 can be a high-definition camera. During the process of the library visual inventory robot performing the book inventory task, camera 13 can be used to take pictures of each batch of books to be inventoried on the bookshelves in the library in order to collect the shelf layer images.

[0034] The processor 10 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0035] It should be noted that the YOLO-based book group region detection and localization system in this embodiment can represent a device equipped with a YOLO-based book group region detection and localization algorithm model. This device can be a chip that can be deployed in a library visual inventory robot.

[0036] The aforementioned edge computing device 15 can be an embedded development board or computing module. These edge computing devices have a built-in neural network processing unit (NPU) specifically designed to accelerate inference computation of deep learning models; they are responsible for running optimized and transformed YOLO models to perform book group region detection and localization on real-time images of bookshelves captured by the library visual inventory robot's camera. They possess key characteristics such as low power consumption, miniaturization, and hardware acceleration capabilities, making them suitable for embedding into library visual inventory robots to complete real-time visual computing tasks.

[0037] Referring to Figure 2, this application embodiment provides a YOLO-based book cluster region detection and localization system, mainly including a data preprocessing module 01, a model training module 02, a model conversion and optimization module 03, an edge inference module 04, and a false detection filtering module 05. The data preprocessing module 01 is used to perform data augmentation and annotation operations on bookshelf layer images captured by a library visual inventory robot to generate a training dataset. The model training module 02 is used to receive the training dataset and train an improved YOLOv8 model to output oriented bounding boxes containing rotation angles for locating book cluster regions in the image. The model conversion and optimization module 03 is used to convert the trained model into a target format suitable for deployment on edge computing devices. The edge inference module 04 is used to load and run the model in the target format on the edge computing device to perform inference on the input bookshelf layer images. The false detection filtering module 05 is used to filter multiple candidate oriented bounding boxes output by the edge inference module and output a unique target bookshelf layer oriented bounding box.

[0038] The YOLO-based book cluster region detection and localization system provided in this application offers a complete end-to-end technical solution. It systematically solves the entire process from image acquisition, model training, edge deployment to result refinement, and is particularly suitable for automated and accurate location of target shelf book areas in library inventory scenarios. Through the aforementioned modules, images captured by the library visual inventory robot's camera are processed and detected to extract valid bookshelf layer image data. Each module is meticulously designed to ensure the system's efficiency and reliability.

[0039] In one embodiment, for the aforementioned data preprocessing module 01, the data augmentation operations in the data preprocessing module include random rotation, brightness adjustment, and blurring of the image to simulate the complex environment of a library; its annotation operations use an annotation tool that supports directional bounding boxes and follow annotation specifications; the annotation specifications include: for the left and right layer images that need to be stitched together, their annotation boxes must completely include the left and right edges of the bookshelf, respectively; the left and right layer images are both bookshelf layer images taken by the aforementioned library visual inventory robot.

[0040] In practice, the system collects floor data images taken by the visual inventory robots in various libraries and performs data augmentation techniques on the image data, including but not limited to image Mosaic enhancement, random image rotation, image color adjustment (brightness and darkness transformation), image blur transformation processing, and scaling of the image to a certain ratio.

[0041] For example, in the process of annotating the bookshelf layer diagram, this embodiment can design a special annotation specification: (1) Two layer images (left and right) taken by the robot, the left side needs to include the left edge of the bookshelf, and the right side needs to include the right edge of the bookshelf, so that the spliced ​​image is the layer of the entire bookshelf; at the same time, the left edge of the bookshelf to the right edge of the bookshelf has included the content of the entire target bookshelf layer, so no matter how many books are placed in the layer, there will be no missing books.

[0042] (2) Use a labeling tool that supports OBB (directed bounding box) format: Rolabelimg. This tool supports both HBB (horizontal bounding box) and OBB (directed bounding box) format labeling, which basically covers the required labeling methods.

[0043] (3) At the same time, in order to prevent the image from tilting due to shooting, the OBB format annotation data can record the rotation angle and can correct the image back in the later stage through affine transformation and perspective transformation.

[0044] The technical advantages of this embodiment are as follows: The data preprocessing module 01 has formulated special image acquisition and annotation specifications for the special characteristics of bookshelf layer detection. It requires that the labeled layer images must include the left and right edges of the bookshelf. At the same time, it adopts an annotation tool that supports the OBB format. This is the basic premise and key point to ensure the quality, consistency and effectiveness of the model training data and the final application.

[0045] In one embodiment, the aforementioned model training module 02 is an optimized model training module, and its implementation process can be as follows: the improved YOLOv8 model architecture adopts a multi-scale feature pyramid network, and a multi-scale training strategy is adopted for the dataset. For example, the trained neural network supports image resolutions that can include multiple sizes such as 320*320, 640*640, and 1024*1024; the improved YOLOv8 model architecture's backbone network adopts an improved CSPDarknet53 structure, in which an SE attention mechanism is added.

[0046] The neck network of the improved YOLOv8 model architecture uses a weighted bidirectional feature pyramid network structure (BiFPN structure) to achieve efficient multi-scale feature fusion.

[0047] The improved YOLOv8 model architecture's detection head is designed in OBB format, outputting a five-parameter bounding box containing center point coordinates, width, height, and rotation angle. This oriented bounding box detection head outputs a five-parameter bounding box including the x-coordinate center point, y-coordinate center point, width, height, and rotation angle. For example, cx (x-coordinate center point): represents the horizontal (x-axis) position of the oriented bounding box's center point in the image coordinate system. This coordinate is a pixel value relative to the top-left corner of the image origin, used to locate the box's center horizontally within the image.

[0048] `cy` (center point on the y-axis): Represents the vertical (y-axis) position of the center point of the oriented bounding box in the image coordinate system. This coordinate is the pixel value relative to the top-left corner of the image origin, used to locate the center of the box vertically within the image; `w` (width): Represents the width of the oriented bounding box, i.e., the pixel size of the box in the horizontal direction. This parameter defines the horizontal span of the box, consistent with the x-axis direction of the image; `h` (height): Represents the height of the oriented bounding box, i.e., the pixel size of the box in the vertical direction. This parameter defines the vertical span of the box, consistent with the y-axis direction of the image; `θ` (rotation angle): Represents the rotation angle of the oriented bounding box relative to the x-axis of the image coordinate system. This angle is usually expressed in radians or degrees and is used to describe the tilt direction of the box. For example, θ=0° indicates that the box is parallel to the x-axis, θ=90° indicates that the box is parallel to the y-axis, and negative values ​​or values ​​greater than 90° indicate that the box is tilted in the opposite direction or in a more complex way; wherein, during the process of the model training module training the improved YOLOv8 model with the training dataset, the periodicity problem of angle regression is handled by the Circular Smooth Label technique for the angle output of the detection head.

[0049] The loss function uses CIoU Loss (rotated box loss) and Angle Loss (optimized angle loss). For example, the training parameters in this embodiment can be set as follows: using the AdamW optimizer, the initial learning rate is set to 0.001; a cosine annealing learning rate scheduling strategy is adopted; the batch size can be dynamically adjusted according to the GPU memory, and the maximum supported batch_size=32 is 32; the training cycle is set to 300 epochs, including a warmup phase.

[0050] Calculation steps and process: The training framework of this model has been upgraded in several core aspects based on the YOLOv8 framework. It mainly focuses on the task of rotating object detection. Its training calculation process is an efficient, multi-stage forward and backward propagation process, which aims to accurately calculate the regressed rotating bounding boxes.

[0051] (1) Computation and optimization in forward propagation: a. Multi-scale input and data augmentation: At the beginning of model training, the bookshelf layer images taken by the robot are input. First, according to the multi-scale training strategy (e.g., 320*320, 640*640, 1024*1024), the images are randomly scaled and filled, and supplemented with data augmentation techniques such as Mosaic, random rotation, Gaussian blur, and brightness transformation to improve the generalization ability of the model. At the same time, multi-scale training enables the model to adapt to targets of different sizes.

[0052] b. In the feature extraction of the backbone network, a CSPDarknet53 network structure with an added Squeeze-and-Excitation (SE) attention mechanism is adopted. The size of the input image I is denoted as... ,Right now: (Formula 1) The input image I is fed into the backbone network, which is based on the CSPDarknet53 architecture. Its core consists of multiple CSP modules, which can effectively reduce the amount of computation and increase the gradient flow.

[0053] In each CSP module, the input feature map is denoted as X, where the size of feature map X is denoted as... ,Right now: (Formula 2) After one convolutional module, the feature map is divided into two parts, denoted as follows: , ,in After a series of convolutional layer processing (Block): including a residual module consisting of a series of convolutions, batch normalization, and SiLU activation functions, the following is obtained: and A concatenation operation (denoted as Concat here) is performed, and finally, a convolutional layer (Conv) is used for feature fusion to output a new feature map. ,Right now: (Formula 3) In the key CSP module, the SE attention module is integrated. This module performs compressed computation (global average pooling) on ​​the feature map X. First, after inputting X, the spatial dimension is compressed to obtain the channel compressed descriptor Z, where the nth element is denoted as... ,Right now: (Formula 4) The weights of each channel are learned through two fully connected layers (with dimensionality reduction and restoration in between), using the ReLU function in between. As the activation function, it is finally passed through the Sgimoid function. The normalized weights S are obtained; where W1 represents a real matrix of shape (C / r, C); W2 represents a real matrix of shape (C, C / r), where C is the number of channels and r is the reduction ratio factor; after the above operations, the shape of S is still (C, H', W'): (Formula 5) Finally, through recalibration, the learned channel weights S are multiplied channel by channel with the original input feature image X to generate the corrected feature map. The feature map size remains (C, H', W'), and The formula for calculating the nth element is: (Formula 6) The SE module can effectively adaptively compute and calibrate the response of channel features by explicitly modeling the dependencies between channels, enabling the network to pay more attention to feature channels with richer information and suppress unimportant information channels.

[0054] c. A weighted bidirectional feature pyramid network (BiFPN) is used in the YOLO neck network for multi-scale information fusion. BiFPN fuses features at different resolutions through a bidirectional (top-down + bottom-up) path combined with skip connections. It introduces learnable weights for each input feature, allowing the network to learn the importance of different input features during feature fusion. For the fusion node, the weighted fusion can be represented as: (Formula 7) where Input features; These are the learnable weights corresponding to each input; It is a minimum value to prevent division by zero; It is a characteristic of participating in the fusion; through rapid, non-negative normalization, it ensures the stability of the fusion operation.

[0055] Ultimately, BiFPN outputs a feature pyramid {N3, N4, N5} with multi-scale contextual information, providing rich feature representations for the detection head.

[0056] d. Detection Head and OBB Output: The detection head features a decoupled design, enabling it to handle classification and regression tasks. Specifically, for rotated bounding box detection, the regression branch outputs a five-parameter OBB-format prediction.

[0057] Classification branch: mainly used for probability calculation of the category, that is: a C-dimensional vector, representing the probability that an object exists at this position and the output is of the c-th class, where C is the total number of classes.

[0058] Regression branch: Primarily used for bounding box prediction, a 6-dimensional vector. .in: It is the coordinate offset of the center point; These are the width and height scaling values; It is a predicted value of the rotation angle; It is the confidence level of the target.

[0059] Finally, the sigmoid function is used to convert the relative prediction values ​​into absolute OBB parameters through a decoding operation:

[0060]

[0061]

[0062]

[0063]

[0064] in, The coordinates of the top-left corner of the current grid. These are the width and height of the anchor box. The corresponding angle decoding method.

[0065] e. Optimized Loss Calculation: For example, during the training process of the model training module, the following loss calculation steps are performed for each training sample: Based on the predicted oriented bounding box parameters output by the improved YOLOv8 model and the labeled true oriented bounding box parameters, calculate the CIoU loss used to optimize the center point coordinates, width, and height of the bounding box; Based on the rotation angle of the predicted oriented bounding box and the true rotation angle, use the circular smoothing label technique to handle the periodicity of the angle and calculate the corresponding angle loss; Combine the CIoU loss and the angle loss to form part of the total regression loss of the model training module.

[0066] In practical implementation, the total loss function It is classification loss Confidence loss and regression loss The weighted sum, i.e.: (Formula 8) where the regression loss It consists of two parts: the loss of the rotating frame and the rotation angle loss, that is: (Formula 9) The loss function calculation for CIoU is mainly used to optimize the set position of the rotated bounding boxes (including center point, width, and height). CIoU comprehensively considers the overlap area, center point distance, and aspect ratio to calculate the predicted bounding box. With real frame The loss between the two frames, the formula for calculating the loss of the rotating frame is: (Formula 10) (Formula 11) where, represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c represents the diagonal length of the minimum bounding rectangle; This represents the weighting function, where v represents the parameter that measures the consistency of the aspect ratio.

[0067] The angle loss function calculation with CLS is mainly used to optimize rotation angles. CLS technology transforms the angle regression problem into a classification problem, which can effectively overcome the boundary discontinuity problem caused by the periodicity of angles. Its main calculation process includes: 1. Angle discretization: Discretize the continuous angle range (-90°~90°) into n intervals.

[0068] 2. CLS generation: For true angles Instead of treating it as a single label, it is transformed into a smooth one-hot label vector Y. In the label vector, The interval containing the value and its adjacent intervals are assigned non-zero values, generated by a Gaussian function window, with a peak value at... At this point, the Gaussian function window is as follows: (Formula 12) where yes The corresponding central region value, This is a hyperparameter that controls the smoothness. Finally, the vector Y is normalized so that its elements are all equal to 1.

[0069] 3. Network Prediction and Loss Calculation: The network predicts an N-dimensional vector P, where vector P is represented as... , This represents the predicted probability that the angle belongs to the i-th interval; finally... Focal Loss is used for calculation, that is: (Formula 13) Where, It is an adjustable focusing parameter.

[0070] The design of the angle loss function with CSL transforms sharp boundary labels into a smooth label distribution, making the angle learning process smoother and increasing robustness to noise, which can effectively improve the accuracy of rotating box angle prediction.

[0071] 2) Backpropagation and optimization: After calculating the total loss... Then, the gradient of the loss with respect to all trainable parameters (including the backbone network, BiFPN, detector head, and weights in SE and BiFPN) is calculated using the backpropagation algorithm.

[0072] The AdamW optimizer was chosen to update the parameters. AdamW introduces a weight decay regularization method on top of Adam, which can more effectively prevent overfitting during training. The formula for updating the relevant parameters is as follows: (Formula 14) where, This is the bias correction value for the first moment estimate of the gradient. This is the deviation correction value for the second moment of the gradient. For learning rate, It is the weight decay coefficient. This is a very small value to prevent the denominator from being 0 during calculation.

[0073] In scheduling the learning rate, a pre-annealing strategy is adopted, which is usually combined with the Warmup phase.

[0074] Warmup phase: In the first few epochs of training, the learning rate increases linearly or exponentially from a small value to the initial learning rate. This helps the model maintain stability in the early stages of training.

[0075] Cosine annealing strategy: After the Warmup phase, the cosine annealing strategy is adopted. The learning rate will gradually decay to close to 0 according to the cosine annealing strategy. The calculation formula of the cosine annealing strategy is as follows: (Formula 15) where, The initial learning rate, To minimize the learning rate, It is the current epoch number. It represents the total number of epochs.

[0076] Batch size setting: To fully utilize GPU memory during model training, the batch size is set to be dynamically adjustable, with a maximum supported batch_size of 32. In actual training, gradient accumulation techniques may be used to simulate larger batch sizes.

[0077] In one embodiment, the aforementioned model conversion and optimization module 03 is used to convert the trained model into a neural network model exchange format, and can use a neural network toolkit to optimize it into a target format, wherein the target format represents a format that can run efficiently on the edge computing device.

[0078] As an example, the neural network toolkit of this application embodiment can adopt the RKNN-Toolkit tool, which is a software development kit specifically used for deep learning model conversion, quantization and deployment on its neural network processing unit NPU; in specific implementation, the trained model can be converted into a target format suitable for deployment on edge computing devices in the following ways: (1) Convert the YOLO model trained by PyTorch into the ONNX format model.

[0079] Understandably, ONNX (Open Neural Network Exchange) is an open, cross-platform neural network model exchange format designed to solve the model compatibility problem between different deep learning frameworks and achieve seamless migration and deployment of models on different tools and hardware. In this computation process, PyTorch needs to perform a forward propagation of the model once during the computation and track all the operators executed. These operations, along with the network's weight parameters, are then used to construct a directed acyclic graph according to the ONNX standard format and saved as a .onnx file.

[0080] (2) Operator optimization: Optimize the computation graph structure in the ONNX model; this step is based on the .onnx file obtained in the previous step. The purpose is to simplify the computation graph, reduce the amount of computation and memory access in the inference process, and greatly improve the inference speed of the model.

[0081] (3) Dynamic axis setting: Set appropriate dynamic input dimensions.

[0082] (4) Accuracy verification: The accuracy of the converted ONNX model and the initial PyTorch model is verified and compared to ensure that the accuracy of the model before and after conversion remains consistent. The specific calculation process can be as follows: Process 1. Load the trained PyTorch YOLO model and call torch.load() to load the YOLO model. Process 2. Set the mode to model evaluation mode and call model.eval(). This step is crucial because it will disable behaviors specific to the training stage, such as Dropout and BatchNorm layers, to ensure that the inference process is deterministic and needs to be consistent with the behavior of the converted ONNX model.

[0083] Step 3. Perform forward propagation and trace operators: Call the `torch.onnx.export()` function, which executes the model's `forward` method once with the prepared `dummy_input`. The model begins real computation, with the output starting from the input layer and proceeding through all operations, including convolutions, activation functions, BN layers, and YOLO-specific detection heads. During forward propagation, the PyTorch tracer records all called atomic operators. It records the operation type, input tensor, output tensor, and operator attributes.

[0084] Step 4. Constructing the ONNX computation graph: The tracker in step 3 uses all the collected information to construct a directed acyclic graph (DAG). This graph consists of nodes and edges, where nodes represent operators and edges represent tensors flowing between nodes. The most important aspect is the setting of dynamic axes, specified in the `dynamic_axes` parameter of the `export` function, which enables the model to support dynamic batching and dynamic scaling.

[0085] Step 5. Parameter serialization: All weight parameters of the model are extracted from PyTorch's Tensor format and embedded as constant nodes into the ONNX computation graph. These constant nodes are part of the graph and are saved together with the computation graph.

[0086] Step 6. Finally, the serialized data stream is written to the specified .onnx file for model accuracy verification. Once the verification is passed, the edge computing device can be deployed.

[0087] In one embodiment, for the aforementioned edge inference module 04, the edge inference module executes the following processing flow during runtime to achieve rapid detection of the bookshelf shelf image on an edge computing device with limited resources: (1) Size compression processing: The input bookshelf shelf image is compressed according to a preset ratio to generate compressed image data, so as to reduce the amount of data and computational load of subsequent processing; For example, after the image is captured, the length and width of the image are compressed proportionally, and the compression ratio is 1.5 times. Experiments have shown that it can effectively speed up the detection speed of the image, and at the same time effectively reduce the memory overhead when calculating the image, effectively solving the problem of memory surge caused by the large size of the image.

[0088] As an example, size compression can be performed as follows: (1.1) Image input and hardware decoding: Convert the compressed JPEG image data into the original pixel matrix with minimal CPU overhead. Convert the bookshelf shelf photos captured by the camera into a JPEG encoded stream and call the V4L2 dedicated hardware decoder to realize image input and hardware decoding.

[0089] (1.2) Image preprocessing: The decoded image is converted into the format, size, and data type required for the model input. A new image is obtained by scaling the input image by a factor of 1.5 (width and height). The corresponding compressed width and height are denoted as... The width and height of the original image input are denoted as... ,Right now: (Formula 16) After compressing the width and height, the image pixel values ​​are normalized. First, the mean of the image pixel values ​​is used. With variance The formula for calculating the normalized tensor of an image is as follows: (Formula 17) Where, This represents the normalized pixel values ​​of the image. The pixel matrix representing the original image, normalizing pixel values ​​from [0,255] to [0,1].

[0090] (2) Efficient memory and computation scheduling: When the compressed image is input into the model of the target format for inference, zero-copy memory management technology is adopted to enable the neural network processing unit (NPU) of the edge computing device to directly read the compressed image data, thereby reducing the data migration overhead between the CPU and the neural network processing unit (NPU); and an asynchronous pipeline design is adopted to enable the size compression processing, model inference and subsequent result post-processing tasks to be executed in parallel; as an example, efficient memory and computation scheduling can be performed in the following way: (2.1) Zero-copy memory management technology is used to map the memory where the preprocessed tensor is located to the address space where the NPU is located, without the need to create memory space through the CPU, and the memory block can be directly accessed through the NPU.

[0091] After receiving the inference task, the NPU driver reads the input tensor from the shared memory and then performs the entire forward propagation of the YOLO model on the NPU. After the computation is complete, the results of the output layer are written to the shared memory area of ​​the NPU output. This approach can largely eliminate the overhead caused by expensive data copying between CPU memory and NPU memory.

[0092] Meanwhile, to accelerate inference, asynchronous inference pipeline technology is adopted. After the CPU initiates the current inference task, it can immediately process the post-processing of the previous image or the pre-processing of the next layer image without waiting for the current task to complete, thus realizing pipeline parallelism and greatly enhancing inference and processing efficiency.

[0093] (2.2) The original output of the ONNX model is transformed into the final detection box and label category. The output tensor of the NPU is read from the shared memory. A multi-threaded parallel processing method is adopted to allocate the output of different sizes to different CPU threads for processing.

[0094] (3) Coordinate restoration and output: After obtaining the oriented bounding box coordinates of the model inference output of the target format, restore them to the size coordinate system of the original bookshelf layer image according to the preset ratio, so as to output the final accurate coordinates for positioning and cropping.

[0095] It is understandable that after obtaining the coordinates of the book group region, the relevant coordinates are restored according to the corresponding proportions, and the effective region of the book group is extracted based on the restored coordinates and rotation angle. This operation can effectively reduce the time and memory usage of the model in the calculation process of the book group region.

[0096] Furthermore, regarding the aforementioned false detection filtering module 05, this application embodiment can design the false detection filtering module 05 based on a geometric feature-based false detection filtering mechanism. It is understood that the reason for designing the false detection filtering module 05 in this application is as follows: First, the inventory robot cannot completely capture photos of the entire layer when taking pictures; therefore, multiple photos (two or more) of one layer are needed. Simultaneously, during the shooting process, due to inconsistencies in the fixed-point positions, the positions of the captured images will also differ, resulting in the capture of image information from the upper, lower, left, and right parts of the layer, ultimately interfering with the summarized book information of the target layer. In addition, since multiple detection boxes may be generated during the detection process, it is necessary to filter these multiple detection boxes in such cases.

[0097] In one embodiment, for the above situation, the filtering rules of the false detection filtering module 05 in this embodiment are as follows: Process 51: Based on the horizontal coordinate center point cx, vertical coordinate center point cy, width w, height h and rotation angle θ output by the improved YOLOv8 model, calculate the area of ​​each candidate oriented bounding box to obtain the area result of each candidate oriented bounding box; Process 52: Sort the area results of each candidate oriented bounding box according to the size of the area, and select the oriented bounding box with the largest area; Process 53: If there are multiple oriented bounding boxes with the largest area, calculate the Euclidean distance between the center point of each oriented bounding box with the largest area and the center point of the bookshelf layer image, and select the oriented bounding box with the smallest Euclidean distance to the center point of the bookshelf layer image as the target bookshelf layer oriented bounding box.

[0098] It is understood that in the embodiments of this application, when there are multiple detection boxes in the image captured by the robot (with interference from other layers), the target layer occupies the largest proportion in the captured image, meaning that the detection box is the most complete, while other layers occupy a small part of the image area, and the center point of the detection box is closest to the center point of the image; therefore, we can determine which target detection box is our target by the size of the final recognition box and the Euclidean distance between the center of the detection box and the center point of the image, which can better hit the target detection box and eliminate other false detection interference.

[0099] As an example, the false detection filtering module 05 can execute the filtering rules as follows: (Sub-step 1) Detect the book group region on the input bookshelf layer image. If there is only one detection result, the book group region is extracted directly based on the five parameters of the detection result; if there are multiple detection results (two or more detection results), there are false detection boxes, and false detection box filtering is required. Proceed to sub-step 2.

[0100] (Sub-step 2) Extract multiple detection box results (five parameters: x-axis center point cx, y-axis center point cy, width w, height h, and rotation angle θ). Calculate the area of ​​each detection box using these five parameters, using the following formula: First, use n to represent the total number of detection boxes in the image; i represents the index of the label box, i.e.: (Formula 18) Next, based on the recognition results (five parameters) in each detection box, w and h are extracted to calculate the area of ​​each detection box, i.e.: (Formula 19) Next, the area results for each detection box are summarized and sorted (from largest to smallest), that is: (Formula 20) (Formula 21) (Sub-step 3) Obtain the five-parameter data of the box with the largest area as the positioning data of the target layer; if two maximum values ​​are found, proceed to sub-step 4): Obtain the data with the largest area S as the positioning data of the target layer, that is: (Formula 22) (Sub-step 4) Calculation when there are two identical boxes with the maximum value in the area sorting, extracting the target area of ​​the book group in the accurate layer: First, when calculating the maximum box, it is found that the area values ​​of two boxes are simultaneously the maximum area, that is: (Formula 23) Next, obtain the five parameters corresponding to the top two areas S and extract the coordinates of the center point of the coordinate frame, i.e.: (Formula 24) (Formula 25) Simultaneously, this embodiment also needs to calculate the coordinates of the center point of the entire image, where... To capture the width of the image, The height of the captured image, i.e.: (Formula 26) Next, calculate the Euclidean distance between the center points of target_box1 and target_box2 and the center point of the image, and take the set of five parameters with the smallest Euclidean distance, that is: (Formula 27) (Formula 28) (Formula 29) Finally, the information contained in the final target layer location frame is stored in... middle.

[0101] The technical effect of this embodiment is that the false detection filtering module 05 adopts a false detection filtering mechanism based on the geometric feature filtering strategy of the detection box area and the Euclidean distance at the center end, which effectively eliminates the interference of multiple detection boxes and improves the extraction of the target layer area. This is the core post-processing design to ensure the accuracy of subsequent image stitching and OCR recognition.

[0102] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0104] If the integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0105] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0107] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0108] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A YOLO-based system for detecting and locating book clusters, characterized in that, include: The data preprocessing module is used to perform data augmentation and annotation operations on the bookshelf shelf images taken by the robot to generate a training dataset; The model training module is used to receive the training dataset and train the improved YOLOv8 model so that it outputs oriented bounding boxes containing rotation angles to locate the book group region in the bookshelf layer image; the model conversion and optimization module is used to convert the trained model into a target format suitable for deployment on edge computing devices. The edge inference module is used to load and run the model in the target format on the edge computing device to perform inference on the bookshelf layer image; The false detection filtering module is used to filter multiple candidate oriented bounding boxes output by the edge reasoning module and output the target bookshelf layer oriented bounding box.

2. The system as described in claim 1, characterized in that, The training objective of the model training module is to learn to regress oriented bounding boxes for defining the group of books from the input bookshelf layer images based on the training dataset; wherein the oriented bounding boxes are used to crop the image of the target layer.

3. The system as described in claim 1, characterized in that, The data augmentation operation in the data preprocessing module includes at least random rotation, brightness adjustment, and blurring of the bookshelf shelf images to simulate the complex environment of a library. In the annotation operation of the data preprocessing module, for the left and right shelf images to be stitched together, the annotation box of the left shelf image completely encompasses the left edge of the bookshelf, and the annotation box of the right shelf image completely encompasses the right edge of the bookshelf. Both the left and right shelf images are bookshelf shelf images captured by the robot.

4. The system according to claim 2 or 3, characterized in that, The improved YOLOv8 model integrates a CSPDarknet53 backbone network with an SE attention mechanism to extract and focus on the features of book cluster regions in the bookshelf layer image; a neck network with a weighted bidirectional feature pyramid network is used to fuse the multi-scale features output by the backbone network; the improved YOLOv8 model is also configured to output a detection head that outputs oriented bounding boxes with horizontal center point, vertical center point, width, height, and rotation angle; wherein, during the training of the improved YOLOv8 model by the model training module using the training dataset, the angle output of the detection head is processed using a circular smoothing label technique.

5. The system according to claim 4, characterized in that, During the training process, the model training module performs the following loss calculations for each training sample: Based on the predicted oriented bounding box parameters output by the improved YOLOv8 model and the labeled true oriented bounding box parameters, it calculates the CIoU loss used to optimize the bounding box center point coordinates, width, and height; based on the rotation angle of the predicted oriented bounding box and the true rotation angle, it uses the circular smoothing label technique to handle the angle periodicity problem and calculates the corresponding angle loss; the CIoU loss and the angle loss are combined to form part of the total regression loss of the model training module.

6. The system according to claim 4, characterized in that, The model conversion and optimization module is used to: convert the trained model into a neural network model exchange format and optimize it into a target format, wherein the target format represents a format that can run efficiently on the edge computing device.

7. The system according to claim 1 or 6, characterized in that, The edge inference module performs the following during runtime: compressing the bookshelf layer image to a preset size to generate compressed image data; when inputting the compressed image into the target format model for inference, zero-copy memory management technology is used to enable the neural network processing unit (NPU) of the edge computing device to read the compressed image data; After obtaining the oriented bounding box coordinates of the model inference output in the target format, they are restored to the size coordinate system of the original bookshelf shelf image according to the preset ratio to output coordinates for positioning and cropping.

8. The system according to claim 4, characterized in that, The false positive filtering module is also used to perform the following: based on the horizontal coordinate center point, vertical coordinate center point, width, height and rotation angle output by the improved YOLOv8 model, calculate the area of ​​each candidate oriented bounding box, and obtain the area result of each candidate oriented bounding box; sort the area results of each candidate oriented bounding box according to the size of the area, and select the oriented bounding box with the largest area; if there are multiple oriented bounding boxes with the largest area, calculate the Euclidean distance between the center point of each oriented bounding box with the largest area and the center point of the bookshelf layer image, and select the oriented bounding box with the smallest Euclidean distance to the center point of the bookshelf layer image as the target bookshelf layer oriented bounding box.

9. A robot, characterized in that, The robot includes a camera, a processor, a YOLO-based book cluster region detection and localization system as described in any one of claims 1 to 8, and an edge computing device; wherein the camera is configured to capture images of the bookshelf shelves; and the edge computing device is used to run the YOLOv8 model in the YOLO-based book cluster region detection and localization system and to perform inference on the bookshelf shelf images.

10. The robot as described in claim 9, characterized in that, The robot is a mobile library visual inventory robot; wherein, when the YOLO-based book group area detection and localization system is run by the library visual inventory robot, it is used to locate the book group area of ​​the target inventory layer from the multi-layer bookshelf layer images captured by the library visual inventory robot.

Images