Book checking method and device based on image recognition, electronic equipment and medium

Abstract

The invention relates to a book checking method and device based on image recognition, electronic equipment and a medium, and relates to the technical field of image processing. The book checking method based on image recognition comprises the steps of controlling a checking robot to perform mobile shooting on corresponding paths in a plurality of preset application scenes so as to determine a plurality of corresponding test images, and determining a target checking path based on the plurality of test images; according to the inventory demand information, controlling the inventory robot to perform mobile shooting on the target inventory path, and receiving an inventory image sent by the inventory robot; on the basis of preset bookshelf layout information, actual angle deviation of corresponding sub-books in the inventory images is determined, a target basic direction is determined according to the actual angle deviation, image correction and splicing are carried out on the inventory images according to the target basic direction, and associated inventory images are determined; and performing picture analysis on the associated inventory image to determine a target data frame, and determining book corresponding information based on the target data frame and the bookshelf layout information.

Description

Technical Field

[0001] The present application relates to the field of image processing technologies, for example, to a method, device, electronic device, and medium for book inventory based on image recognition. Background Art

[0002] With the progress of social economy, people's spiritual needs are gradually increasing, and books are an important part of spiritual products. Books are generally stored in libraries, and the number of books in libraries is very large. In order to ensure reasonable protection of books, it is necessary to conduct regular book inventories. In some libraries, manual book sorting and then inventory taking are relied on, but the number of books is large, the inventory rate is slow, and the effect is also poor.

[0003] In related technologies, generally, bookshelves are photographed at fixed positions, and then image processing is used for inventory taking. However, simple image recognition is affected by many factors, such as environmental brightness and shooting angle. These factors may cause occlusion and misalignment between books on the bookshelf, and distortion of character size, font, color, etc., resulting in an increase in the difficulty of recognizing book images, and thus the overall current inventory efficiency is relatively low. Summary of the Invention

[0004] The present application aims to provide a method, device, electronic device, and medium for book inventory based on image recognition, which can improve the book inventory efficiency.

[0005] According to one aspect of the present application, a method for book inventory based on image recognition is proposed, including: obtaining the inventory requirement information of target books in a library and the road information corresponding to a preset library layout, and controlling a inventory-taking robot to move and take pictures on the paths corresponding to multiple preset application scenarios to determine corresponding multiple test images, and determining a target inventory path based on the multiple test images; controlling the inventory-taking robot to move and take pictures on the target inventory path according to the inventory requirement information, and receiving the inventory pictures sent by the inventory-taking robot; determining the actual angular offset of the corresponding sub-books in the inventory pictures based on the preset bookshelf layout information, and determining a target basic direction according to the actual angular offset, so as to perform image correction and stitching on the inventory pictures according to the target basic direction to determine associated inventory pictures; performing picture analysis on the associated inventory pictures to determine target data frames, and determining book corresponding information based on the target data frames and the bookshelf layout information; where the target data frames correspond to the target books one by one.

[0006] According to some embodiments, based on the preset bookshelf layout information, determine the actual angular offset of the corresponding sub-books in the inventory image, and based on the actual angular offset, determine the target base direction to perform image correction and stitching on the inventory image to determine the associated inventory image, including: extracting the working attributes of the inventory robot from the inventory requirement information; determining the actual angular offset according to the working attributes; determining the offset difference between the inventory images according to the actual angular offset; determining the target base direction based on the preset difference classification information and the offset difference; performing image correction on the inventory image according to the target base direction; determining the association relationship between the inventory images according to the bookshelf layout information; and stitching the corrected inventory images based on the association relationship to determine the associated inventory image.

[0007] According to some embodiments, perform picture analysis on the associated inventory image to determine the target data frame, and based on the target data frame and the bookshelf layout information, determine the book corresponding information, including: inputting the associated inventory image into a preset customized detection model to output the position data of the target data frame; comparing the position data with the bookshelf layout information to determine the book corresponding information.

[0008] According to some embodiments, comparing the position data with the bookshelf layout information to determine the book corresponding information, including: obtaining the encoded scan information sent by the inventory robot and the preset book encoding method; based on the book encoding method, comparing the encoded scan information, the position data and the bookshelf layout information to determine the book corresponding information.

[0009] According to some embodiments, obtain the inventory requirement information of the target books in the library and the road information corresponding to the preset library layout, and control the inventory robot to move and take pictures on the corresponding paths in multiple preset application scenarios to determine the corresponding multiple test images. Based on the multiple test images, determine the target inventory path, including: obtaining the inventory requirement information and the library layout, and determining the area range to be inventoried according to the library layout; obtaining the road information corresponding to the area range, and setting the preliminary inventory path based on the road information; controlling the inventory robot to move and take pictures on the preliminary inventory path in multiple preset application scenarios to determine the corresponding multiple test images; and performing image analysis on the multiple test images to determine the target inventory path.

[0010] According to some embodiments, image analysis is performed on multiple test images to determine a target inventory path, including: performing image analysis on multiple test images to determine preferred shooting attributes and preferred shooting positions corresponding to multiple preset application scenarios; determining, according to the preferred shooting attributes and preferred shooting positions, preferred inventory paths corresponding to the multiple preset application scenarios in a preliminary inventory path; extracting an inventory type from inventory requirement information; and selecting an inventory position from the preferred inventory paths according to the inventory type, so as to generate a target inventory path based on the preferred inventory path and the inventory position.

[0011] According to some embodiments, the above method further includes: extracting book placement information from shelf layout information; and determining whether a book is missing according to the book corresponding information and the book placement information.

[0012] According to one aspect of the present application, a book inventory device based on image recognition is provided, including: A path determination module, configured to obtain inventory requirement information of a target book in a library and road information corresponding to a preset library layout, and control a inventory robot to move and shoot on paths corresponding to multiple preset application scenarios to determine corresponding multiple test images, and determine a target inventory path based on the multiple test images; An inventory image receiving module, configured to control the inventory robot to move and shoot on the target inventory path according to the inventory requirement information, and receive the inventory images sent by the inventory robot; An image correction and stitching module, configured to determine an actual angular offset of a corresponding sub-book in the inventory image based on the preset shelf layout information, and determine a target basic direction according to the actual angular offset, so as to perform image correction and stitching on the inventory image according to the target basic direction to determine an associated inventory image; A book corresponding module, configured to perform picture analysis on the associated inventory image to determine a target data frame, and determine book corresponding information based on the target data frame and the shelf layout information; wherein, the target data frame corresponds to the target book one by one.

[0013] Optionally, the image correction and stitching module is specifically configured to: Extract the working attributes of the inventory robot from the inventory requirement information; Determine the actual angular offset according to the working attributes; Determine the offset difference between the inventory images according to the actual angular offset; Determine the target basic direction based on the preset difference classification information and the offset difference; Perform image correction on the inventory images according to the target basic direction; Determine the association relationship between the inventory images according to the shelf layout information; Stitch the corrected inventory images based on the association relationship to determine the associated inventory images.

[0014] Optionally, the book correspondence module is specifically configured to: Input the associated inventory images into a preset customized detection model and output the position data of the target data frame; Compare the position data with the bookshelf layout information to determine the book correspondence information.

[0015] Optionally, when the book correspondence module compares the position data with the bookshelf layout information to determine the book correspondence information, it is specifically configured to: Obtain the encoded scan information sent by the inventory robot and the preset book encoding method; Based on the book encoding method, compare the encoded scan information, position data, and bookshelf layout information to determine the book correspondence information.

[0016] Optionally, the path determination module is specifically configured to: Obtain the inventory requirement information and the library layout, and determine the area range to be inventoried according to the library layout; Obtain the road information corresponding to the area range and set the preliminary inventory path based on the road information; Under multiple preset application scenarios, control the inventory robot to move and take pictures on the preliminary inventory path to determine the corresponding multiple test images; Perform image analysis on the multiple test images to determine the target inventory path.

[0017] Optionally, when the path determination module performs image analysis on the multiple test images to determine the target inventory path, it is specifically configured to: Perform image analysis on the multiple test images to determine the preferred shooting attributes and preferred shooting positions corresponding to the multiple preset application scenarios; According to the preferred shooting attributes and preferred shooting positions, determine the preferred inventory paths corresponding to the multiple preset application scenarios in the preliminary inventory path; Extract the inventory type from the inventory requirement information; According to the inventory type, select the inventory positions in the preferred inventory paths to generate the target inventory path based on the preferred inventory paths and the inventory positions.

[0018] Optionally, the book inventory device based on image recognition further includes a book missing analysis module for: Extract the book placement information from the bookshelf layout information; Determine whether there are missing books according to the book correspondence information and the book placement information.

[0019] According to one aspect of the present application, an electronic device is provided. The electronic device includes: a processor; a memory storing a computer program, which when executed by the processor, causes the processor to execute the book inventory method based on image recognition as described above.

[0020] According to one aspect of the present application, a non-transitory computer-readable medium is provided, on which readable instructions are stored, which when executed by a processor, cause the processor to execute the book inventory method based on image recognition as described above.

[0021] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application.

[0022] Through the above embodiments provided by the present application, by controlling the inventory robot to move and take pictures on the corresponding paths in multiple preset application scenarios, it can flexibly adapt to the environmental conditions in different areas of the library. This method significantly reduces the image quality problems caused by environmental factors compared with the traditional fixed-position shooting. Determining the target inventory path based on multiple test images can automatically screen out suitable shooting paths, effectively avoiding problems such as book occlusion and misalignment caused by poor shooting angles, and improving the efficiency of image acquisition. Through the preset bookshelf layout information, combined with the actual angle offset in the inventory images, high-precision correction and stitching of the inventory images are achieved, reducing problems such as image distortion and character deformation caused by shooting angle differences, making the processed associated inventory images clearer and more accurate, and greatly improving the accuracy and efficiency of image recognition. Performing picture analysis on the associated inventory images can accurately identify the target data frames (i.e., the image areas corresponding to the target books one by one), and based on this data and the bookshelf layout information, quickly determine the corresponding information of the books, simplifying the manual verification steps in the traditional inventory and significantly improving the inventory efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application; Figure 2 It is a flowchart of a book inventory method based on image recognition provided by an embodiment of the present application; Figure 3 It is a block diagram of a book inventory device based on image recognition provided by an embodiment of the present application; Figure 4 A structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some but not all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0026] In addition, the term "and / or" in this article is only a relational expression describing associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after, unless otherwise specified.

[0027] The embodiments of the present application will be further described in detail below with reference to the accompanying drawings of the specification.

[0028] Figure 1 A schematic diagram of an application scenario according to an exemplary embodiment is shown. There are a large number of bookshelves and books in the library, and inventory robots can be set at fixed positions in the library. The method of the present application can be built on a book inventory server. The book inventory server can control the inventory robot to perform a test run to determine the inventory path under different light and other scenarios. When performing the book inventory task, the target inventory path can be set based on the actual inventory requirement information and sent to the inventory robot, so that the inventory robot can take images during the target inventory on the path, and then perform image analysis to determine the corresponding book information.

[0029] The specific implementation manner can refer to the following embodiments.

[0030] Figure 2 A flowchart of a book inventory method based on image recognition provided by an embodiment of the present application. The method of this embodiment can be applied to a book inventory server. As Figure 2 shown, the method includes: step S20, step S21, step S22, and step S23.

[0031] In step S20, obtain the inventory requirement information of the target book in the library and the road information corresponding to the preset library layout, and control the inventory robot to move and take pictures in multiple preset application scenarios to determine corresponding multiple test images, and determine the target inventory path based on the multiple test images.

[0032] In this application, the target books can be used to represent all the books that need to be inventoried. The inventory demand information may include the brightness at the current moment, the inventory type, the start time of the inventory, the area to be inventoried, and shooting requirements such as the speed and angle of the robot during inventory. The inventory robot can be a robot set in the library for inventory. There can be a camera device on the robot for rotating shooting, or multiple camera devices can be set for shooting in corresponding directions.

[0033] The library contains roads that allow staff and readers to pass through. The location information of these roads in the library and the relative position relationship between the roads can be used as road information, which includes multiple paths. The preset application scenarios in this application can be used to represent inventory scenarios corresponding to different environmental brightness, different inventory times, and inventory types.

[0034] According to the exemplary embodiment, the inventory demand information sent by the staff and the road information corresponding to the preset library layout can be received. The inventory robot is controlled to traverse the corresponding path for shooting to obtain multiple test images. Then, based on the preset path selection conditions, such as the shortest path, or the most time-saving, or the best shooting effect, the multiple test images can be compared to determine at which position the captured images meet the conditions, and then the positions are connected to obtain the target inventory path.

[0035] In step S21, according to the inventory demand information, the inventory robot is controlled to move and shoot on the corresponding target inventory path, and the inventory images sent by the inventory robot are received.

[0036] According to the exemplary embodiment, the inventory demand information can be sent to the inventory robot so that it can correspondingly match the target inventory path and move and shoot on the path. The captured inventory images are sent to the book inventory robot.

[0037] In some implementation manners, the inventory demand information may include the target inventory path.

[0038] In other implementation manners, the book inventory robot can determine the target inventory path based on the inventory demand information, and then send the inventory demand information and the target inventory path to the inventory robot together so that the inventory robot can perform the shooting work.

[0039] In step S22, based on the preset bookshelf layout information, the actual angle offset of the corresponding sub-books in the inventory image is determined, and based on the actual angle offset, the target basic direction is determined to perform image correction and stitching on the inventory image according to the target basic direction to determine the associated inventory image.

[0040] In this application, the bookshelf layout information can be used to represent information such as the position, arrangement, and distance between each bookshelf in the library. The actual angular offset can be the offset angle of the sub-book relative to the preset standard direction, and the target base direction can be used to represent the corrected direction, which is not necessarily in the same proportion as the physical object and may also be skewed.

[0041] A bookshelf image under the current bookshelf layout information can be pre-shot as the standard direction, and then the inventory image can be analyzed to obtain the actual direction of the sub-books. By subtracting the standard direction, the actual angular offset can be obtained. Then, according to the pre-set corresponding relationship, the target base direction corresponding to the actual angular offset can be obtained. Then, the inventory image can be corrected based on the target base direction.

[0042] Due to the different positions of books on the bookshelf, when the inventory robot takes pictures, the shapes and angles of different books in the image are different. Therefore, an image association model can be pre-set, and the bookshelf layout information and the corrected inventory image can be input into the image association model to output the spliced associated inventory image.

[0043] In step S23, the associated inventory image is analyzed to determine the target data frame, and based on the target data frame and the bookshelf layout information, the book corresponding information is determined; where the target data frame corresponds to the target book one by one.

[0044] In this application, the target data frame can be a data frame that frames different books, which can include the position information of the data frame and the image containing the data frame. The book corresponding relationship can be used to represent the relative position relationship of different books on the same bookshelf, the relative position relationship of books on different bookshelves, and the relative position relationship between bookshelves.

[0045] According to the exemplary embodiment, a preset YOLO model or SSD (Single Shot MultiBoxDetector, a single-stage object detection algorithm) model can be used to output the target data frame. Based on the arrangement relationship between the data frames corresponding to each book in the target data frame and the actual book arrangement relationship in the bookshelf layout information, the book corresponding relationship can be obtained. The book corresponding relationship can clarify whether the correct book is placed at each book placement position on the bookshelf.

[0046] In this application, by controlling the inventory robot to move and take pictures along paths corresponding to multiple preset application scenarios, it can flexibly adapt to the environmental conditions in different areas of the library. Compared with traditional fixed-position shooting, this method greatly reduces image quality problems caused by environmental factors. Determining the target inventory path based on multiple test images can automatically select appropriate shooting paths, effectively avoiding problems such as book occlusion and misalignment caused by poor shooting angles, and improving the efficiency of image acquisition. Through the preset bookshelf layout information and combined with the actual angle offset in the inventory images, high-precision correction and stitching of the inventory images are achieved, reducing problems such as image distortion and character deformation caused by shooting angle differences, making the processed associated inventory images clearer and more accurate, and greatly improving the accuracy and efficiency of image recognition. Performing picture analysis on the associated inventory images can accurately identify the target data frames (i.e., the image areas corresponding to the target books one by one), and based on this data and the bookshelf layout information, quickly determine the corresponding information of the books, simplifying the manual verification steps in traditional inventory, and significantly improving the inventory efficiency and accuracy.

[0047] According to some embodiments, the working attributes of the inventory robot can be extracted from the inventory requirement information; according to the working attributes, the offset difference between the inventory images can be determined; based on the preset difference classification information and the offset difference, the actual angle offset can be determined; according to the actual angle offset, the target basic direction can be determined; according to the target basic direction, image correction is performed on the inventory images; according to the bookshelf layout information, the association relationship between the inventory images can be determined; based on the association relationship, the corrected inventory images are stitched together to determine the associated inventory images.

[0048] In this application, the working attributes may include the moving speed of the inventory robot, the camera pan-tilt angle, the image upload method, etc. Among them, the image upload method includes uploading in chronological order of timestamps or uploading as a whole according to the camera numbers. Sub-books can be used to represent the books contained in each inventory image. The offset difference can be used to represent the difference between the actual angle offsets of the inventory images. The preset difference classification information can be used to characterize the corresponding relationship between different offset differences, different shooting situations and the target basic direction, such as the image taken is clearer, or the image is more upright, or there are more sub-books in the image, etc. Therefore, the corresponding target basic direction can be selected according to the requirements. In the case of not setting requirements, it can be defaulted to the target basic direction corresponding to the offset difference with a clearer image.

[0049] According to an exemplary embodiment, first extract the working attributes of the inventory robot from the inventory requirement information, and then, based on the images captured by the inventory robot at different positions, combine with the pan-tilt to adjust the angle and calculate the offset angle of each inventory image relative to the preset standard direction. In some implementation manners, a feature detection algorithm in image processing technology can be used to detect key points in the image, and the relative rotation angle between the images can be calculated by matching the key points.

[0050] In some implementation manners, statistical analysis can also be performed on all actual angle offsets based on the difference classification information and the offset difference, and the offset angle that meets the requirement conditions or is closest to the preset standard direction can be found as the target base direction. The requirement conditions can include the most occurrences. Then, a preset image transformation function can be used to perform rotation and translation transformation on each inventory image to correct it to the target base direction.

[0051] The number of layers of the bookshelf, the number of grids on each layer, and the relative position relationship between adjacent grids can be extracted from the bookshelf layout information, and then the position relationship between the sub-books in the inventory image can be determined as the association relationship of the inventory image. According to the relative positions and orders in the association relationship, the corrected inventory images are spliced to finally generate an associated inventory image.

[0052] In some implementation manners, a weighted average fusion algorithm can be used to process the overlapping parts between the images during splicing.

[0053] In this application, by extracting the working attributes of the inventory robot from the inventory requirement information, the actual angle offset of the corresponding sub-books in the inventory image can be calculated more accurately. This calculation method based on the robot's working attributes is more flexible and accurate than the traditional method and can adapt to the shooting requirements in different scenarios. According to the calculated actual angle offset, the offset difference between the inventory images is further determined. Through the preset difference classification information, these offset differences can be intelligently classified, and based on this, the target base direction is determined, improving the accuracy of image correction and reducing the need for manual intervention. According to the determined target base direction, image correction is performed on the inventory image, which can reduce the distortion and deformation that are not suitable for image analysis. The association relationship between the inventory images is determined according to the bookshelf layout information, and the corrected images are spliced based on these association relationships, improving the accuracy and efficiency of image splicing and making the generated associated inventory image more complete and clear.

[0054] According to some embodiments, the associated inventory image can be input into a preset customized detection model to output the position data of the target data frame; the position data is compared with the bookshelf layout information to determine the book corresponding information.

[0055] In this application, a target detection model for book inventory can be established in advance. In some implementation manners, the YOLO model can be used as a basis, and its loss function can be improved to avoid duplication and coverage of the output data frames, so as to obtain a customized detection model.

[0056] In some implementation manners, the associated inventory image is input into the customized detection model, and the position data of the target data frame is output. Generally, the image content framed in the target data frame is the spine part of the book. Generally, the position data includes the coordinate data of the four vertices of the data frame. The shelf layout information includes the placement positions of different sub-books. Based on this, a comparison is made with the position data to determine the correspondence between the target data frame and the sub-books, that is, which actual book the book in any target data frame corresponds to.

[0057] This application can automatically identify and locate the target data frame in the associated inventory image by using the customized detection model, which greatly reduces the workload of manual annotation and recognition and improves the accuracy of the inventory. At the same time, through automated processing, the inventory cycle is significantly shortened and the work efficiency is improved. By comparing the detected position data of the target data frame with the shelf layout information, each book can be accurately corresponded to its physical position on the shelf. This precise matching not only helps to quickly locate the books, but also provides a reliable basis for subsequent book management and search.

[0058] According to some embodiments, the encoded scan information sent by the inventory robot and a preset book encoding method can be obtained; based on the book encoding method, the encoded scan information, the position data, and the shelf layout information are compared to determine the book correspondence information.

[0059] In this application, the content in the image generally captured by the inventory robot is the spine, and a pre-set label can be pasted on the spine. The label corresponds to the book one by one. The label forms of different types of books are different, and the label forms of different books of the same type are the same, but there is a unique corresponding code. In some implementation manners, the book encoding method can be a barcode or a string of non-repeating characters.

[0060] According to the exemplary embodiment, an image acquisition device and a scanning device are provided on the inventory robot. If the book encoding method is a character, then the scanning device is also an image acquisition device. First, the encoded scan information sent by the inventory robot and the preset book encoding method are received, and then based on the book encoding method, the book corresponding to each code in the encoded scan information is determined correspondingly. The position data and the shelf layout information can be compared to obtain the book corresponding to the position. Based on this, the correspondence between the code - the identified position - the actual position can be further determined, that is, the book correspondence information.

[0061] This application combines location data and encoded scanning information for comparison, which can double-confirm the identity and location of books, greatly reducing the problem of inaccurate inventory due to incorrect or missing single information. This dual-verification mechanism significantly improves the accuracy of book inventory. Using the preset book coding method and bookshelf layout information, books can be automatically identified and matched without manual checking one by one, realizing the intelligent management of books. This not only improves management efficiency but also reduces labor costs.

[0062] According to some embodiments, inventory requirement information and library layout can be obtained, and based on the library layout, the area range to be inventoried can be determined; the road information corresponding to the area range can be obtained, and a preliminary inventory path can be set based on the road information; in multiple preset application scenarios, the inventory robot can be controlled to move and take pictures on the preliminary inventory path to determine corresponding multiple test images; the multiple test images can be analyzed for image analysis to determine the target inventory path.

[0063] In this application, when the library is established, the corresponding library layout can be set and stored. The library layout can include the positions of bookshelves in the library, the setting of passable roads, and the working areas of staff, etc. There may be some areas where bookshelves and roads are set, but they are only suitable for the stay of staff and the storage and charging of inventory robots. Therefore, there are some areas that do not need to be inventoried, and the remaining areas can be used as the area range to be inventoried.

[0064] In some implementation manners, the area range to be inventoried and the corresponding road information can be obtained from the library layout, and the roads that can be associated with each bookshelf within the area range to be inventoried can be linked to obtain a preliminary inventory path with non-repeated roads.

[0065] In this application, in different preset application scenarios, the inventory robot is controlled to move and take pictures on the preliminary inventory path to obtain multiple test images. The rule for image selection can be preset, and the multiple test images can be analyzed for image analysis to obtain the target inventory path.

[0066] This application determines the area to be inventoried quickly by presetting the library layout, and sets the initial inventory path based on the road information, greatly reducing the time and effort of manually planning the inventory path. At the same time, by using the inventory robot to move and take pictures on the initial path and combining image analysis technology, the position and status of books can be accurately identified, improving the accuracy and efficiency of the inventory. The inventory can be carried out in multiple preset application scenarios, such as normal working hours, closing hours, etc., ensuring the flexibility and adaptability of the inventory work. Regardless of how the traffic flow and light conditions in the library change, the shooting parameters and path planning strategy of the inventory robot can be adjusted to ensure the smooth progress of the inventory work. By performing image analysis on multiple test images, the deficiencies and shooting blind spots in the initial inventory path can be found, and then the path can be optimized and adjusted. This can not only reduce the moving distance and time of the inventory robot, but also improve the shooting quality and image recognition rate, ensuring the comprehensiveness and accuracy of the inventory results.

[0067] According to some embodiments, image analysis can be performed on multiple test images to determine the preferred shooting attributes and preferred shooting positions corresponding to multiple preset application scenarios; according to the preferred shooting attributes and preferred shooting positions, determine the preferred inventory paths corresponding to each of the multiple preset application scenarios in the initial inventory path; extract the inventory type from the inventory requirement information; according to the inventory type, select the inventory positions in the preferred inventory paths, so as to generate the target inventory path based on the preferred inventory paths and the inventory positions.

[0068] In this application, image analysis can be first performed on the test images to determine the image content therein, and determine what shooting attributes and shooting positions can capture the complete and clear book spines of books under different preset application scenarios, and use the shooting attributes and shooting positions as the preferred shooting attributes and preferred shooting positions. The shooting attributes can include exposure, focal length, pan-tilt angle, etc.

[0069] In the initial inventory path, locate the preferred shooting positions, and then select and connect the shortest paths involved in these positions as the preferred inventory paths. In some implementation manners, a preferred inventory path determination model can be established, and the preferred shooting attributes, preferred shooting positions, and initial inventory path are input into the preferred inventory path determination model to output the preferred inventory path. The preferred inventory path in this application is not necessarily the shortest, but the way with the shortest path in the clearest manner.

[0070] The inventory type can include random inventory, comprehensive inventory, and periodic inventory. Different inventory types may require different inventory positions, that is, not all paths in the preferred inventory paths need to be visited. The inventory positions can be found according to the inventory type, the inventory positions are selected in the preferred inventory paths, and then the corresponding paths are selected to connect these inventory positions to obtain the target inventory path.

[0071] Through the analysis of multiple test images, this application can accurately identify the best shooting attributes and shooting positions in different preset application scenarios. This ensures that during the actual inventory process, the inventory robot can capture high-quality and high-definition book images, laying a solid foundation for subsequent image recognition and inventory work. Based on the preferred shooting attributes and positions, the optimal inventory paths can be further selected from the preliminary inventory paths. These paths not only cover all the book positions that need to be inventoried, but also avoid unnecessary repetitions and redundancies, thus greatly improving the inventory efficiency.

[0072] According to some embodiments, the book placement information can also be extracted from the bookshelf layout information; based on the book corresponding information and the book placement information, it is determined whether there are missing books.

[0073] In some implementation manners, the book placement information is extracted from the bookshelf layout information, that is, the correct placement position of the book at the current moment. The position in the book corresponding relationship is compared with the correct placement position to determine whether there is a book placed at the correct position, and further determine whether there are missing books.

[0074] By extracting the book placement information from the bookshelf layout information, this application can accurately master the specific position of each book on the bookshelf. This includes the number, layer, and specific grid position of the bookshelf, etc., ensuring the accuracy of the book position. The book corresponding information is efficiently compared with the book placement information. It can quickly identify which books should be on the bookshelf and which books are actually not in the expected position. Based on the above comparison results, the missing situation of the books can be discovered in a timely manner. Once it is detected that the corresponding information of a certain book exists, but the corresponding position cannot be found in the bookshelf placement information, it can be determined that the book is missing. This instant feedback mechanism helps the library administrator take actions quickly, such as searching for books and updating inventory information.

[0075] The device embodiments of this application are described below, which can be used to execute the method embodiments of this application. For the details not disclosed in the device embodiments of this application, reference can be made to the method embodiments of this application.

[0076] Figure 3 It is a block diagram of a book inventory device based on image recognition provided for the embodiments of this application. As Figure 3 shown, the book inventory device 300 based on image recognition includes a path determination module 301, an inventory image receiving module 302, an image correction and stitching module 303, and a book corresponding module 304.

[0077] A path determination module 301, configured to obtain the inventory requirement information of a target book in a library and the road information corresponding to a preset library layout, and control a inventory robot to move and take pictures on paths corresponding to multiple preset application scenarios, so as to determine corresponding multiple test images, and determine a target inventory path based on the multiple test images; An inventory image receiving module 302, configured to control the inventory robot to move and take pictures on the target inventory path according to the inventory requirement information, and receive the inventory images sent by the inventory robot; An image correction and stitching module 303, configured to determine the actual angular offset of a corresponding sub-book in the determined inventory image based on the preset bookshelf layout information, and determine a target basic direction according to the actual angular offset, so as to perform image correction and stitching on the inventory image according to the target basic direction to determine an associated inventory image; A book correspondence module 304, configured to perform picture analysis on the associated inventory image to determine a target data frame, and determine book correspondence information based on the target data frame and the bookshelf layout information; wherein, the target data frame corresponds to the target book one by one.

[0078] Optionally, the image correction and stitching module 303 is specifically configured to: Extract the working attribute of the inventory robot from the inventory requirement information; Determine the actual angular offset according to the working attribute; Determine the offset difference between the inventory images according to the actual angular offset; Determine the target basic direction based on the preset difference classification information and the offset difference; Perform image correction on the inventory images according to the target basic direction; Determine the association relationship between the inventory images according to the bookshelf layout information; Stitch the corrected inventory images based on the association relationship to determine an associated inventory image.

[0079] Optionally, the book correspondence module 304 is specifically configured to: Input the associated inventory image into a preset customized detection model, and output the position data of the target data frame; Compare the position data with the bookshelf layout information to determine the book correspondence information.

[0080] Optionally, when the book correspondence module 304 compares the position data with the bookshelf layout information to determine the book correspondence information, it is specifically configured to: Obtain the encoded scan information sent by the inventory robot and the preset book encoding method; Compare the encoded scan information, the position data and the bookshelf layout information based on the book encoding method to determine the book correspondence information.

[0081] Optionally, the path determination module 301 is specifically configured to: Obtain the inventory requirement information and the library layout, and determine the area range to be inventoried according to the library layout; Obtain the road information corresponding to the area range, and set a preliminary inventory path based on the road information; Under multiple preset application scenarios, control the inventory robot to move and shoot on the preliminary inventory path to determine multiple corresponding test images; Perform image analysis on the multiple test images to determine the target inventory path.

[0082] Optionally, when the path determination module 301 performs image analysis on multiple test images to determine the target inventory path, it is specifically configured to: Perform image analysis on the multiple test images to determine the preferred shooting attributes and preferred shooting positions corresponding to multiple preset application scenarios; According to the preferred shooting attributes and preferred shooting positions, determine the preferred inventory paths corresponding to multiple preset application scenarios respectively in the preliminary inventory path; Extract the inventory type from the inventory requirement information; According to the inventory type, select the inventory positions in the preferred inventory paths to generate the target inventory path based on the preferred inventory paths and the inventory positions.

[0083] Optionally, the book inventory device 300 based on image recognition further includes a book missing analysis module 305, which is used for: Extract the book placement information from the bookshelf layout information; Determine whether there are missing books according to the book corresponding information and the book placement information.

[0084] The device executes functions similar to the methods provided above. For other functions, refer to the previous descriptions and will not be elaborated here.

[0085] Figure 4 It is a schematic structural diagram of the electronic device provided in the embodiment of the present application. As Figure 4 shown, the electronic device 400 in this embodiment may include: a memory 401 and a processor 402.

[0086] A computer program is stored on the memory 401. When the computer program is executed by the processor 402, the foregoing processor 402 executes the method in the above embodiment.

[0087] Among them, the processor 402 and the memory 401 are connected, such as through a bus.

[0088] Optionally, the electronic device 400 may further include a transceiver. It should be noted that in practical applications, the number of transceivers is not limited to one, and the structure of the electronic device 400 does not constitute a limitation on the embodiments of the present application.

[0089] The processor 402 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of the present application. The processor 402 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0090] The bus may include a path for transmitting information between the above components. The bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0091] The memory 401 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it may also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0092] The memory 401 is used to store the application program code for executing the solution of the present application, and is controlled by the processor 402 to execute. The processor 402 is used to execute the application program code stored in the memory 401 to implement the content shown in the foregoing method embodiments.

[0093] Among them, the electronic device includes but is not limited to: mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. It can also be a server, etc. Figure 4 The illustrated electronic device is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0094] The electronic device of this embodiment can be used to execute the method of any of the foregoing embodiments, and its implementation principle and technical effects are similar, so they will not be elaborated here.

[0095] The present application also provides a computer-readable storage medium storing a computer program that can be loaded and executed by a processor to perform the method in the foregoing embodiments.

[0096] Those of ordinary skill in the art can understand that all or part of the steps for implementing the foregoing method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the foregoing method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

Claims

1. A book inventory method based on image recognition, characterized in that: include: Obtaining inventory demand information of target books in the library and road information corresponding to the preset library layout, and controlling the inventory robot to move and shoot on the corresponding paths under multiple preset application scenarios to determine the corresponding multiple test images, and determining the target inventory path based on the multiple test images; According to the inventory demand information, control the inventory robot to move and shoot on the target inventory path, and receive the inventory image sent by the inventory robot; Based on the preset bookshelf layout information, the actual angle offset of the corresponding sub-book in the inventory image is determined, and according to the actual angle offset, the target basic direction is determined, so as to perform image correction and splicing on the inventory image according to the target basic direction to determine the associated inventory image; Perform image analysis on the associated inventory image to determine a target data frame, and determine book corresponding information based on the target data frame and the bookshelf layout information; wherein the target data frame corresponds to the target book one by one.

2. The method according to claim 1, characterized in that The method of determining the actual angle offset of the sub-books corresponding to the inventory image based on the preset bookshelf layout information, and determining the target basic direction according to the actual angle offset, so as to perform image correction and splicing on the inventory image and determine the associated inventory image, includes: Extracting the working attributes of the inventory counting robot from the inventory counting requirement information; Determining the actual angle offset according to the working attribute; Determining the offset difference between the inventory images according to the actual angle offset; Determining a target base direction based on preset difference classification information and the offset difference; Performing image correction on the inventory image according to the target base direction; Determining the association relationship between the inventory images according to the bookshelf layout information; The corrected inventory images are spliced ​​based on the association relationship to determine the associated inventory images.

3. The method according to claim 1, characterized in that The performing image analysis on the associated inventory image to determine the target data frame, and determining the corresponding information of the book based on the target data frame and the bookshelf layout information, includes: Inputting the associated inventory image into a preset customized detection model, and outputting the position data of the target data frame; The location data is compared with the bookshelf layout information to determine the corresponding information of the book.

4. The method according to claim 3, characterized in that The comparing the location data with the bookshelf layout information to determine the book corresponding information includes: Obtaining the coding scanning information and the preset book coding method sent by the inventory counting robot; Based on the book encoding method, the encoding scan information, the location data and the bookshelf layout information are compared to determine the corresponding information of the book.

5. The method according to claim 1, characterized in that The method of obtaining inventory demand information of target books in the library and road information corresponding to a preset library layout, and controlling the inventory robot to move and shoot on corresponding paths under multiple preset application scenarios to determine a plurality of corresponding test images, and determining a target inventory path based on the plurality of test images, includes: Obtaining the inventory demand information and the library layout, and determining the area to be inventoried according to the library layout; Obtaining road information corresponding to the area range, and setting a preliminary inventory route based on the road information; In a plurality of preset application scenarios, controlling the inventory counting robot to move and shoot on the preliminary inventory counting path to determine a plurality of corresponding test images; Image analysis is performed on the multiple test images to determine the target inventory path.

6. The method according to claim 5, characterized in that The performing image analysis on the plurality of test images to determine the target inventory path includes: Performing image analysis on the multiple test images to determine preferred shooting attributes and preferred shooting positions corresponding to the multiple preset application scenarios; Determining, in the preliminary inventory path, preferred inventory paths corresponding to each of the plurality of preset application scenarios according to the preferred shooting attributes and the preferred shooting positions; Extracting an inventory type from the inventory requirement information; According to the inventory type, an inventory location is selected in the preferred inventory path to generate the target inventory path based on the preferred inventory path and the inventory location.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: Extracting book placement information from the bookshelf layout information; Whether a book is missing is determined according to the book corresponding information and the book placement information.

8. A book inventory device based on image recognition, characterized in that: include: A path determination module is used to obtain the inventory demand information of the target books in the library and the road information corresponding to the preset library layout, and control the inventory robot to move and shoot on the corresponding paths under multiple preset application scenarios to determine the corresponding multiple test images, and determine the target inventory path based on the multiple test images; An inventory image receiving module, used to control the inventory robot to move and shoot on the target inventory path according to the inventory requirement information, and receive the inventory image sent by the inventory robot; An image correction and stitching module is used to determine the actual angle offset of the corresponding sub-books in the inventory image based on the preset bookshelf layout information, and determine the target basic direction according to the actual angle offset, so as to perform image correction and stitching on the inventory image according to the target basic direction to determine the associated inventory image; The book corresponding module is used to perform image analysis on the associated inventory image to determine the target data frame, and determine the book corresponding information based on the target data frame and the bookshelf layout information; wherein the target data frame corresponds to the target book one by one.

9. An electronic device, characterized in that: include: processor; A memory storing a computer program, which, when executed by the processor, enables the processor to execute the book inventory method based on image recognition as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the instructions are executed by a processor, the processor executes the book inventory method based on image recognition as described in any one of claims 1-7.

Images