Image recognition-based library checking method, device and equipment, medium and program product

Abstract

The invention provides a library checking method based on image recognition, which can be applied to the technical field of financial science and technology, and particularly relates to the technical field of image recognition. The method is applied to an unmanned aerial vehicle end, and comprises the following steps: obtaining label images of articles in a warehouse area; image recognition processing is executed on the label image through a recognition network, a character recognition result of the label image is obtained, and the recognition network is obtained through calculation path simplification processing and quantization error simulation training processing based on contribution degree evaluation; and comparing the character recognition result with a record of an inventory database in real time, and outputting a database checking result. The invention further provides a library checking device and equipment based on image recognition, a storage medium and a program product.

Description

Technical Field

[0001] The present application relates to the field of financial technology, specifically to the field of image recognition technology, and more specifically to a database search method, device, equipment, medium and program product based on image recognition. Background Art

[0002] With the development of the banking service industry, the efficiency of traditional vault inspections has gradually fallen behind the existing banking industry's work efficiency in the digitalization process. Traditional vault inspections rely on manual counting, that is, manual entry into the vault to check the inventory labels of each valuable item in turn, or by installing fixed cameras on the shelves for monitoring.

[0003] However, there are some issues with manual counting and camera monitoring. One is that they are extremely inefficient. Manual counting takes a long time, and counting personnel are more likely to become fatigued while working in the warehouse. Furthermore, the naked eye is easily disturbed by the combined effects of ambient light and blurred labels, which can easily lead to counting errors. Using fixed cameras for identification and counting is susceptible to blind spots, making it impossible to effectively identify stacked or irregularly placed items, resulting in a high error rate. Summary of the Invention

[0004] In view of the above problems, the present application provides a library search method, apparatus, device, medium and program product based on image recognition.

[0005] According to the first aspect of the present application, a library search method based on image recognition is provided, which is applied to a drone end, and the method includes: obtaining a label image of an item in the warehouse area; performing image recognition processing on the label image through a recognition network to obtain a character recognition result of the label image, wherein the recognition network is obtained through a calculation path simplification processing based on contribution evaluation and a quantization error simulation training processing; outputting a library search result based on a real-time comparison of the character recognition result with the records in the inventory database.

[0006] According to an embodiment of the present application, the recognition network includes a feature extraction channel, a sequence extraction channel and an attention channel. The recognition network-based image recognition processing is performed on the label image to obtain the character recognition result of the label image, including: performing feature extraction processing on the label image through the feature extraction channel to obtain a spatial feature map; performing temporal feature extraction processing on the spatial feature map through the sequence extraction channel to obtain context-aware sequence features; performing attention weighted processing on the sequence features through the attention channel to output the character recognition result.

[0007] According to an embodiment of the present application, the method for simplifying computational paths based on contribution evaluation includes: respectively calculating the contribution evaluation value of each computational path in the initial identification network; sorting each of the computational paths according to the contribution evaluation value to obtain a computational path sequence set; and, based on the computational path sequence set, removing computational paths in the initial identification network whose contribution evaluation value is lower than a predetermined threshold to obtain a simplified identification network.

[0008] According to an embodiment of the present application, the calculation paths with contribution evaluation values ​​lower than a predetermined threshold in the initial recognition network are removed according to the calculation path sequence set to obtain a simplified recognition network, including: removing feature dimension-raising paths with contribution evaluation values ​​lower than a predetermined threshold in the extended transformation layer of the multi-layer transformation structure of the feature extraction channel according to the calculation path sequence set; and / or removing temporal state transfer paths with contribution evaluation values ​​lower than a predetermined threshold in the bidirectional state transfer unit of the sequence extraction channel according to the calculation path sequence set; and / or removing weight-associated transfer paths with contribution evaluation values ​​lower than a predetermined threshold in the weight mapping unit of the attention channel according to the calculation path sequence set.

[0009] According to an embodiment of the present application, obtaining the label image of the items in the warehouse area includes: collecting the original optical data of the label of the items in the warehouse area; performing numerical domain conversion processing of dynamic range compression on the original optical data to generate an image numerical sequence for dimensionality reduction storage; and obtaining the label image based on the image numerical sequence.

[0010] According to an embodiment of the present application, the numerical domain conversion processing of performing dynamic range compression on the original optical data to generate an image numerical sequence for dimensionality reduction storage includes: determining the numerical dynamic range of the original optical data based on the original optical data; determining the compression mapping boundary of the original optical data based on the numerical dynamic range; and performing dynamic range compression mapping on the original optical data based on the compression mapping boundary to obtain the image numerical sequence.

[0011] According to an embodiment of the present application, the method for quantization error simulation training processing includes: inserting an error simulation node in the forward calculation path of the simplified recognition network and establishing a gradient direct transfer path in the reverse calculation path to construct a simulation recognition network, wherein the error simulation node is used to perform a forward distortion injection operation in the training of the simulation recognition network; using a sample label image to train the simulation recognition network to generate a standard recognition network; removing the error simulation node and the gradient direct transfer path in the standard recognition network to obtain the recognition network after the quantization error simulation training processing.

[0012] According to an embodiment of the present application, the error simulation node is inserted into the forward calculation path of the recognition network after the simplified processing, including: inserting a first error simulation node at the feature map generation end of each convolution transformation layer in the feature extraction channel to perform discretization distortion injection on the spatial feature map data; and / or inserting a second error simulation node at the weight parameter calculation end of the sequence extraction channel to perform fixed-point truncation distortion injection on the timing modeling parameters; and / or inserting a third error simulation node at the correlation calculation output end of the attention channel to perform nonlinear distribution adapted quantization distortion injection on the attention weight distribution data.

[0013] According to an embodiment of the present application, establishing a gradient direct transfer path in the reverse calculation path of the simplified recognition network includes: obtaining an upstream gradient data stream of the error simulation node in response to the reverse calculation path passing through the error simulation node; and directly transferring the upstream gradient data stream to a downstream calculation node connected to the output end of the error simulation node to skip the calculation process inside the error simulation node.

[0014] According to an embodiment of the present application, the collection of the original optical data of the labels of the items in the warehouse area includes: obtaining multiple original images shot from multiple angles at the same shooting position of the items in the warehouse area, and extracting local feature matching points of each of the original images; aligning the multiple original images according to the local feature matching points of each of the original images; performing pixel weighted fusion processing on the aligned multiple original images to generate the original optical data of the labels of the items in the warehouse area.

[0015] According to an embodiment of the present application, the method for multi-angle shooting at the same shooting position includes: setting multiple horizontal shooting rings around the same shooting position of the items in the warehouse area, wherein the pitch angle interval between adjacent horizontal shooting rings relative to the same shooting position in the vertical direction is a first predetermined angle, each of the horizontal shooting rings includes a plurality of spaced-apart shooting positions, and the azimuth angle interval between two adjacent shooting positions on each horizontal shooting ring relative to the same shooting position in the horizontal direction is a second predetermined angle.

[0016] According to an embodiment of the present application, the method for determining the shooting position includes: collecting an infrared scanning image sequence of the vertical height of the stacking area of ​​the items in the warehouse area; calculating the distribution of the thermodynamic gradient change rate of adjacent planes in the infrared scanning image sequence based on the infrared scanning image sequence; in response to the thermodynamic gradient change rate exceeding a critical threshold, marking the corresponding spatial coordinates where the thermodynamic gradient change rate exceeds the critical threshold as the stacking boundary position; based on the stacking boundary position, determining the three-dimensional spatial coordinates of the shooting position of the items in the warehouse area.

[0017] According to an embodiment of the present application, the method for multi-angle shooting at the same shooting position further includes: in response to the determination of the stacking boundary position, setting the surrounding radius of the horizontal shooting ring to a preset distance, and performing multi-angle image acquisition within the preset distance range.

[0018] The second aspect of the present application provides a library query device based on image recognition, which is applied to a drone end. The device includes: a first acquisition module, configured to execute a natural language request received from a user through a digital human interface; a first processing module, configured to perform image recognition processing on the label image through a recognition network to obtain a character recognition result of the label image, wherein the recognition network is obtained through a calculation path simplification processing based on contribution evaluation and a quantization error simulation training processing; a second processing module, configured to perform real-time comparison between the character recognition result and the records in the inventory database, and output the library query result.

[0019] The third aspect of the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0020] The fourth aspect of the present application further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the computer program or instructions are executed by a processor.

[0021] The fifth aspect of the present application further provides a computer program product, comprising a computer program or instructions, which implement the steps of the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0023] Figure 1A diagram schematically illustrates an application scenario of a library search method, apparatus, device, medium, and program product based on image recognition according to an embodiment of the present application;

[0024] Figure 2 The following schematically shows a flow chart of a library query method based on image recognition for a drone according to an embodiment of the present application;

[0025] Figure 3 The flowchart of step S200 of the image recognition-based library query method for a drone terminal according to an embodiment of the present application is schematically shown;

[0026] Figure 4 A flowchart schematically illustrates a method for simplifying calculation paths and processing an initial identification network based on contribution evaluation according to an embodiment of the present application;

[0027] Figure 5 Schematically shows a flowchart of step S203 of the method for simplifying the calculation path based on contribution evaluation to initially identify the network according to an embodiment of the present application;

[0028] Figure 6 The flowchart of step S100 of the image recognition-based library query method for a drone according to an embodiment of the present application is schematically shown;

[0029] Figure 7 The flowchart of step S120 of the image recognition-based library query method for a drone terminal according to an embodiment of the present application is schematically shown;

[0030] Figure 8 A flowchart schematically illustrates a method for a recognition network after simplified quantization error simulation training processing according to an embodiment of the present application;

[0031] Figure 9 A first supplementary flow chart schematically illustrates step S204 of the method for identifying a network after simplified quantization error simulation training processing according to an embodiment of the present application;

[0032] Figure 10 A second supplementary flow chart schematically illustrates step S204 of the method for identifying a network after simplified quantization error simulation training processing according to an embodiment of the present application;

[0033] Figure 11 The flowchart of step S110 of the image recognition-based library query method for a drone terminal according to an embodiment of the present application is schematically shown;

[0034] Figure 12 A flowchart of a method for multi-angle shooting at the same shooting position according to an embodiment of the present application is schematically shown;

[0035] Figure 13 The flowchart of the method for determining the shooting position according to the embodiment of the present application is schematically shown;

[0036] Figure 14 The following schematically shows a structural block diagram of an image recognition-based library query device for a drone according to an embodiment of the present application;

[0037] Figure 15 A block diagram of an electronic device suitable for implementing a library search method based on image recognition according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0038] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present application.

[0039] The terms used herein are only for describing specific embodiments and are not intended to limit this application. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0040] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0041] When expressions such as “at least one of A, B, and C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art. For example, “a system having at least one of A, B, and C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc. When expressions such as “at least one of A, B, or C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art. For example, “a system having at least one of A, B, or C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.

[0042] The image recognition-based database query method, device, equipment, medium and program product of the present application can be used in the technical field of business scheduling in the financial field, and can also be used in any field other than the financial field. The application field of the image recognition-based database query method, device, equipment, medium and program product of the present application is not limited.

[0043] In the technical solution of this application, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, application and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0044] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided in the embodiments of the present application all provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0045] Traditional vault checks rely on manual counting or fixed camera surveillance, which is inefficient and prone to image obstruction due to stacking. Technicians in this field have attempted to use drones to cruise around the warehouse and conduct checks. However, drone-based checks involve two major issues. Limited by the computing power of the drone's onboard chip, simultaneous recognition during the recording process requires a large computational load, requiring the drone to remain stationary for one to two minutes to completely identify a label on a shelf. The drone's range is insufficient to support the needs of checking large warehouse areas. If a drone captures images and transmits them back to a server for recognition, this is also known as asynchronous recognition and checking. The drone flies, and the server performs its own calculations. However, due to the high recognition error rate for complex stacked items and highly reflective surfaces, server calculation errors often require the drone to be instructed to return to the previous shooting point for re-recording, resulting in still low efficiency. Furthermore, unclear images from the drone during flight are common, requiring the drone to be redeployed for a second shot, significantly impacting inventory checking efficiency.

[0046] The embodiments of the present application provide a database search method based on image recognition.

[0047] Figure 1 The application scenario diagram of the image recognition-based library search method, device, equipment, medium and program product according to the embodiment of the present application is schematically shown.

[0048] like Figure 1 As shown, the network 104 is used to provide a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the edge server 105. The network 104 can include various connection types, such as wired, wireless communication links or fiber optic cables.

[0049] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with an edge server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as financial service applications, web browser applications, email clients, social platform software, etc.

[0050] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to onboard computers, tablet computers, laptop computers, and desktop computers.

[0051] The edge server 105 can be a server that provides various services, such as a background management server (for example only) that supports information browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server can analyze and process received data such as user requests, and feed back processing results (such as web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0052] It should be noted that, in the first aspect, the image recognition-based library query method provided in the embodiments of the present application can be executed by the edge server 105, and in the second aspect, the image recognition-based library query method provided in the embodiments of the present application can be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, in the third aspect, the image recognition-based library query apparatus provided in the embodiments of the present application can be set in the edge server 105, and in the fourth aspect, the image recognition-based library query apparatus provided in the embodiments of the present application can be set in the first terminal device 101, the second terminal device 102, or the third terminal device 103. The image recognition-based library query method provided in the embodiments of the present application can also be executed by a server or server cluster that is different from the edge server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the edge server 105. Accordingly, the image recognition-based library query device provided in the embodiment of the present application can also be set in a server or server cluster that is different from the edge server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the edge server 105. In this embodiment, the drone end is equipped with the edge server 105, and the image recognition-based library query method of this embodiment is executed by the edge server 105. In other embodiments, the image recognition-based library query method of this embodiment can also be executed by other terminal devices equipped on the drone end, and there is no limitation to this.

[0053] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0054] The following will be based on Figure 1 The scene described by Figures 2 to 13 The image recognition-based database search method of the application embodiment is described in detail.

[0055] Figure 2 The figure schematically shows a flowchart of image recognition-based library query for a drone according to an embodiment of the present application.

[0056] like Figure 2 As shown, the image recognition-based database search method of this embodiment includes operations S100 to S300.

[0057] In operation S100, label images of items in the warehouse are obtained. In some embodiments, dynamic image acquisition is performed on items in the warehouse by a drone, and the raw optical data is obtained and then fused to form a label image. In other embodiments, the shooting position can also be determined by a spatial coordinate positioning strategy during the dynamic image acquisition process, and the integrity of the label image can be improved based on a multi-angle surround acquisition mechanism. In this embodiment, a label image containing labels of items in the warehouse is captured by image acquisition. For example, the drone can fly according to a predetermined spatial trajectory and perform image acquisition on a target area on the shelf. The area contains labels that identify item information, and the acquired label image forms the basis for subsequent processing. In some embodiments, a global inspection path of the drone covering all shelf areas is planned based on a static environmental map of the warehouse area. During the flight of the drone, point cloud obstacle data of the onboard laser radar and near-field distance data of the ultrasonic wave are obtained in real time. The flight trajectory is dynamically corrected to generate a collision-free local path. When it is detected that the deviation between the local path and the actual flight direction exceeds a safety threshold, the attitude controller is triggered to adjust the rotor power output to maintain the tracking accuracy of the global path. For example, before entering the first shelf aisle, the lidar detects that the aisle width is less than the preset value. The control center recalculates the flight yaw angle based on this data, making the drone move along the center line of the aisle and maintain a safe distance on both sides.

[0058] In operation S200, image recognition processing is performed on the label image through the recognition network to obtain a character recognition result of the label image, wherein the recognition network is obtained through a calculation path simplification process based on contribution evaluation and a quantization error simulation training process. In some embodiments, the recognition network removes low-contribution transmission paths through a calculation path contribution evaluation mechanism, that is, by evaluating the degree of influence of different calculation paths within the recognition network on the final output result, removing redundant paths with weak influence, and simulating the errors that may be caused by low-precision label images to perform model adaptive training to obtain an optimized recognition network. In this embodiment, the optimized recognition network is directly mounted on the drone end to extract character information from the label image. In some embodiments, after the feature extraction channel of the recognition network extracts the spatial feature map of the label image, the sequence extraction channel of the recognition network constructs a context-aware feature sequence, and the attention channel of the recognition network dynamically weights and generates character recognition results.

[0059] In operation S300, the inventory search result is output based on the real-time comparison of the character recognition result with the records in the inventory database. In some embodiments, the character recognition result is checked for consistency with the inventory records in the database records. Once a deviation is found between the recognition result and the record information, a warning signal containing an abnormal position identifier is generated; in other embodiments, if the recognition result completely matches the record information, a status confirmation signal is generated. In this embodiment, the status information of the inventory items is confirmed through a real-time comparison mechanism, and the final verification conclusion is formatted and output. For example, when it is identified that a specific number in the database record is not in the item location, the system will immediately mark the specific number of the item and trigger an alarm; otherwise, a normal report will be generated if all items match. The report may include the spatial coordinates of the abnormal item, a summary of image evidence and statistical information, and can be stored using tamper-proof data storage technology.

[0060] The image recognition-based inventory checking method disclosed in this embodiment deploys a lightweight recognition network that has been optimized through computational path simplification and quantization error simulation training on an unmanned terminal, thereby achieving real-time processing of image acquisition, character recognition, and data comparison, significantly improving the efficiency and accuracy of inventory verification and eliminating communication delays in the image return server. At the same time, the autonomous movement of the drone solves the problem of secondary shooting caused by stacked occluded labels, effectively resolving the low efficiency of traditional methods, the delay caused by reliance on return server processing, and the need for repeated shooting due to poor image quality.

[0061] like Figure 3 As shown, in some other exemplary embodiments, the recognition network includes a feature extraction channel, a sequence extraction channel and an attention channel, and the recognition network-based image recognition processing is performed on the label image to obtain the character recognition result of the label image, including: operations S210 to S230.

[0062] In operation S210, feature extraction processing is performed on the label image through the feature extraction channel to obtain a spatial feature map. In some embodiments, the feature extraction channel extracts the texture and structural features of the label image through a local perception domain scanning mechanism to construct a multi-dimensional spatial feature mapping relationship. In this embodiment, the spatial dimension is gradually compressed and the feature dimension is enhanced through cascaded feature transformation units to form a feature map with semantic representation capabilities. For example, when a label image containing an item number is input, the feature extraction channel first detects basic edge and texture information through a local perception domain scanning mechanism, and then combines these basic features at a higher level of abstraction through a local perception domain scanning mechanism to form a feature expression describing the character components (such as stroke endpoints and curve segments). Finally, the local perception domain scanning mechanism integrates these component features and outputs a spatial feature map representing the complete character area and its spatial relationship.

[0063] In operation S220, the sequence extraction channel performs temporal feature extraction on the spatial feature map to obtain context-aware sequence features. In some embodiments, the sequence extraction channel uses a bidirectional state transfer unit to analyze the spatial sequence dependencies of the spatial feature map and establish a contextual association model between characters. In this embodiment, the spatial feature map is expanded into a feature sequence in a predetermined direction (such as from left to right or from top to bottom), and the long-distance dependencies between sequence elements are modeled in both the forward and reverse dimensions. The forward state transfer path captures historical feature dependencies, and the reverse state transfer path integrates future feature constraints. For example, the spatial feature map output by the feature extraction channel is expanded in column order and fed into the channel. The forward transfer unit gradually fuses the feature information of the current element with the element on the left to generate a first state vector, while the reverse transfer unit gradually fuses the feature information of the current element with the element on the right to generate a second state vector. Finally, the state fusion outputs a sequence feature containing complete context information.

[0064] In operation S230, the attention channel performs attention weighting processing on the sequence features and outputs the character recognition result. In some embodiments, the attention channel includes a dynamic focusing mechanism and a classification generation mechanism, wherein the dynamic focusing mechanism calculates the weight distribution assigned to different positions in the sequence features for the current character to be decoded; the classification generation mechanism performs weighted fusion on the sequence features based on the weight distribution and decodes the category probability of the current character. In this embodiment, for each target character decoding node in the sequence features, the feature weight distribution can also be adaptively calculated and weighted aggregation performed to solve the problem of character misalignment or partial occlusion. For example, in the attention channel, when the system needs to decode the third character in the label image, the dynamic focusing mechanism first analyzes the entire sequence features and generates a weight distribution map. The weight distribution map shows high weight values ​​at feature positions that potentially contain information about the third character; then the classification generation mechanism performs weighted summation on the sequence features based on the weight distribution to obtain a focused feature vector, and based on this vector, predicts the specific category of the third character through classification. This process is repeated until the complete character sequence is generated and the character recognition result is determined. The image recognition-based database lookup method disclosed in this embodiment constructs a character recognition framework through the cascaded collaboration of a feature extraction channel, a sequence extraction channel, and an attention channel. The feature extraction channel extracts discriminative visual features from the label image to obtain a spatial feature map; the sequence extraction channel models the structural dependencies between characters to obtain context-aware sequence features; and the attention channel implements adaptive decoding and outputs the character recognition results. This structured channel design not only ensures the ability to recognize complex labels (such as tilted, blurred, and partially occluded), but its modular architecture also provides a clear differentiated processing foundation for simplified and optimized computational paths, ultimately enabling efficient execution of high-precision character recognition tasks on resource-constrained drone edge devices.

[0065] like Figure 4 As shown, in some other exemplary embodiments, the method for simplifying the calculation path based on contribution evaluation includes: operations S201 to S203.

[0066] In operation S201, the contribution evaluation value of each computational path in the initial recognition network is calculated respectively. In some embodiments, the contribution evaluation value is comprehensively evaluated by quantifying the frequency of activation of the computational path and its sensitivity to changes in the final recognition output. The core is to analyze the actual influence of a specific computational path on the decision result in multiple forward reasonings, wherein the contribution evaluation value is proportional to the cumulative gradient amplitude of the computational path. In this embodiment, the weight of each computational path in the global computational graph is dynamically evaluated. For example, a batch of labeled image samples are input to the initial recognition network, the frequency of the active state of each computational path during the sample processing is recorded, and the offset amplitude of the output result after temporarily blocking the computational path is observed. These two types of information are combined to generate a numerical measure representing the importance of the computational path.

[0067] In operation S202, the computation paths are sorted according to their contribution evaluation values ​​to obtain a computation path sequence set. In some embodiments, a path optimization sequence is generated using a descending contribution evaluation strategy, with high-contribution paths being prioritized. In this embodiment, a quantifiable path importance distribution map is established using a global sorting strategy.

[0068] In operation S203, based on the set of computational path sequences, computational paths with contribution evaluation values ​​lower than a predetermined threshold in the initial recognition network are removed to obtain a simplified recognition network. In some embodiments, low-contribution computational paths are screened based on a preset threshold and structured removal is performed to retain only efficient computational paths that are critical to the recognition task. In this embodiment, the redundancy of the recognition network is compressed by a selective pruning operation. For example, a predetermined threshold is set as a critical point, all computational paths at the end position in the contribution evaluation value sequence are located, and they are removed or logically shielded from the initial recognition network to obtain a more compact and lightweight recognition network. In some embodiments, a path replacement algorithm can also be used to ensure the integrity of the recognition network. For example, when the contribution evaluation value of a computational path is lower than a dynamic threshold, its output node is redirected to the input end of the computational path with an adjacent high contribution evaluation value. The image recognition-based library search method disclosed in this embodiment significantly reduces the computational complexity of the recognition network while maintaining recognition accuracy through a computational path optimization mechanism driven by contribution evaluation values.

[0069] like Figure 5 As shown, in some other exemplary embodiments, removing the calculation paths with contribution evaluation values ​​lower than a predetermined threshold in the initial recognition network according to the calculation path sequence set to obtain the simplified recognition network includes: operations S2031 to S2033.

[0070] In operation S2031, based on the computational path of the feature extraction channel associated with the computational path sequence set, the feature dimension-upgrading path whose contribution evaluation value is lower than a predetermined threshold in the extended transformation layer of the multi-layer transformation structure of the feature extraction channel is removed. In some embodiments, the multi-layer transformation structure refers to a hierarchical module composed of a plurality of cascaded feature transformation units, wherein the extended transformation layer is responsible for upward expansion mapping of the dimensional space of the input features of the label image, and the feature dimension-upgrading path is the computational path that realizes the dimensional expansion function. In this embodiment, the feature extraction channel is optimized by pruning the feature dimension-upgrading path with the lowest evaluation value in the extended transformation layer. For example, when there are multiple parallel feature dimension-upgrading paths in a certain extended transformation layer, the feature dimension-upgrading path with the lowest evaluation value is removed and the main feature dimension-upgrading path is retained.

[0071] In operation S2032, based on the computational paths associated with the sequence extraction channel in the computational path sequence set, sequential state transfer paths in the bidirectional state transfer unit of the sequence extraction channel whose contribution evaluation values ​​are lower than a predetermined threshold are removed. In some embodiments, the bidirectional state transfer unit includes two independent sequential state transfer paths, one in a forward direction and one in a reverse direction. The sequential state transfer path refers to the state transfer link between neurons within the unit. In this embodiment, redundant sequential state transfer paths are located and removed through neuron importance analysis, and sequence modeling efficiency is optimized. For example, in the forward transfer path of the bidirectional state transfer unit, if the Nth sequential state transfer path is identified as having the lowest contribution to the overall state update, then this sequential state transfer path is removed.

[0072] In operation S2033, the calculation path sequence of the attention channel is centrally associated with the calculation path sequence, and the weight-associated transfer paths with contribution evaluation values ​​lower than a predetermined threshold in the weight mapping unit of the attention channel are removed. In some embodiments, the weight mapping unit establishes a mapping relationship between features and characters through an association matrix, and the weight-associated transfer path refers to the parameter connection between rows and columns of the association matrix. In this embodiment, weakly associated connections are eliminated by a matrix sparsification method. The image recognition-based library search method disclosed in this embodiment achieves precise allocation of computing resources through a channel-by-channel differentiated pruning strategy. The feature extraction channel focuses on dimensionality compression, the sequence extraction channel optimizes temporal dependencies, and the attention channel focuses on key associations. The three work together to achieve the optimal balance between computing power and accuracy.

[0073] like Figure 6 As shown, in some other exemplary embodiments, the step of obtaining the label image of the items in the warehouse includes: operations S110 to S130.

[0074] In operation S110, raw optical data from the labels of the items in the storage area is collected. In some embodiments, the raw optical data includes multidimensional spectral information that characterizes the optical response characteristics of the item surface. The raw optical data may include both visible and infrared spectral information. In this embodiment, a dynamic exposure control mechanism is used to adaptively adjust the sensitivity of the photosensitive element. For example, when a drone hovers at a designated location in front of a shelf, the drone captures raw optical data from the storage area containing spectral information in different bands.

[0075] In operation S120, the raw optical data is converted to a numerical domain for dynamic range compression to generate an image value sequence for dimensionality reduction storage. In some embodiments, this conversion process compresses the raw data with a wide dynamic range into a limited numerical interval, eliminating interference from extreme values ​​and increasing information density. In this embodiment, data representation efficiency can also be optimized through adaptive bit allocation, for example, using a logarithmic compression strategy for high dynamic range regions and a linear scaling strategy for low dynamic range regions to generate a uniformly encoded image value sequence. Alternatively, the collected raw optical data can be mapped to a predefined discrete numerical space based on its physical properties to generate a compact image value sequence more suitable for subsequent processing.

[0076] In operation S130, the label image is obtained according to the image value sequence. In some embodiments, the image value sequence is spatially arranged and reconstructed to form a two-dimensional pixel matrix, and the matrix generates a visual label image through color space conversion and interpolation optimization. In this embodiment, a label image in a standard format is generated by inverse mapping the image value sequence to the image space. For example, the compressed image value sequence is rearranged into a row pixel array according to a preset resolution rule, and then converted into an RGB format label image that can be processed by the recognition network through a color restoration algorithm. The image recognition-based library search method disclosed in this embodiment significantly reduces the data storage and transmission level while retaining key recognition features by compressing the dynamic range and converting the format of the original optical data, so that the drone end can efficiently process the label image.

[0077] like Figure 7 As shown, in some other exemplary embodiments, performing a numerical domain conversion process of dynamic range compression on the original optical data to generate an image numerical sequence for dimensionality reduction storage includes: operations S121 to S123.

[0078] In operation S121, a numerical dynamic range of the raw optical data is determined based on the raw optical data. In some embodiments, the numerical dynamic range refers to the quantitative span between the maximum and minimum valid response values ​​in the raw optical data, reflecting the breadth of the data distribution. In this embodiment, the dynamic range parameter is obtained by statistically analyzing the extreme value distribution and central tendency characteristics of the raw optical data. For example, local brightness peaks are extracted from highly reflective areas, dark feature points are captured from shadow areas, and the global brightness interval span of the raw optical data is calculated. The upper and lower bounds of the valid numerical distribution of the raw optical data are determined, and the absolute value difference between the upper and lower bounds is calculated as a measure of the numerical dynamic range.

[0079] In operation S122, a compression mapping boundary of the original optical data is determined based on the numerical dynamic range. In some embodiments, the compression mapping boundary can be dynamically set based on the storage precision limit of the target numerical space and the regional response characteristics of the original optical data. In some specific embodiments, a high protection threshold boundary can be set for the first text area and a low protection threshold boundary can be set for the second background area, to form an adaptive compression mapping curve.

[0080] In operation S123, according to the compression mapping boundary, dynamic range compression mapping is performed on the original optical data to obtain the image value sequence. In some embodiments, the mapping process uses a piecewise linear transformation mechanism to project the original optical data to the target discrete value domain. In this embodiment, data dimensionality reduction can be achieved through a boundary-driven numerical redistribution algorithm or bit depth compression can be achieved through a quantization step controller. For example, the minimum end of the compression mapping boundary is set to correspond to the starting value of the target numerical sequence, and the maximum end corresponds to the ending value. The original optical data is linearly mapped to the target discrete value according to its relative position in the boundary interval, and finally an equivalent low-bit image value sequence is generated; for example, high dynamic pixel values ​​are mapped to the first sub-segment of the compression interval, and medium dynamic pixel values ​​are mapped to the second sub-segment to generate an N-bit integer image value sequence. The image recognition-based library query method disclosed in this embodiment balances data accuracy and storage efficiency through an adaptive boundary compression mechanism to achieve efficient image processing of edge devices.

[0081] like Figure 8 As shown, in some other exemplary embodiments, the method for quantization error simulation training processing includes: operations S204 to S206.

[0082] In operation S204, an error simulation node is inserted into the forward computation path of the simplified recognition network and a gradient pass-through path is established in the backward computation path to construct a simulated recognition network. The error simulation node is used to perform forward distortion injection during training of the simulated recognition network. In some embodiments, the error simulation node is configured to inject numerical discretization distortion during forward propagation to simulate the numerical distortion effect in the computational environment of low-precision labeled images. The gradient pass-through path is configured to maintain the continuity of the gradient data flow during backward propagation. In this embodiment, by inserting an error simulation node at a key interface in the forward computation and configuring a gradient pass-through path in the backward computation path, a simulated recognition network capable of simulating quantization error for training is constructed. For example, a first error simulation node is inserted at the output of a convolutional computation layer. This node performs a discretization distortion operation on the output spatial feature map during forward computation. Simultaneously, a gradient pass-through path is configured in the backward computation path so that the gradient data received by this node is directly passed to the convolution parameter update process at its input.

[0083] In operation S205, the simulated recognition network is trained using sample labeled images to generate a standard recognition network. In some embodiments, training is performed using sample labeled images containing labeled images of real scenes. By alternating forward distortion injection and backward gradient pass-through to update the parameters of the simulated recognition network, the simulated recognition network is continuously subjected to forward distortion interference. Through a quantitative perceptual training process, the simulated recognition network's adaptability to distortion is optimized, thereby generating a standard recognition network. For example, a batch of labeled image samples of a vault scene is input. During forward inference, distortion signals are continuously injected through the error simulation node. During backward propagation, gradients are used to update the weight parameters of the simulated recognition network through the gradient pass-through path. After multiple rounds of iteration, the output of the simulated recognition network stabilizes under the distortion environment, generating a standard recognition network.

[0084] In operation S206, the error simulation nodes and the gradient direct transfer path in the standard recognition network are removed to obtain the recognition network after the quantized error simulation training process. In some embodiments, the error simulation nodes and gradient direct transfer paths added in the training phase are removed to restore the original calculation path. At this time, the weight parameters of the recognition network have the ability to maintain a reliable recognition rate in a low-precision environment. In this embodiment, the error simulation nodes are deleted and the gradient direct transfer path is released, and the optimized weight parameters are retained to form a recognition network that can be directly deployed on a low-precision computing unit. The image recognition-based library query method disclosed in this embodiment actively introduces controllable forward calculation distortion and establishes a gradient lossless return mechanism during the training phase, so that the recognition network fully adapts to the numerical constraints of the target deployment environment at the model parameter level, ensuring that the lightweight network after pruning and simplification can still maintain character recognition performance close to the original accuracy when running on the low-precision chip on the drone end, fundamentally solving the problem of accuracy loss in edge deployment.

[0085] like Figure 9 As shown, in some other exemplary embodiments, inserting an error simulation node into the forward calculation path of the simplified recognition network includes: operations S2041 to S2043.

[0086] In operation S2041, a first error simulation node is inserted into the feature map generation end of each convolution transform layer in the feature extraction channel to perform discretization distortion injection on the spatial feature map data. In some embodiments, the convolution transform layer refers to a processing unit that generates two-dimensional feature data by sliding calculation of the convolution kernel, and its feature map generation end is the location where the unit outputs data; discretization distortion injection refers to the operation of forcibly mapping continuous floating-point feature values ​​to a discrete low-bit numerical space. This process introduces approximation errors. In this embodiment, the quantization effect is simulated by applying numerical discrete constraints to the feature generation node. For example, the first error simulation node is deployed at the feature map generation end of the convolution transform layer. The node processes the floating-point values ​​of each channel in the generated feature map according to a discrete rule (e.g., rounding to the nearest discrete level value) to generate a spatial feature map with discretization distortion, which is then passed to the downstream sequence extraction channel.

[0087] In operation S2042, a second error simulation node is inserted at the weight parameter calculation end of the sequence extraction channel to perform fixed-point truncation distortion injection on the timing modeling parameters. In some embodiments, the weight parameter calculation end refers to the parameter matrix position within the sequence extraction channel used to calculate the timing state transfer weights; the timing modeling parameters are weight coefficients involved in solving the state transfer equation; fixed-point truncation distortion injection refers to the operation of restricting high-precision weight parameters to a fixed-point representation range with a limited bit width, thereby simulating the calculation hardware limitations by constraining the quantization accuracy of the state transfer weight parameters. For example, a second error simulation node is inserted at the weight parameter calculation end position of the bidirectional state transfer unit of the sequence extraction channel. This node constrains the timing modeling parameters to be used according to discrete rules (e.g., truncated to a 16-bit fixed-point representation domain) and performs state transfer calculations using the quantized weights.

[0088] In operation S2043, a third error simulation node is inserted at the output of the correlation calculation of the attention channel to perform nonlinear distribution-adapted quantization distortion injection on the attention weight distribution data. In some embodiments, the node uses a nonlinear scaling method to numerically remap the attention weight distribution data, maintaining the relative relationship between weights while adapting to low-precision representation. The correlation calculation output refers to the position where the unnormalized weight distribution is output after the correlation calculation between features is completed; the attention weight distribution data is a set of scores that characterize the importance of different feature positions; and nonlinear distribution-adapted quantization distortion injection refers to the operation of using a probability distribution-sensitive mapping function (such as logarithmic domain compression) to perform numerical compression. In this embodiment, the fidelity of weight compression is improved through probability-aware nonlinear quantization. For example, after the correlation calculation of the second attention channel is completed, the third error simulation node is inserted. This node converts the original correlation score into a discrete weight distribution through a discrete rule and then passes it to the normalization link. The image recognition-based database lookup method disclosed in this embodiment uses a channel-differentiated distortion injection strategy, uniformly discretizes feature maps, performs fixed-point truncation of timing weights, and implements nonlinear probability adaptation of attention weights to accurately simulate the quantitative characteristics of each computing link of edge devices, providing key guarantees for the stable operation of the recognition network in the low-precision environment of edge chips.

[0089] like Figure 10 As shown, in some other exemplary embodiments, establishing a gradient direct transfer path in the reverse calculation path of the simplified recognition network includes: operations S2044 to S2045.

[0090] In operation S2044, in response to the reverse computation path passing through the error simulation node, the upstream gradient data stream of the error simulation node is obtained. In some embodiments, undistorted gradient data at the input of the error simulation node is captured during the backpropagation phase of training. In this embodiment, the undistorted gradient data at the input is extracted and temporarily stored in real time via a gradient pass-through path. For example, when backpropagating to the first error simulation node during training, the system automatically records the gradient tensor data received at the input of this node and transmitted back by subsequent network layers.

[0091] In operation S2045, the upstream gradient data stream is directly transferred to the downstream computing node connected to the output of the error simulation node, thereby skipping the computational process within the error simulation node. In some embodiments, a gradient pass-through path is established to bypass the computational process of the error simulation node, directly assigning the captured gradient stream to the upstream computing unit connected to the output of the error simulation node. This avoids the gradient blocking problem caused by the non-differentiable distortion operation and maintains the integrity of gradient propagation. In this embodiment, a gradient pass-through path is configured to connect the upstream and downstream nodes. For example, for the first error simulation node inserted into the feature extraction channel, the upstream gradient data stream obtained at its input is directly transferred to the convolutional transformation layer upstream of the node for parameter update, completely skipping the reverse computational process of the discretization distortion operation within the node. The image recognition-based library search method disclosed in this embodiment solves the problem of non-differentiable quantization nodes by constructing a gradient pass-through path mechanism within the error simulation node, ensuring that the weights of the recognition network can still stably converge to the optimal solution under high-intensity distortion simulation environments.

[0092] like Figure 11 As shown, in some other exemplary embodiments, collecting the original optical data of the labels of the items in the warehouse area includes: operations S111 to S113.

[0093] In operation S111, multiple original images taken from multiple angles at the same shooting position of the items in the warehouse area are obtained, and local feature matching points are extracted from each of the original images. In some embodiments, the local feature matching points are determined by analyzing the center points of regions with significant gradient changes or texture structures in the image. In some embodiments, each local feature matching point carries a spatial vector representation describing the characteristics of the local region. In this embodiment, visual key points that can be correlated with each other are extracted from images of different perspectives. For example, in an original image taken from a first angle of a shelf label, a first key point at the corner of the numbered character is identified and its description vector is calculated; in the corresponding image taken from a second angle, a second key point with a similar description vector is found in an area with a similar spatial position to the first key point as a local feature matching point.

[0094] In operation S112, the multiple original images are aligned based on the local feature matching points of each original image. In some embodiments, a perspective transformation relationship between the multiple original images is determined using a transformation matrix to eliminate geometric distortion caused by perspective differences. In this embodiment, geometric distortion caused by differences in shooting positions is eliminated using a multi-perspective projection transformation. For example, an optimal transformation matrix is ​​determined based on multiple pairs of local feature matching point coordinates between the first and second original images. This transformation matrix is ​​then applied to the second original image to perform a perspective transformation, so that the pixel coordinate spaces of the shelf label regions in the two original images completely overlap, thereby achieving alignment.

[0095] In operation S113, pixel weighted fusion processing is performed on the aligned multiple original images to generate the original optical data of the labels of the items in the warehouse area. In some embodiments, weights are assigned according to the credibility of the pixel positions, and weighted average calculation is performed on the corresponding pixel values ​​in the multiple aligned images. In this embodiment, multi-perspective information is integrated by a weighted method of credibility evaluation. For example, for the three aligned label images, the clarity evaluation value of the first image, the noise suppression ability of the second image, and the shadow removal effect of the third image are respectively detected at the first pixel position; the fusion weight of each image at the pixel position is dynamically calculated based on the three indicators, and finally the distortion-free high-fidelity pixel value of the position is generated. The image recognition-based library search method disclosed in this embodiment significantly improves the detail integrity of the label area image through feature matching, geometric alignment and credible fusion of multi-angle images, effectively solves the problem of local occlusion or reflection interference caused by stacking of items, and provides high-quality input data for subsequent character recognition. In other embodiments, an adjustable polarization filter assembly can be further provided at the front end of the optical acquisition device of the drone. The adjustable polarization filter assembly includes a filter and a micromotor. The optical plane of the filter is fixedly installed at an angle to the main optical axis of the camera. The filter can also be driven by the micromotor to perform continuous rotation of 0-360 degrees around the optical axis, dynamically matching the polarization direction of the ambient light, suppressing the mirror reflection interference of metal shelves and precious metal surfaces, and solving the problem of label overexposure in the warehouse environment through the active matching mechanism of the polarization direction, thereby greatly improving the recognition reliability.

[0096] like Figure 12 As shown, in some other exemplary embodiments, the method for multi-angle shooting at the same shooting position includes: operation S1111.

[0097] In operation S1111, multiple horizontal camera rings are arranged around the same photographic location of the items in the storage area. The vertical pitch angle interval between adjacent horizontal camera rings relative to the same photographic location is a first predetermined angle. Each horizontal camera ring includes multiple spaced-apart photographic positions, and the horizontal azimuth angle interval between two adjacent photographic positions in each horizontal camera ring relative to the same photographic location is a second predetermined angle. In some embodiments, the first and second horizontal camera rings are spaced-apart vertically to cover different vertical pitch viewing angles. Multiple photographic positions are evenly distributed in azimuth within the ring surface of each horizontal camera ring to cover the horizontal azimuth viewing angle. In this embodiment, the first predetermined angle controls the vertical coverage density, and the second predetermined angle controls the horizontal sampling density. For example, the pitch angle of the first horizontal camera ring is set as a base angle, and the second horizontal camera ring is offset from the base angle by a first pitch angle interval. Each horizontal camera ring is provided with N photographic positions spaced in azimuth by a second predetermined angle within the ring surface, thereby achieving three-dimensional spherical coverage of the target object. The image recognition-based library search method disclosed in this embodiment eliminates visual blind spots through a three-dimensional shooting position layout, ensures full-view acquisition of complex stacking scenes, and completely solves the character recognition blind spot problem caused by the limitation of a single viewing angle.

[0098] like Figure 13 As shown, in some other exemplary embodiments, the method for determining the shooting position includes: operations S111A to S111D.

[0099] In operation S111A, a sequence of infrared scanning images of the vertical height of the stacked area of ​​the items in the storage area is collected. In some embodiments, continuous vertical scanning is performed to obtain the thermodynamic distribution characteristics of the stacked structure in the vertical direction. This distribution can represent the physical contact relationship between the items. In this embodiment, three-dimensional thermodynamic data is constructed by scanning multiple height levels along the vertical axis using a drone. For example, after controlling the drone to perform a horizontal scan at a first height level of the stacked area, it vertically descends to a second height level to perform a complementary scan, thereby forming a sequence of infrared scanning images covering the entire stacked height.

[0100] In operation S111B, based on the infrared scanning image sequence, the distribution of thermodynamic gradient change rates between adjacent planes in the infrared scanning image sequence is calculated. In some embodiments, the thermodynamic gradient change rate is calculated by analyzing the difference in thermal radiation intensity at corresponding spatial locations in the infrared scanning image sequence, reflecting the energy transfer characteristics of the contact surfaces between adjacent objects. In this embodiment, the characteristics of interlayer thermal conduction mutations are quantified through temperature field gradient vector analysis. For example, the absolute difference in temperature values ​​between vertically adjacent first and second infrared images is calculated pixel by pixel and divided by the interlayer spacing to generate a two-dimensional distribution map representing the thermodynamic gradient change rate.

[0101] In operation S111C, in response to the thermodynamic gradient rate of change exceeding a critical threshold, the spatial coordinate corresponding to the location where the thermodynamic gradient rate of change exceeds the critical threshold is marked as a stacking boundary. In some embodiments, when the thermodynamic gradient rate of change exceeds a preset physical constraint value, an item layer boundary is determined to exist at that location. In this embodiment, a threshold trigger mechanism is used to automatically identify the stacking structure, and the center point of the area is mapped to the three-dimensional coordinate system of the storage area and marked as the stacking boundary.

[0102] In operation S111D, based on the stacking boundary position, the three-dimensional spatial coordinates of the shooting position of the items in the storage area are determined. In some embodiments, the stacking boundary position is used as a reference point and the shooting position is calculated in combination with the preset shooting angle constraint. For example, after identifying the first stacking boundary position, the position is used as the center of the sphere to calculate the spherical coordinate point that meets the preset pitch angle and azimuth conditions, and use it as the target shooting position for label image acquisition. The image recognition-based library search method disclosed in this embodiment accurately identifies the stacking boundary of items through the characteristics of the thermodynamic gradient change rate, provides a physical position basis for multi-angle shooting, and effectively solves the problem of layered misjudgment of traditional visual methods in complex stacking scenes.

[0103] like Figure 12 As shown, in some other exemplary embodiments, the method for multi-angle shooting at the same shooting position further includes: operation S1112.

[0104] In operation S1112, in response to the determination of the stacking boundary position, the surrounding radius of the horizontal shooting ring is set to a preset distance, and multi-angle image acquisition is performed within the preset distance range. In some embodiments, the preset distance is dynamically configured according to the stacking boundary characteristics to ensure that the shooting angle of view can completely cover the associated area of ​​the stacking boundary. For example, when the stacking boundary is detected, the surrounding radius of the horizontal shooting ring is set to a first preset distance, and then a multi-angle close-range high-precision image acquisition operation is performed under the constraint of the surrounding radius. The image recognition-based library search method disclosed in this embodiment, through the adaptive control of the shooting distance triggered by the stacking boundary, maximizes the information capture efficiency while ensuring image clarity, and significantly improves the efficiency of multi-angle image acquisition operations in complex stacking scenarios.

[0105] In this embodiment, the appointment request sent by the customer can also execute the operations S100 to S300, operation S210, operation S220, operation S230, operation S201, operation S202, operation S203, operation S2031, operation S2032, operation S2033, operation S110, operation S120, operation S130, operation S121, operation S122, operation S123, operation S204, operation S205, operation S206, operation S2041, operation S2042, operation S2043, operation S2044, operation S2045, operation S111, operation S112, operation S113, operation S1111, operation S111A, operation S111B, operation S111C, operation S111D, and operation S1112 described above on the system side, which will not be repeated here.

[0106] Based on the above-mentioned image recognition-based library query method, this application also provides an image recognition-based library query device for the drone end. Figure 14 The device is described in detail.

[0107] Figure 14 The structural block diagram of the image recognition-based library query device for a drone according to an embodiment of the present application is schematically shown.

[0108] like Figure 14 As shown, the image recognition-based library search device 200 of this embodiment includes a first acquisition module 210 , a first processing module 220 , and a second processing module 230 .

[0109] The first acquisition module 210 is configured to execute a natural language request received from a user via the digital human interface. In one embodiment, the first acquisition module 210 can be used to execute the operation S100 described above. Furthermore, the first acquisition module 210 can also be used to execute the operations S110, S120, S130, S121, S122, S123, S111, S112, S113, S1111, S111A, S111B, S111C, S111D, and S1112 described above, which are not further described herein.

[0110] The first processing module 220 is configured to perform image recognition processing on the label image through a recognition network to obtain a character recognition result of the label image, wherein the recognition network is obtained through a calculation path simplification process based on contribution evaluation and a quantization error simulation training process. In one embodiment, the first processing module 220 can be used to perform the operation S200 described above, which will not be repeated here. At the same time, the first processing module 220 can also be used to perform the operations S210, operation S220, operation S230, operation S201, operation S202, operation S203, operation S2031, operation S2032, operation S2033, operation S204, operation S205, operation S206, operation S2041, operation S2042, operation S2043, operation S2044, and operation S2045 described above, which will not be repeated here.

[0111] The second processing module 230 is configured to perform a real-time comparison between the character recognition result and the records in the inventory database and output the inventory search result. In one embodiment, the second processing module 230 can be used to perform the operation S300 described above, which will not be repeated here.

[0112] According to embodiments of the present application, any multiple modules among the first acquisition module 210, the first processing module 220, and the second processing module 230 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. According to embodiments of the present application, at least one of the first acquisition module 210, the first processing module 220, and the second processing module 230 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or any suitable combination of these. Alternatively, at least one of the first acquisition module 210, the first processing module 220, and the second processing module 230 may be at least partially implemented as a computer program module that, when executed, performs the corresponding functionality.

[0113] Figure 15 A block diagram of an electronic device suitable for implementing a library search method based on image recognition according to an embodiment of the present application is schematically shown.

[0114] like Figure 15As shown, an electronic device 900 according to an embodiment of the present application includes a processor 901, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 902 or programs loaded from a storage unit 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to the embodiment of the present application.

[0115] Various programs and data required for the operation of the electronic device 900 are stored in the RAM 903. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method flow according to the embodiment of the present application by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flow according to the embodiment of the present application by executing the programs stored in one or more memories.

[0116] According to an embodiment of the present application, electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to bus 904. Electronic device 900 may also include one or more of the following components connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 908 including a hard disk; and a communication section 909 including a network interface card such as a LAN card or modem. Communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 910 as needed, so that computer programs read from the removable media can be installed into storage section 908 as needed.

[0117] This application also provides a computer-readable storage medium, which may be included in the apparatus described in the above embodiments, or may exist independently without being incorporated into the apparatus. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of this application is implemented.

[0118] According to an embodiment of the present application, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0119] The embodiments of the present application also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the image recognition-based library search method provided in the embodiments of the present application.

[0120] The computer program executes the above functions defined in the apparatus of the embodiment of the present application when the computer program is executed by the processor 901. According to the embodiment of the present application, the apparatus described above can be implemented by a computer program module.

[0121] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0122] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from a removable medium 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiment of the present application are performed. According to the embodiment of the present application, the above-described means and the like can be implemented by computer program modules.

[0123] According to an embodiment of the present application, the program code for executing the computer program provided by the embodiment of the present application can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0124] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0125] Those skilled in the art will appreciate that the features described in the various embodiments of this application may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in this application. In particular, the features described in the various embodiments of this application may be combined and / or coupled in various ways without departing from the spirit and teachings of this application. All such combinations and / or couplings fall within the scope of this application.

[0126] The embodiments of the present application have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present application. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present application, those skilled in the art may make various substitutions and modifications, and these substitutions and modifications should all fall within the scope of the present application.

Claims

1. A database search method based on image recognition, characterized in that: Applied to a drone, the method includes: Get label images of items in the warehouse; performing image recognition processing on the label image through a recognition network to obtain a character recognition result of the label image, wherein the recognition network is obtained through a calculation path simplification processing based on contribution evaluation and a quantization error simulation training processing; The character recognition result is compared with the records in the inventory database in real time, and the inventory search result is output.

2. The method according to claim 1, characterized in that The recognition network includes a feature extraction channel, a sequence extraction channel, and an attention channel. The image recognition processing is performed on the label image based on the recognition network to obtain a character recognition result of the label image. Performing feature extraction processing on the label image through the feature extraction channel to obtain a spatial feature map; Performing temporal feature extraction processing on the spatial feature map through the sequence extraction channel to obtain context-aware sequence features; Attention weighted processing is performed on the sequence features through the attention channel, and the character recognition result is output.

3. The method according to claim 2, characterized in that The method for simplifying the calculation path based on contribution evaluation includes: Calculate the contribution evaluation value of each calculation path in the initial identification network respectively; Sort the calculation paths according to the contribution evaluation value to obtain a calculation path sequence set; According to the calculation path sequence set, calculation paths with contribution evaluation values ​​lower than a predetermined threshold in the initial recognition network are removed to obtain a simplified recognition network.

4. The method according to claim 3, characterized in that The step of removing the calculation paths having contribution evaluation values ​​lower than a predetermined threshold in the initial recognition network according to the calculation path sequence set to obtain a simplified recognition network includes: According to the computation paths of the feature extraction channel that are collectively associated with the computation path sequence, removing feature dimension-raising paths whose contribution evaluation values ​​in the extended transformation layers are lower than a predetermined threshold value in the multi-layer transformation structure of the feature extraction channel; and / or removing, based on the calculation path sequence set associated with the calculation path of the sequence extraction channel, the timing state transfer paths in the bidirectional state transfer unit of the sequence extraction channel whose contribution evaluation values ​​are lower than a predetermined threshold; And / or based on the calculation path sequence centrally associating the calculation path sequence of the attention channel, removing the weight-associated transfer path whose contribution evaluation value in the weight mapping unit of the attention channel is lower than a predetermined threshold.

5. The method according to claim 3, characterized in that The step of obtaining label images of items in the storage area includes: Collecting original optical data of labels of items in the storage area; Performing a numerical domain conversion process of dynamic range compression on the raw optical data to generate an image numerical sequence for dimensionality reduction storage; The label image is obtained according to the image value sequence.

6. The method according to claim 5, characterized in that The performing of a numerical domain conversion process of dynamic range compression on the original optical data to generate an image numerical sequence for dimensionality reduction storage includes: determining a numerical dynamic range of the original optical data according to the original optical data; determining a compression mapping boundary of the original optical data according to the numerical dynamic range; According to the compression mapping boundary, dynamic range compression mapping is performed on the original optical data to obtain the image value sequence.

7. The method according to claim 5, characterized in that The method for quantization error simulation training processing includes: Inserting an error simulation node in a forward calculation path of the simplified recognition network and establishing a gradient direct transfer path in a reverse calculation path to construct a simulated recognition network, wherein the error simulation node is used to perform a forward distortion injection operation in training the simulated recognition network; Using sample labeled images to train the simulated recognition network to generate a standard recognition network; The error simulation node and the gradient direct transfer path in the standard recognition network are removed to obtain the recognition network after the quantization error simulation training process.

8. The method according to claim 7, characterized in that Inserting an error simulation node into the forward calculation path of the simplified recognition network includes: Inserting a first error simulation node at the feature map generation end of each convolution transformation layer in the feature extraction channel to perform discretization distortion injection on the spatial feature map data; and / or inserting a second error simulation node at a weight parameter calculation end of the sequence extraction channel to perform fixed-point truncation distortion injection on the timing modeling parameters; And / or insert a third error simulation node at the correlation calculation output end of the attention channel to perform quantization distortion injection for nonlinear distribution adaptation on the attention weight distribution data.

9. The method according to claim 7, characterized in that The step of establishing a gradient direct transfer path in the reverse calculation path of the simplified recognition network comprises: In response to the reverse calculation path passing through the error simulation node, acquiring an upstream gradient data flow of the error simulation node; The upstream gradient data flow is directly transferred to the downstream computing node connected to the output end of the error simulation node, so as to skip the computing process inside the error simulation node.

10. The method according to claim 5, characterized in that The collecting of original optical data of labels of items in the storage area includes: Acquire multiple original images of the items in the storage area taken from the same shooting position at multiple angles, and extract local feature matching points of each of the original images; Aligning the multiple original images according to the local feature matching points of the original images; Pixel weighted fusion processing is performed on the aligned multiple original images to generate original optical data of the labels of the items in the warehouse area.

11. The method according to claim 10, characterized in that The method for multi-angle shooting at the same shooting position includes: A plurality of horizontal shooting rings are provided around the same shooting position of the items in the storage area, wherein the pitch angle interval in the vertical direction between adjacent horizontal shooting rings relative to the same shooting position is a first predetermined angle, each of the horizontal shooting rings includes a plurality of shooting positions distributed at intervals, and the azimuth angle interval in the horizontal direction between two adjacent shooting positions on each horizontal shooting ring relative to the same shooting position is a second predetermined angle.

12. The method according to claim 11, characterized in that The method for determining the shooting position includes: Collecting a sequence of infrared scanning images of the vertical height of the stacking area of ​​the items in the storage area; Calculating, based on the infrared scanning image sequence, a distribution of thermodynamic gradient change rates of adjacent planes in the infrared scanning image sequence; In response to the rate of change of the thermodynamic gradient exceeding a critical threshold, marking the corresponding spatial coordinates where the rate of change of the thermodynamic gradient exceeds the critical threshold as a stacking boundary position; Based on the stacking boundary position, the three-dimensional spatial coordinates of the shooting position of the items in the storage area are determined.

13. The method according to claim 12, characterized in that The method for multi-angle shooting at the same shooting position also includes: In response to the determination of the stacking boundary position, the surrounding radius of the horizontal shooting ring is set to a preset distance, and multi-angle image acquisition is performed within the preset distance range.

14. A library search device based on image recognition, characterized in that: Applied to a drone, the device includes: a first acquisition module configured to execute a natural language request received from a user via a digital human interface; a first processing module configured to perform image recognition processing on the label image through a recognition network to obtain a character recognition result of the label image, wherein the recognition network is obtained through a calculation path simplification processing based on contribution evaluation and a quantization error simulation training processing; The second processing module is configured to perform real-time comparison between the character recognition result and the records in the inventory database, and output the inventory query result.

15. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images