A two-dimensional code intelligent scanning method and system based on multi-modal perception

Abstract

The application provides a two-dimensional code intelligent scanning method and system based on multi-modal perception, relates to the technical field of two-dimensional code scanning, and solves the technical problem of low recognition efficiency of the prior art in a complex scene. The method comprises the following steps: collecting a visual image, a depth image, an infrared reflection intensity image and radar point cloud data of a to-be-scanned object to obtain multi-dimensional data; performing feature extraction on the multi-dimensional data to obtain multi-dimensional features; inputting the multi-dimensional features into a Dempster Shafer evidence theory fusion device to output a credibility; and deciding a scanning mode of the to-be-scanned object based on the credibility. The application can be applied to industrial production, logistics storage and other processes with high scanning accuracy requirements.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of two-dimensional code scanning, and specifically relates to a two-dimensional code intelligent scanning method and system based on multi-modal perception. BACKGROUND

[0002] Nowadays, two-dimensional code scanning technology has been widely applied to many fields such as industrial production, logistics and warehousing, mobile payment and the like. However, most of the traditional two-dimensional code scanning methods are based on single-modal perception, such as relying only on visual images. In the face of complex scenes such as strong light reflection, object occlusion and low light, significant technical defects are exposed. Specifically, when there is strong light reflection on the surface of a two-dimensional code, single-modal visual acquisition will cause the image to be overexposed due to light reflection, the gray scale features of black and white modules are blurred, and thus the decoding algorithm fails; in the case that a two-dimensional code is partially occluded, single-modal perception cannot obtain complete image information, which easily causes incomplete feature extraction and leads to a significant decrease in recognition accuracy; and in a low-light environment, the contrast of visual images is reduced, and texture features become unobvious, which also makes scanning and recognition difficult, and it is difficult to meet the actual application requirements. SUMMARY

[0003] The application provides a two-dimensional code intelligent scanning method and system based on multi-modal perception, which solves the technical problem of low recognition efficiency of existing two-dimensional code scanning technology in complex scenes.

[0004] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0005] In a first aspect, a two-dimensional code intelligent scanning method based on multi-modal perception is provided, comprising:

[0006] Collecting visual images, depth images, infrared reflection intensity maps and radar point cloud data of an object to be scanned to obtain multi-dimensional data;

[0007] Performing feature extraction on the multi-dimensional data to obtain multi-dimensional features, wherein the multi-dimensional features include texture features extracted based on visual images, curvature features calculated based on infrared reflection intensity maps and depth images, and normal vector features calculated based on radar point cloud data;

[0008] Inputting the multi-dimensional features into a Dempster Shafer evidence theory fusion device to output credibility;

[0009] Making a decision on a scanning mode of the object to be scanned based on the credibility, wherein the scanning mode includes a direct scanning mode, a light reflection suppression mode, an occlusion compensation mode and a low light enhancement mode.

[0010] It should be noted that the Dempster Shafer evidence theory fusion device of the present application is a calculation framework for realizing multi-modal feature confidence fusion, the core of which is to convert the texture features from visual images, the curvature features from infrared and depth images, and the normal vector features from radar point cloud data into the confidence of the decodable state of the two-dimensional code according to the Dempster Shafer evidence theory by constructing a recognition framework, defining a basic probability assignment model and evidence combination rules.

[0011] Based on the above technical solution, in the two-dimensional code intelligent scanning method based on multi-modal perception provided by the present application, by collecting visual images, depth images, infrared reflection intensity maps and radar point cloud data of the object to be scanned, a multi-dimensional data fusion system is constructed, which can effectively solve the recognition failure problem of traditional single-modal scanning in complex scenes such as strong light reflection, shielding and low light. This method can accurately judge the decodable state of the two-dimensional code by extracting multi-dimensional features such as texture, curvature and normal vector, and using Dempster Shafer evidence theory for confidence fusion, and then using different scanning modes to realize intelligent recognition and scanning of the two-dimensional code. The combination of multi-modal perception and intelligent decision mechanism not only improves the environmental adaptability and recognition confidence of two-dimensional code scanning, but also reduces manual intervention through automatic mode switching, significantly improves scanning efficiency and reliability, and provides technical support for industrial production, logistics and other scenes with high scanning accuracy requirements.

[0012] In combination with the first aspect described above, in a possible implementation manner, the texture feature acquisition manner comprises:

[0013] The visual image is subjected to block processing to obtain a plurality of image blocks;

[0014] The local binary pattern (LBP) feature vector of each image block is extracted;

[0015] Statistical features in each image block are calculated, including the contrast, entropy value and energy of the image block gray level co-occurrence matrix;

[0016] The LBP feature vector and the statistical features are combined to obtain the texture features of the visual image.

[0017] In combination with the first aspect described above, in a possible implementation manner, the curvature feature acquisition manner comprises:

[0018] The gradient amplitude ir and the Hessian matrix eigenvalues λ1 and λ2 of the infrared reflection intensity map I are calculated;

[0019] Based on the gradient amplitude , Hessian matrix eigenvalues λ1, λ2 and the depth map D(x, y), the curvature feature K is calculated according to the curvature feature calculation formula.

[0020] In combination with the first aspect, in a possible implementation manner, the curvature feature calculation formula satisfies:

[0021] ; wherein, S pix represents the physical size of the pixel, the unit is mm, μ represents the depth weight coefficient, and the value range is [0.2, 0.4].

[0022] In combination with the first aspect, in a possible implementation manner, the method for obtaining the normal vector feature comprises:

[0023] KD Tree index is established for each point in the radar point cloud data;

[0024] The spatial position of the point p i is searched for a neighborhood point set N(p i ) within a search radius r;

[0025] The covariance matrix of the neighborhood point set N(p i ) is calculated, and the eigenvector corresponding to the minimum eigenvalue in the covariance matrix is taken as the normal vector of the point p i ;

[0026] The normal vector direction histogram of each point is counted to obtain the normal vector feature.

[0027] In combination with the first aspect, in a possible implementation manner, the method for inputting the multi-dimensional feature into the Dempster Shafer evidence theory fusion device comprises:

[0028] An identification framework Θ={m1=“decodable”, m2=“needs enhancement”, m3=“undecodable”} is constructed; wherein, the identification framework is used to judge the belonging category of the two-dimensional code scanning result of the object to be scanned;

[0029] The basic probability m t (A) of the texture feature F k , the curvature feature F n and the normal vector feature F i is respectively calculated; wherein, the basic probability represents the confidence degree that the two-dimensional code scanning result of the object to be scanned belongs to the category A∈Θ in the identification framework under the condition of the feature F i∈{t,k,n} ;

[0030] The multi-dimensional features are fused based on the evidence combination rule of the Dempster Shafer evidence theory to obtain the fused basic probability m total(A); where, the fusion basic probability represents the comprehensive confidence level of determining whether the QR code scanning result of the object to be scanned belongs to category A after integrating the basic probabilities of multidimensional features;

[0031] The credibility level Bel=m is calculated according to the formula. total (m1)+λ×m total (m2); where λ represents the weighting coefficient.

[0032] In conjunction with the first aspect above, in one possible implementation, the basic probability m i The formula for (A) is: ;in, This indicates that feature i is the preset baseline feature value of category A, and σ represents the scaling parameter;

[0033] The fusion basic probability m total The formula for (A) is: ;in, To represent the empty set, The symbol represents the intersection.

[0034] In conjunction with the first aspect above, in one possible implementation, the decision-making process based on the scanning mode of the object to be scanned based on confidence level includes:

[0035] If the confidence level is greater than T1, then execute the direct scan mode;

[0036] If T2 ≤ confidence level ≤ T1, then pattern matching is performed based on multidimensional features, including:

[0037] When the curvature feature of the target region exists K max >K th At that time, the reflection suppression mode is activated; K max K represents the maximum curvature eigenvalue. th Indicates the preset curvature threshold;

[0038] When the peak ratio of the normal vector histogram of the target region is <H th At that time, the occlusion compensation mode is executed; the normal vector histogram is obtained based on the statistical analysis of the normal vectors of all points, H th Indicates the preset peak-to-peak ratio threshold;

[0039] When the texture features of the target region exist F t_max <F t_th At that time, low-light enhancement mode is executed; F t_max F represents the maximum texture feature value. t_th This indicates a preset texture feature threshold;

[0040] If the credibility is less than T2, then the manual verification mode is triggered;

[0041] The target region represents a two-dimensional code scanning region, and is obtained by edge detection or image recognition of the image by the visual sensor; T1 represents a preset first threshold value, and T2 represents a preset second threshold value.

[0042] In combination with the first aspect, in a possible implementation, the automatic execution operation of the light reflection suppression mode includes: turning off the light of the visual sensor, raising the infrared emission power to a preset value, and deflecting the scanning angle by a preset angle in the direction of the rate of change of the curvature feature;

[0043] The automatic execution operation of the occlusion compensation mode includes: controlling the radar sensor to perform point cloud scanning in a preset pitch angle range;

[0044] The automatic execution operation of the low-light enhancement mode includes: activating the light of the visual sensor and activating the infrared auxiliary lighting.

[0045] In a second aspect, a two-dimensional code intelligent scanning system based on multi-modal perception is provided, including:

[0046] A data acquisition module is configured to acquire a visual image, a depth image, an infrared reflection intensity map, and radar point cloud data of a to-be-scanned object, and obtain multi-dimensional data.

[0047] A feature extraction module is configured to extract features from the multi-dimensional data, and obtain multi-dimensional features, including texture features extracted based on the visual image, curvature features calculated based on the infrared reflection intensity map and the depth image, and normal vector features calculated based on the radar point cloud data.

[0048] A fusion decision module is configured to input the multi-dimensional features into a Dempster Shafer evidence theory fusion device, and output a credibility.

[0049] The credibility is used to make a decision on a scanning mode of the to-be-scanned object; the scanning mode includes a direct scanning mode, a light reflection suppression mode, an occlusion compensation mode, and a low-light enhancement mode.

[0050] In a third aspect, a two-dimensional code intelligent scanning device based on multi-modal perception is provided, including a communication unit and a processing unit.

[0051] The communication unit is configured to receive a visual image, a depth image, an infrared reflection intensity map, and radar point cloud data of a to-be-scanned object, to obtain multi-dimensional data, and to send a decision result of a scanning mode.

[0052] The processing unit is configured to extract features from the multi-dimensional data, and obtain multi-dimensional features including texture features, curvature features, and normal vector features.

[0053] inputting the multi-dimensional features into a Dempster Shafer evidence theory combiner to output a belief;

[0054] deciding a scanning mode for the object to be scanned based on the belief, the scanning mode including a direct scanning mode, a highlight suppression mode, an occlusion compensation mode, and a low-light enhancement mode.

[0055] In a fourth aspect, the present application provides a two-dimensional code intelligent scanning device based on multi-modal perception, comprising: a processor and a storage medium; the storage medium comprises instructions, and the processor is configured to execute the instructions to implement the method described in the first aspect and any possible implementation manner of the first aspect. The two-dimensional code intelligent scanning device can be an electronic device or a chip in an electronic device.

[0056] In a fifth aspect, the present application provides a computer-readable storage medium, which stores instructions, and when the instructions are executed on the two-dimensional code intelligent scanning device based on multi-modal perception, the two-dimensional code intelligent scanning device based on multi-modal perception executes the method described in the first aspect and any possible implementation manner of the first aspect.

[0057] In a sixth aspect, the present application provides a computer program product comprising instructions, and when the computer program product is executed on the two-dimensional code intelligent scanning device based on multi-modal perception, the two-dimensional code intelligent scanning device based on multi-modal perception executes the method described in the first aspect and any possible implementation manner of the first aspect.

[0058] The present application provides a two-dimensional code intelligent scanning method and system based on multi-modal perception, which can realize adaptive and accurate decision of two-dimensional code recognition in complex environments. At the feature extraction level, by fusing the infrared reflection intensity and the geometric feature quantization method of depth data, the gradient mutation and depth discontinuity caused by surface reflection or occlusion are converted into calculable feature parameters, making the algorithm more sensitive to environmental interference. The statistical analysis method of normal vector feature converts the three-dimensional space direction distribution into a histogram peak ratio, providing a quantitative basis for occlusion judgment. At the decision fusion level, the belief calculation framework based on evidence theory constructs a mathematical mapping model of multi-feature confidence, effectively handles the conflict and complementary relationship of multi-source data, and enables the system to dynamically switch the scanning mode according to the comprehensive confidence of multi-dimensional features such as texture, curvature, and normal vector. This not only avoids the risk of misjudgment of a single feature, but also realizes adaptive optimization of the scanning strategy, ultimately improving the recognition reliability and environmental adaptability of two-dimensional code scanning in complex scenes such as strong reflection, low light, and partial occlusion.

[0059] It should be understood that the description of technical features, technical solutions, advantages or similar language in this application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it can be understood that the description of a feature or advantage means that the specific technical feature, technical solution or advantage is included in at least one embodiment. Therefore, the description of technical features, technical solutions or advantages in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and advantages described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantages of a particular embodiment. In other embodiments, additional technical features and advantages can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creating any creative labor.

[0061] Figure 1 A system architecture diagram of a two-dimensional code intelligent scanning method based on multi-modal perception provided by an embodiment of the present application;

[0062] Figure 2 A flowchart of a two-dimensional code intelligent scanning method based on multi-modal perception provided by an embodiment of the present application;

[0063] Figure 3 A flowchart of another two-dimensional code intelligent scanning method based on multi-modal perception provided by an embodiment of the present application;

[0064] Figure 4 A flowchart of another two-dimensional code intelligent scanning method based on multi-modal perception provided by an embodiment of the present application;

[0065] Figure 5 A structural diagram of a two-dimensional code intelligent scanning method based on multi-modal perception provided by an embodiment of the present application;

[0066] Figure 6 A hardware structure diagram of a two-dimensional code intelligent scanning method based on multi-modal perception provided by an embodiment of the present application. DETAILED DESCRIPTION

[0067] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" herein is only a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone. In addition, "at least one" means one or more, and "multiple" means two or more. "First", "second", and the like do not limit the quantity and execution order, and "first", "second", and the like do not necessarily mean different.

[0068] It should be noted that in the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of "exemplary" or "for example" is intended to present relevant concepts in a concrete manner.

[0069] The two-dimensional code intelligent scanning method based on multi-modal perception provided by the embodiments of the present application can be applied to a two-dimensional code intelligent scanning system based on multi-modal perception as shown in Figure 1 As shown in Figure 1 The communication system includes a data acquisition module, a feature extraction module, and a fusion decision module.

[0070] The data acquisition module is configured to acquire a visual image, a depth image, an infrared reflection intensity map, and radar point cloud data of the object to be scanned, and obtain multi-dimensional data.

[0071] The feature extraction module is configured to extract features from the multi-dimensional data to obtain multi-dimensional features, including texture features extracted based on the visual image, curvature features calculated based on the infrared reflection intensity map and the depth image, and normal vector features calculated based on the radar point cloud data.

[0072] The fusion decision module is configured to input the multi-dimensional features into a Dempster Shafer evidence theory fusion device and output a credibility.

[0073] The scanning mode of the object to be scanned is determined based on the credibility, and the scanning mode includes a direct scanning mode, a reflection suppression mode, a shielding compensation mode, and a low light enhancement mode.

[0074] To solve the technical problem of recognition failure of a two-dimensional code in a complex scene such as strong reflection, shielding, and low light in the prior art, the embodiments of the present application provide a two-dimensional code intelligent scanning method based on multi-modal perception, which includes the following steps:

[0075] Collect visual images, depth images, infrared reflection intensity maps and point cloud data of the object to be scanned to obtain multi-dimensional data;

[0076] Feature extraction is performed on the multi-dimensional data to obtain multi-dimensional features, including texture features extracted based on visual images, curvature features calculated based on infrared reflection intensity maps and depth images, and normal vector features calculated based on point cloud data.

[0077] The multi-dimensional features are input into a Dempster Shafer evidence theory fusion device to output a credibility.

[0078] Based on the credibility, a scanning mode of the object to be scanned is determined, including a direct scanning mode, a reflection suppression mode, an occlusion compensation mode and a low light enhancement mode.

[0079] Based on this, the method can improve the environmental adaptability and recognition reliability of the two-dimensional code scanning through multi-modal data fusion and intelligent decision mechanism.

[0080] As shown in Figure 2 The two-dimensional code intelligent scanning method based on multi-modal perception provided by the embodiments of the application comprises the following steps:

[0081] S1, collecting visual images, depth images, infrared reflection intensity maps and point cloud data of the object to be scanned to obtain multi-dimensional data.

[0082] The visual images are obtained by a visual sensor (such as a camera), the depth images are obtained by a structured light sensor or a binocular stereo vision camera, the infrared reflection intensity maps are collected by an infrared light source in cooperation with an infrared camera, and the point cloud data are obtained by scanning a laser radar (LiDAR) or a millimeter wave radar.

[0083] In some implementations, the arrangement of each sensor can adopt a coaxial integrated design, an array distribution or a modular combination to ensure the time and space synchronous collection of multi-source data, for example, the camera, the infrared sensor and the laser radar are integrated in the same plane of the scanning device, and synchronous sampling is realized through hardware triggering.

[0084] S2, feature extraction is performed on the multi-dimensional data to obtain multi-dimensional features.

[0085] The multi-dimensional features include texture features extracted based on visual images, curvature features calculated based on infrared reflection intensity maps and depth images, and normal vector features calculated based on point cloud data.

[0086] In some implementations, the texture features can be obtained by traditional methods such as local binary pattern (LBP), histogram of oriented gradient (HOG), or Gabor filter, or can be extracted by convolutional neural network (CNN) for end-to-end feature extraction of visual images, or can be calculated by gray level co-occurrence matrix (GLCM) to obtain statistics such as contrast and entropy; the curvature features can be calculated by differential geometry method (such as gradient-based surface fitting), or can be extracted by morphological operator to obtain the curvature change of the image edge; the normal vector features are calculated by point cloud registration algorithm, octree index, or RANSAC (random sample consensus) algorithm.

[0087] It should be noted that, in order to realize the multi-dimensional representation of the surface micro-texture, macro-geometric morphology and spatial orientation of the two-dimensional code, feature extraction is performed on the obtained multi-dimensional data, so that different modal data can effectively reflect the influence of environmental interference (such as reflection and shielding) on the features of the two-dimensional code.

[0088] For example, in a low-light scene, the texture features of the infrared reflection intensity map can be extracted instead of the visual image to avoid feature loss caused by insufficient light.

[0089] S3, input the multi-dimensional features into a Dempster Shafer evidence theory fusion device, and output the credibility.

[0090] The Dempster Shafer evidence theory fusion device is a multi-source information decision-making model based on evidence theory. By defining the recognition framework of “decodable”, “need to enhance”, and “uncodeable”, the confidence of each modal feature is converted into basic probability assignment (BPA), and then the conflict and complementary relationship of multi-source data is processed by using evidence combination rule, and finally the comprehensive credibility is output.

[0091] In some implementations, improved Dempster combination rules (such as Yager rule, DSmT rule) can be used to process highly conflicting evidence, or genetic algorithm and particle swarm optimization algorithm can be used to adaptively adjust the feature weight coefficient.

[0092] It should be noted that the fusion device does not depend on prior probability, and can improve the decision reliability by accumulating evidence, and is suitable for uncertain reasoning of multi-modal data.

[0093] S4, making a decision on the scanning mode of the object to be scanned based on the credibility.

[0094] The credibility reflects the comprehensive confidence that the two-dimensional code can be correctly decoded in the current environment, and the automatic switching of the scanning strategy is realized based on a preset threshold.

[0095] In some implementations, the scanning mode can be dynamically adjusting sensor parameters (such as camera exposure time, infrared light source power) according to the confidence, or triggering multi-sensor collaborative scanning (such as multi-angle laser radar scanning, multi-spectral image fusion), or starting hardware compensation mechanism (such as adjusting the scanning angle of the mechanical arm).

[0096] It should be noted that the decision logic of the scanning mode needs to be decoupled from the hardware execution module to support policy customization configuration in different scenarios.

[0097] For example, in an industrial pipeline scenario, when the confidence is in the T2T1 interval and the curvature feature is abnormal, the anti-reflection mode can be automatically triggered to eliminate the interference of the metal surface reflection by turning off the visible light source, enhancing the infrared illumination, and adjusting the scanning angle.

[0098] Based on the above technical solutions, the two-dimensional code intelligent scanning method based on multi-modal perception provided by the application solves the technical bottleneck of traditional single-mode scanning in complex environments by constructing a "multi-source data acquisition cross-modal feature extraction evidence theory fusion adaptive mode decision" processing system. The method uses the complementary nature of multi-dimensional data such as vision, depth, infrared, and point cloud, and combines the uncertainty reasoning ability of evidence theory to realize accurate judgment of the state of the two-dimensional code and intelligent adaptation of the scanning strategy, effectively solving the problems of image distortion caused by strong reflection, feature missing caused by partial occlusion, and imaging blur caused by low light, etc. It provides a universal technical solution for high-precision scanning needs in the fields of intelligent manufacturing, smart logistics, etc.

[0099] In a possible implementation of the embodiment of the application, the above S1 can be implemented by the following steps, which are described in detail as follows:

[0100] S101, acquiring a visual image of the object to be scanned by a visual sensor.

[0101] The visual sensor can use a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) camera, which focuses on the scanning area through an optical lens, and acquires RGB or grayscale images according to a preset frame rate (such as 30fps) and resolution (such as 1920x1080). The exposure time, gain, and other parameters of the camera can be automatically adjusted according to the ambient light intensity to avoid overexposure or underexposure.

[0102] In some implementations, the timestamps of the visual image and other sensor data can be aligned through a hardware trigger mechanism (such as a general-purpose input / output interface signal GPIO signal) or a software synchronization protocol (such as the Network Time Protocol NTP), to ensure the spatiotemporal consistency of multi-modal data.

[0103] It should be noted that the field of view angle of the visual sensor needs to cover the maximum spatial range where the two-dimensional code may appear, and the lens needs to have an anti-glare coating to reduce the impact of strong light reflection on image quality.

[0104] For example, in a logistics warehouse scenario, an industrial-grade area camera can be used to dynamically adjust the frame rate according to the speed of the conveyor belt to ensure clear imaging of the two-dimensional code in motion.

[0105] S102, collect infrared reflection intensity map and depth image by using infrared sensor and depth sensor.

[0106] The infrared sensor is composed of an infrared light source (such as an 850nm infrared LED) and an infrared camera, which generates an infrared reflection intensity map by emitting infrared light and receiving reflected signals, which can reflect the reflection characteristics of the object surface to infrared light, and is suitable for strong light reflection or low light scenes. The depth sensor can use a ToF (Time of Flight) camera or a structured light sensor to measure the flight time of light or the distortion of the pattern to calculate the three-dimensional coordinates of the object surface and generate a depth image D(x,y), where (x,y) represents the spatial coordinates in millimeters.

[0107] In some implementations, the infrared sensor and the depth sensor can be integrated into the same module to improve data synchronization by sharing the optical path; or they can be deployed separately and data registration can be achieved through spatial calibration algorithms (such as hand-eye calibration).

[0108] It should be noted that the power of the infrared light source needs to be adjusted according to the scene requirements, such as reducing the power to avoid infrared reflection saturation when scanning metal surfaces, and increasing the power to enhance the signal strength in dark environments.

[0109] For example, in a medical cold chain scenario, a ToF depth sensor is used with a narrowband infrared light source to penetrate the condensation mist and collect two-dimensional code infrared reflection map and depth data on the vaccine packaging box.

[0110] S103, collect point cloud data by radar sensor and establish KD Tree index.

[0111] The radar sensor can use a laser radar to emit laser or electromagnetic waves and receive echoes to generate point cloud data containing three-dimensional coordinates (x,y,z) and reflection intensity. For each point cloud point p i , a KD Tree index is established based on its spatial position for subsequent neighborhood point search and normal vector calculation.

[0112] In some implementations, the laser radar can be a mechanical rotating type (such as a 16-line laser radar) or a solid-state type, and the point cloud density can be adjusted according to the scanning range and accuracy requirements (such as 10 points per square centimeter).

[0113] It should be noted that point cloud data acquisition needs to be synchronized with visual and infrared data in time. The sampling delay between sensors can be compensated by hardware clock (such as GPS clock) or software interpolation (such as linear interpolation).

[0114] For example, in an industrial robot sorting scenario, solid-state LiDAR is used to collect point cloud data of the package surface in real time. KD Tree indexing can accelerate point cloud registration and obstacle detection, and provide spatial location basis for occlusion compensation mode.

[0115] Based on the above technical solution, a multi-sensor collaborative acquisition and data synchronization mechanism was implemented to accurately acquire multi-dimensional data such as visual, infrared, depth, and point cloud data, providing a diverse information foundation for subsequent feature extraction and fusion decision-making. This data acquisition scheme not only adapts to complex environments such as strong reflections, low light, and partial occlusion, but also improves the spatiotemporal consistency of multimodal data through KD Tree indexing and hardware synchronization technology, laying a data foundation for intelligent QR code scanning.

[0116] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S2 can be implemented through the following S201, S202 and S203, which are explained in detail below:

[0117] S201. Perform block processing on the visual image and extract texture features.

[0118] Specifically, the visual image is first divided into multiple 8×8 pixel image blocks, and local texture features are focused through block processing.

[0119] Then, a Local Binary Pattern (LBP) feature vector is calculated for each image patch: by comparing the gray values ​​of the center pixel with those of its 8 neighboring pixels, an 8-bit binary code is generated, and its histogram is used as the LBP feature.

[0120] Next, the gray-level co-occurrence matrix (GLCM) statistical features of the image patch are calculated, including contrast (reflecting texture sharpness), entropy (reflecting texture randomness), and energy (reflecting texture uniformity).

[0121] Finally, the LBP feature vector is combined with the GLCM statistical features to form the final texture feature vector.

[0122] In some implementations, the block size can be adaptively adjusted; for example, high-resolution images can use 16×16 pixel blocks to balance computational efficiency and feature accuracy.

[0123] S202. Calculate curvature features based on infrared reflection intensity map and depth image.

[0124] wherein the curvature feature calculation formula is: ;

[0125] In the formula, S pix is a pixel physical size (unit: mm) determined by sensor parameters.

[0126] λ1 and λ2 are Hessian matrix eigenvalues of the infrared reflection intensity map I ir , used to represent second-order gradient changes of the image.

[0127] is a gradient amplitude of the infrared reflection intensity map, reflecting a pixel intensity change rate.

[0128] D(x, y) is a depth map, and is a second-order partial derivative of the depth, representing a depth change trend.

[0129] μ is a depth weight coefficient, with a value range of [0.2, 0.4], which can be increased in a strong light reflection scene to enhance the influence of the depth feature.

[0130] In some implementations, the calculation of the Hessian matrix can be preprocessed by a Gaussian convolution kernel to reduce the interference of noise on gradient calculation.

[0131] It should be noted that the curvature feature K is used to represent the local bending degree of the object surface. When there is a light reflection point on the surface of the two-dimensional code, the mutation of the infrared reflection intensity will cause and |λ1|+|λ2| to increase, so that the value of K exceeds the threshold value.

[0132] For example, if a water droplet causes light reflection on the surface of the two-dimensional code, the gradient amplitude of the infrared reflection intensity map will significantly increase, and the calculated value of K may reach 0.7. If the preset threshold value K th =0.5, the light reflection suppression mode will be triggered in the subsequent process.

[0133] S203, calculating a normal vector feature based on the radar point cloud data. Specifically, it includes;

[0134] (1) Establishing a KDTree index for each point in the radar point cloud data to quickly search for neighborhood points.

[0135] (2) Taking point p i as the center, searching for a neighborhood point set N(p i ) within a radius r (with a value of 0.510 mm, which can be adjusted according to the point cloud density).

[0136] (3) Calculating the covariance matrix of the neighborhood point set N(p i ), and taking the eigenvector corresponding to the minimum eigenvalue as the normal vector of point p iThe normal vector represents the surface orientation at that point.

[0137] (4) Divide the 360° direction into 16 angular regions, and statistically analyze the normal vector direction histogram of all points to obtain the normal vector characteristics.

[0138] In some implementations, the size of the neighborhood point set must satisfy |N(p i )|≥5, to ensure the statistical significance of the covariance matrix calculation.

[0139] It should be noted that the peak ratio of the normal vector histogram is the ratio of the number of intervals with the highest count in the normal vector direction histogram to the number of intervals with the second highest count, i.e., peak bin count / second peak bin count, where bin represents the angular region, which can be used to determine the surface continuity: when the QR code is occluded, the normal vector direction of the occluded region becomes discrete, which will lead to a decrease in the peak ratio.

[0140] Therefore, normal vector features are used to reflect the three-dimensional orientation distribution and continuity of an object's surface. By statistically analyzing the peak ratio of the normal vector direction histogram, it is possible to quantitatively determine whether there are occlusions or other factors on the QR code surface that cause the normal vector direction to be discrete, thus providing a three-dimensional spatial feature basis for scanning mode decision-making.

[0141] For example, the surface normal vectors of a complete QR code have a consistent direction, and the histogram shows a single peak with a peak-to-peak ratio of 3.0. If it is obscured by the edge of a cardboard box, the peak-to-peak ratio of the normal vector histogram may drop to 1.5. If a preset threshold H is set... th If the value is 2.0, the occlusion compensation mode will be triggered in subsequent processes.

[0142] Based on the above technical solution, a multi-dimensional quantitative representation of the QR code surface state is achieved by fusing texture, curvature, and normal vector features. Texture features are used to capture the grayscale distribution pattern of the visual image, curvature features are used to quantify the abrupt changes in infrared and depth caused by reflection, and normal vector features are used to statistically analyze the orientation distribution of the three-dimensional surface. These features provide multi-dimensional input for subsequent credibility calculations of Dempster-Shafer evidence theory, enabling the system to accurately determine the decodeability of QR codes in complex scenarios such as strong reflection, occlusion, and low light, laying the foundation for intelligent decision-making in scanning modes.

[0143] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S3 can be implemented through the following S301, S302 and S303, which are explained in detail below:

[0144] S301. Define the recognition framework.

[0145] Wherein, the recognition framework Θ is {“decodable”, “need to enhance”, “undecodable”}, used to divide the categories of the two-dimensional code scanning result:

[0146] “decodable” (m1): indicates that the two-dimensional code can be directly decoded under the current feature condition;

[0147] “need to enhance” (m2): indicates that the scanning mode needs to be adjusted (such as anti-reflection suppression, low-light enhancement, etc.) to improve the decoding success rate;

[0148] “undecodable” (m3): indicates that the current condition cannot be decoded and needs manual intervention.

[0149] In some implementations, the categories can be extended according to the scene requirements, for example, adding “blur needs to be focused” and “partially occluded” sub-states to improve the decision granularity.

[0150] It should be noted that the categories of the recognition framework need to be strictly mapped to the subsequent scanning modes, for example, “need to enhance” corresponds to specific modes such as anti-reflection suppression and occlusion compensation, to ensure that the credibility calculation result can directly guide the operation.

[0151] For example, in an industrial pipeline scene, if there is no obvious interference on the surface of the two-dimensional code, the recognition framework determines m1, and the system performs direct scanning; if there is slight reflection, it is determined as m2, triggering the anti-reflection suppression mode.

[0152] S302: Calculate the basic probability distribution of each feature.

[0153] Calculate the basic probability m i (A) for the texture feature Ft, the curvature feature Fk, and the normal vector feature Fn, respectively.

[0154] Wherein, the basic probability calculation formula is:

[0155] ;

[0156] In the formula: F i is the current extracted feature value (i=t, k, n corresponds to texture, curvature, and normal vector, respectively);

[0157] represents the preset reference feature value of feature i in category A, which is determined by historical sample training (for example, the LBP feature distribution of the texture feature reference value in the m1 category corresponds to the artificial judgment of the clear two-dimensional code);

[0158] σ is a scaling parameter, used to control the smoothness of the probability distribution, usually taking 0.1-1.0, which can be obtained by cross-validation optimization.

[0159] In some implementations, the reference feature value Support dynamic update: when the system scans new scenes, if the sample characteristics of continuous successful decoding differ from the original benchmark by more than 10%, it is automatically updated , to improve adaptability.

[0160] It should be noted that the basic probability m i (A) represents the support of a single feature to a class, and the value range is [0, 1], and the sum of the probabilities of the same feature to all classes is 1. For example, if the texture feature matches the benchmark feature m1 well, then m t (m1) tends to 1, m t (m2) and m t (m3) tend to 0.

[0161] For example, assume that the texture feature F t of a certain two-dimensional code has an Euclidean distance of 0.2 with the benchmark feature of the m1 class, and σ = 0.5, then: ; if and are both greater than 0.5, then m t (m1) ≈ 0.67, indicating that the texture feature supports the confidence of “decodable” is 67%.

[0162] S303, fusing the basic probability based on the Dempster combination rule to obtain the credibility Bel.

[0163] The specific steps include:

[0164] 1. Evidence combination: use the combination rule of Dempster-Shafer evidence theory to calculate the fused basic probability m total (A):

[0165] ;

[0166] The numerator is the sum of the products of all sub-feature basic probabilities when the intersection is A, and the denominator is the normalization factor (excluding evidence conflict cases).

[0167] 2. Credibility calculation: calculate the final credibility according to the fusion result:

[0168] ; wherein λ is a weight coefficient (0 ≤ λ ≤ 1), used to adjust the influence of the “need to enhance” class on the credibility, and λ = 0.3-0.5 is often taken in industrial scenes to balance efficiency and accuracy.

[0169] In some implementations, when the evidence conflict degree (the denominator ) exceeds 0.7, it is considered that the multi-feature contradiction is significant, and data needs to be re-collected or the feature extraction module needs to be checked to avoid misjudgment.

[0170] It is noted that the Dempster combination rule can effectively handle the complementarity and conflict of multi-source data: for example, texture features support m1(m t =0.6), but curvature features support m2(m k =0.7), the fused m total (m1) will integrate the support of both, avoiding the dominance of a single feature in decision-making.

[0171] For example, assume:

[0172] Texture features: m t (m1)=0.6, m t (m2)=0.3, m t (m3)=0.1;

[0173] Curvature features: m k (m1)=0.2, m k (m2)=0.7, m k (m3)=0.1;

[0174] Normal vector features: m n (m1)=0.5, m n (m2)=0.3, m n (m3)=0.2.

[0175] Then, when fused, the product sum of the intersection m1 is calculated: m t (m1)×m k (m1)×m n (m1)=0.06;

[0176] The product sum of the intersection m2 is calculated: m t (m2)×m k (m2)×m n (m2)=0.063;

[0177] The product sum of the intersection m3 is calculated: m t (m3)×m k (m3)×m n (m3)=0.002;

[0178] In addition to the above three types of same-class intersections, other combinations belong to conflict situations, such as m1×m2×m3, m1×m2×m2, etc., which need to be calculated one by one Product and sum to get the value of the denominator term;

[0179] Then, according to the formula, it is normalized, assuming that after normalization, m total (m1)≈0.42, m total(m2)≈0.51; Note: This example is a hypothetical result. In actual applications, the product sum of all conflict situations should be calculated.

[0180] If λ = 0.4, the credibility Bel = 0.42 + 0.4 * 0.51 = 0.622, indicating that after integrating the multi-dimensional features such as texture, curvature, and normal vector, the overall support degree of the "decodable" state of the two-dimensional code is about 62%. Its essence is a quantitative index obtained by fusing the confidence of multiple sources of features through Dempster-Shafer evidence theory. Therefore, the higher the value of Bel ∈ [0, 1], the closer the two-dimensional code is to the "decodable" state.

[0181] Based on the above technical solution, by constructing a three-layer recognition framework, defining a basic probability calculation model and Dempster combination rule, the confidence fusion of texture, curvature, and normal vector features is realized. This mechanism not only solves the misjudgment problem of single-modal features in complex scenes (such as when strong light causes texture features to fail, curvature features can compensate), but also dynamically adjusts the influence of the "need to enhance" category through λ weight, so that the credibility calculation result can accurately reflect the decodable state of the two-dimensional code.

[0182] In a possible implementation manner of the embodiment of the present application, in combination with Figure 2 As shown in FIG. 4, the decision-making condition of S4 specifically includes: Figure 3

[0183] (1) When the credibility Bel > T1, it is determined that the two-dimensional code can be directly decoded, and the default scanning parameters are executed.

[0184] Wherein, T1 is a high credibility threshold, usually set to 0.7-0.9, and the specific value depends on the requirement of the scene on scanning reliability, such as 0.85 for industrial scenes and 0.7 for consumer scenes. The execution logic of the direct scanning mode includes:

[0185] 1. The vision sensor acquires images with standard exposure parameters without the need for light compensation.

[0186] 2. Trigger the two-dimensional code decoding algorithm to directly parse the two-dimensional code information in the vision image.

[0187] It should be noted that the direct scanning mode is not completely without considering environmental interference, but the multi-dimensional features have been integrated to determine that the interference does not affect decoding. For example, if there is slight glare, but the fusion credibility of the curvature feature and other features is still > T1, there is no need to start the glare suppression mode.

[0188] ​For example, in a warehouse management scenario, the two-dimensional code is pasted on the flat surface of a carton. The captured visual image has clear texture, continuous depth map and consistent normal vector distribution. The calculated Bel=0.88>T1=0.8. The system directly triggers decoding, and the scanning is completed within 10 ms.

[0189] (2) When the credibility is between T2 and T1, the system executes the corresponding enhanced mode according to the multi-dimensional feature matching result.

[0190] Wherein, T2 is a low credibility threshold (usually 0.3-0.5), at this time, the interference type needs to be located through feature analysis, which specifically includes:

[0191] 1. Anti-reflection suppression mode.

[0192] Trigger condition: curvature feature K max >K th ;

[0193] Wherein, K th is the curvature threshold (such as 0.5), when the maximum curvature feature value of the target area exceeds K th , it indicates that there is infrared reflection intensity mutation caused by reflection.

[0194] Execution operation:

[0195] 1. Turn off the visual sensor light to avoid visible light reflection;

[0196] 2. Increase the infrared emission power to a preset value (such as 80% of the maximum power), and use infrared light to penetrate the reflection layer;

[0197] 3. The scanning angle is deflected by a preset angle in the direction of the maximum curvature feature change rate, for example, with an angle increment of 2°, gradually increasing from 5°, and recalculating the credibility each time the angle is deflected until the credibility is greater than T1.

[0198] 2. Occlusion compensation mode.

[0199] Trigger condition: peak value ratio of normal vector histogram <H th ;

[0200] Wherein, H th is the peak value ratio threshold (such as 2.0), and the peak value ratio = peak value bin count / second peak value bin count. When the value <H th , it indicates that the normal vector distribution is discrete, and there is occlusion.

[0201] Execution operation: control the radar sensor to reciprocate within a preset pitch angle, such as [10°, +10°], and reconstruct the outline of the occluded area through multi-angle point cloud data.

[0202] 3. Low light enhancement mode

[0203] Trigger condition: texture feature F t_max <F t_th ;

[0204] Where F t_th is the texture feature threshold (e.g. 0.6), when the maximum texture feature value F t_max is lower than F t_th , it indicates that the image contrast is insufficient (low light causes).

[0205] Execution operation: activate the visual sensor LED fill light and start the infrared auxiliary lighting.

[0206] In some implementations, if multiple enhancement mode trigger conditions are met at the same time, the system executes according to the preset priority: anti-reflection>shelter compensation>low light enhancement.

[0207] It should be noted that the positioning of the target area can be realized by visual edge detection (such as Canny operator) or deep learning recognition (such as YOLO model), ensuring that the feature analysis is only for the area where the two-dimensional code is located.

[0208] For example, when scanning the two-dimensional code on the glass, the ambient light or light reflection causes the curvature feature K max =0.7>K th =0.5, the anti-reflection mode is executed: turn off the visual light, increase the infrared power to 80%, and deflect the scanning angle to the right side by 10°, then recalculate the multi-dimensional features, if the curvature feature K max drops to 0.4, the credibility increases to 0.75, then direct scanning can be triggered.

[0209] (3) When the credibility Bel

[0210] Where the execution logic of the manual check mode includes:

[0211] 1. Output the visual image, depth map, and visualized superimposed results of infrared reflection intensity map, and mark the suspicious areas (such as reflection points and shelter boundaries) with red color;

[0212] 2. Generate a feature analysis report containing texture feature vector, curvature heat map, and normal vector histogram data;

[0213] 3. Prompt the operator to intervene through sound and light alarm or upper computer interface, for example, display "two-dimensional code cannot be decoded, please check surface contamination or shelter".

[0214] In some implementations, the manual check mode can integrate the physical button function of the code scanning gun, and after the operator confirms the two-dimensional code state, it can forcibly trigger the secondary scanning (such as re-scanning after adjusting the angle) or manually input the two-dimensional code information.

[0215] It should be noted that the manual checking mode is a fault-tolerant mechanism of the system, which is suitable for extreme scenarios such as serious pollution of the two-dimensional code, multi-angle shielding, or sensor failure, at which time the fusion credibility of the multi-dimensional features cannot support automatic decision.

[0216] For example, if the two-dimensional code is covered with oil stains, the visual image texture is blurred (F t_max = 0.3 < F t_th = 0.6), the peak ratio of the normal vector histogram = 1.2 < H th = 2.0, the calculation Bel = 0.25 < T2 = 0.3, the system pops up a prompt box and highlights the oil stain area, and the operator re-calculates the credibility after wiping.

[0217] It should be noted that the preset threshold values of T1 and T2 can be optimized through offline training (such as collecting a large number of industrial scene samples) or online self-adaptation (such as adjusting according to the historical decoding success rate), to ensure that the decision boundary meets the actual application requirements.

[0218] Based on the above technical solutions, through the three-layer decision mechanism driven by credibility, the multi-modal perception data and the scanning strategy can be directly associated through a mathematical model, avoiding the reliance on empirical rules, so that the system has the ability of self-learning and environmental adaptation. The mechanism can accurately judge the decodable state of the two-dimensional code, realize the adaptive optimization of the scanning strategy, and improve the recognition reliability and environmental adaptability in complex scenes such as strong light, low light, and partial shielding.

[0219] Some data in the above formula are calculated by removing the dimension, and the formula is obtained by software simulation of a large amount of data to obtain a formula closest to the actual situation; the preset parameters and the preset threshold values in the formula are set by a person skilled in the art according to the actual situation or obtained by a large amount of data simulation.

[0220] The above mainly introduces the scheme of the embodiments of the present application from the perspective of device implementation. It can be understood that each device, for example, a two-dimensional code intelligent scanning device based on multi-modal perception, contains at least one of the corresponding hardware structure and software module for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0221] The embodiments of the present application can divide the functional units of the two-dimensional code intelligent scanning device based on multi-modal perception according to the above method examples. For example, each functional unit can be divided according to each function, or two or more functions can be integrated in one processing unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. It should be noted that the division of the units in the embodiments of the present application is illustrative, and is only a logical function division. In actual implementation, another division method can be used.

[0222] In the case of using an integrated unit, Figure 5 A possible structural diagram of the two-dimensional code intelligent scanning device based on multi-modal perception (denoted as two-dimensional code intelligent scanning device based on multi-modal perception 50) involved in the above embodiments is shown, which includes a processing unit 501 and a communication unit 502, and can also include a storage unit 503. Figure 5 The structural diagram shown can be used to illustrate the structure of the two-dimensional code intelligent scanning device based on multi-modal perception involved in the above embodiments.

[0223] When Figure 5 When the structural diagram shown is used to illustrate the structure of the two-dimensional code intelligent scanning device based on multi-modal perception involved in the above embodiments, the processing unit 501 is used to control and manage the actions of the two-dimensional code intelligent scanning device based on multi-modal perception, specifically including feature extraction on the collected multi-dimensional data to obtain multi-dimensional features including texture features, curvature features and normal vector features, inputting the multi-dimensional features into a Dempster-Shafer evidence theory fusion device to output a credibility, and deciding a scanning mode of the object to be scanned based on the credibility, the scanning mode including a direct scanning mode, a reflection suppression mode, a shielding compensation mode and a low light enhancement mode.

[0224] The communication unit 502 is used for communication between the two-dimensional code intelligent scanning device based on multi-modal perception and other devices, specifically for receiving visual images, depth images, infrared reflection intensity maps and radar point cloud data of the object to be scanned to obtain multi-dimensional data, and sending the decision result of the scanning mode.

[0225] The storage unit 503 is used to store the program code and data of the two-dimensional code intelligent scanning device based on multi-modal perception, including storing algorithm programs for realizing multi-modal data acquisition, feature extraction, evidence theory fusion and scanning mode decision, and presetting reference feature values, threshold parameters and other data.

[0226] In a possible implementation, the processing unit 501 can also be used to schedule the workflow of the feature extraction module and the fusion decision module, for example, to control the timing of feature extraction, to coordinate the sequence of multi-dimensional feature fusion, etc.

[0227] In a possible implementation, the communication unit 502 is further configured to establish a data transmission link with an external sensor (such as a visual sensor, a depth sensor, an infrared sensor, or a radar sensor), and acquire multi-dimensional perception data in real time. The processing unit 501 is further configured to pre-process the received multi-dimensional data, such as denoising and format conversion, to ensure that the data meets the input requirements of the feature extraction module.

[0228] The processing unit 501 can be a processor or a controller, and the communication unit 502 can be a communication interface, a transceiver, a transceiver, a transceiver circuit, a transceiver device, or the like. The communication interface is a general term, and can include one or more interfaces. The storage unit 503 can be a memory. When the two-dimensional code intelligent scanning device 50 based on multi-modal perception is a chip, the processing unit 501 can be a processor or a controller, the communication unit 502 can be an input interface and / or an output interface, a pin or a circuit, and the like. The storage unit 503 can be a storage unit (for example, a register, a cache, or the like) in the chip, or a storage unit (for example, a read-only memory (ROM), a random access memory (RAM), or the like) located outside the chip.

[0229] The communication unit can also be referred to as a transceiving unit. The antenna and control circuit with transceiving function in the two-dimensional code intelligent scanning device 50 based on multi-modal perception can be regarded as the communication unit 502 of the two-dimensional code intelligent scanning device 50 based on multi-modal perception, and the processor with processing function can be regarded as the processing unit 501 of the two-dimensional code intelligent scanning device 50 based on multi-modal perception. Optionally, the device for realizing the receiving function in the communication unit 502 can be regarded as a communication unit, and the communication unit is configured to perform the receiving steps in the embodiments of the present application, and the communication unit can be a receiver, a receiver, a receiving circuit, or the like. The device for realizing the sending function in the communication unit 502 can be regarded as a sending unit, and the sending unit is configured to perform the sending steps in the embodiments of the present application, and the sending unit can be a transmitter, a sender, a sending circuit, or the like.

[0230] Figure 5The integrated units in the above embodiments can be stored in a computer readable storage medium if the integrated units are implemented in the form of software function modules and sold or used as independent products. Based on such an understanding, the technical solutions of the embodiments of the present application essentially or in other words the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods described in the embodiments of the present application. The storage medium storing the computer software product includes a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.

[0231] Figure 5 The units in the above embodiments can also be referred to as modules, for example, the processing unit can be referred to as a processing module.

[0232] The embodiments of the present application also provide a hardware structure schematic diagram of a two-dimensional code intelligent scanning method device based on multi-modal perception (referred to as a two-dimensional code intelligent scanning device 60 based on multi-modal perception), which is shown in Figure 6 The two-dimensional code intelligent scanning device 60 based on multi-modal perception includes a processor 601, and optionally further includes a memory 602 connected with the processor 601.

[0233] In a first possible implementation manner, referring to Figure 6 The two-dimensional code intelligent scanning device 60 based on multi-modal perception further includes a transceiver 603. The processor 601, the memory 602 and the transceiver 603 are connected through a bus. The transceiver 603 is used for communicating with other devices or communication networks. Optionally, the transceiver 603 can include a transmitter and a receiver. The device for realizing the receiving function in the transceiver 603 can be regarded as a receiver, and the receiver is used for executing the receiving steps in the embodiments of the present application. The device for realizing the sending function in the transceiver 603 can be regarded as a transmitter, and the transmitter is used for executing the sending steps in the embodiments of the present application.

[0234] Based on the first possible implementation manner, Figure 6 The structure schematic diagram shown in the above can be used to show the structure of the two-dimensional code intelligent scanning device based on multi-modal perception involved in the above embodiments.

[0235] Among them, Figure 6 The system chip in the two-dimensional code intelligent scanning device based on multi-modal perception can also be shown. In this case, the actions performed by the two-dimensional code intelligent scanning device based on multi-modal perception described above can be realized by the system chip, and the specific actions performed can be referred to in the above, which will not be described here.

[0236] In the implementation process, the steps in the method provided by the embodiments can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software. The steps of the method disclosed by the embodiments of the present application can be directly embodied as hardware processor execution completion, or execution completion by a combination of hardware and software modules in the processor.

[0237] The processor in the present application can include but is not limited to at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or various types of computing devices running software, such as artificial intelligence processors, each of which can include one or more cores for executing software instructions to perform operations or processing. The processor can be a separate semiconductor chip, or can be integrated with other circuits as a semiconductor chip, for example, it can form a SoC (system on chip) with other circuits such as coding and decoding circuits, hardware acceleration circuits, or various bus and interface circuits, or it can be integrated as a built-in processor in an ASIC. The ASIC that integrates the processor can be packaged separately or packaged together with other circuits. In addition to including cores for executing software instructions to perform operations or processing, the processor can further include necessary hardware accelerators, such as field programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement special logic operations.

[0238] The memory in the embodiments of the present application can include at least one of the following types: read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, and electrically erasable programmable read-only memory (EEPROM). In some scenarios, the memory can also be a compact disc read-only memory (CDROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to this.

[0239] The embodiment of the present application further provides a computer readable storage medium, comprising instructions which, when executed on a computer, cause the computer to perform any of the above methods.

[0240] The embodiment of the present application further provides a computer program product comprising instructions which, when executed on a computer, cause the computer to perform any of the above methods.

[0241] The embodiment of the present application further provides a chip, comprising a processor and an interface circuit, wherein the interface circuit is coupled with the processor, the processor is configured to execute a computer program or instructions to implement the above method, and the interface circuit is configured to communicate with other modules outside the chip.

[0242] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or data storage device including one or more servers, data centers, etc. integrated with the medium. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (solid state disk, SSD)) and the like.

[0243] Although the present application is described herein in conjunction with various embodiments, other variations of the disclosed embodiments can be understood and implemented by those skilled in the art through viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. Some measures described in mutually different dependent claims can be combined and produce good results.

[0244] Although the application has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the scope of the application. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation, as it should be understood that various modifications and equivalents can be used without departing from the spirit and scope of the application. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A QR code intelligent scanning method based on multimodal perception, characterized in that, include: The system collects visual images, depth images, infrared reflectance intensity maps, and radar point cloud data of the object to be scanned to obtain multidimensional data. Multidimensional features are obtained by extracting features from multidimensional data. These multidimensional features include texture features extracted from visual images, curvature features calculated from infrared reflectance intensity maps and depth images, and normal vector features calculated from radar point cloud data. Multidimensional features are input into a Dempster-Shafer evidence theory fusion unit, which outputs confidence scores. The fusion unit is used to perform fusion calculations on multidimensional features based on Dempster-Shafer evidence theory to obtain confidence scores. The decision on the scanning mode is based on the confidence level of the object to be scanned; the scanning modes include direct scanning mode, reflection suppression mode, occlusion compensation mode, and low-light enhancement mode; wherein... The decision-making process for the scanning mode of the object to be scanned based on credibility includes: If the confidence level is greater than T1, then execute the direct scan mode; If T2 ≤ confidence level ≤ T1, then pattern matching is performed based on multidimensional features, including: When the curvature feature of the target region exists K max >K th At that time, the reflection suppression mode is activated; K max K represents the maximum curvature eigenvalue. th Indicates the preset curvature threshold; When the peak ratio of the normal vector histogram of the target region is <H th At that time, the occlusion compensation mode is executed; the normal vector histogram is obtained based on the statistical analysis of the normal vectors of all points, H th Indicates the preset peak-to-peak ratio threshold; When the texture features of the target region exist F t_max <F t_th At that time, low-light enhancement mode is executed; F t_max F represents the maximum texture feature value. t_th This indicates a preset texture feature threshold; If the credibility is less than T2, then the manual verification mode is triggered; Wherein, the target area represents the QR code scanning area, which is obtained by edge detection or image recognition of the image through a visual sensor; T1 represents a preset first threshold, and T2 represents a preset second threshold; The automatic execution operation of the reflection suppression mode includes: turning off the light of the vision sensor, increasing the infrared emission power to a preset value, and deflecting the scanning angle by a preset angle along the direction of the curvature feature change rate. The automatic execution operation of the occlusion compensation mode includes: controlling the radar sensor to perform point cloud scanning within a preset pitch angle range; The automatic execution of the low-light enhancement mode includes: activating the light of the vision sensor and activating infrared auxiliary illumination.

2. The QR code intelligent scanning method based on multimodal perception according to claim 1, characterized in that, The methods for obtaining the curvature features include: Calculation of infrared reflectance intensity diagram I ir gradient magnitude and the eigenvalues ​​λ1 and λ2 of the Hessian matrix; Based on gradient magnitude The curvature feature K is calculated using the Hessian matrix eigenvalues ​​λ1 and λ2 and the depth map D(x,y) according to the curvature feature calculation formula.

3. The QR code intelligent scanning method based on multimodal perception according to claim 2, characterized in that, The curvature characteristic calculation formula satisfies: Among them, S pix represents the physical size of a pixel, and μ represents the depth weighting coefficient.

4. The intelligent QR code scanning method based on multimodal perception according to claim 1, characterized in that, The methods for obtaining the texture features include: The visual image is divided into blocks to obtain multiple image blocks; Extract the local binary pattern (LBP) feature vector for each image patch; Calculate the statistical features within each image patch, including the contrast, entropy, and energy of the image patch's gray-level co-occurrence matrix; By combining LBP feature vectors with statistical features, texture features of visual images can be obtained.

5. The QR code intelligent scanning method based on multimodal perception according to claim 1, characterized in that, The methods for obtaining the normal vector features include: Build a KD Tree index for each point in the radar point cloud data; Based on point p i The set of neighboring points N(p) within the spatial location search radius r i ); Calculate the neighborhood point set N(p) i The covariance matrix of the covariance matrix is ​​used, and the eigenvector corresponding to the smallest eigenvalue in the covariance matrix is ​​taken as the point p. i The normal vector; By statistically analyzing the histogram of the normal vector direction for each point, the characteristics of the normal vector can be obtained.

6. The intelligent QR code scanning method based on multimodal perception according to claim 1, characterized in that, The input of multidimensional features into the Dempster-Shafer evidence theory fusion processor includes: Construct a recognition framework Θ={m1="decodeable",m2="requires enhancement",m3="undecodeable"}; where the recognition framework is used to determine the category to which the QR code scanning result of the object to be scanned belongs; Calculate texture features F respectively t Curvature feature F k , normal vector feature F n The basic probability m i (A); where the basic probability is represented in feature F i∈{t,k,n} Under the given conditions, determine the confidence level of the QR code scanning result of the object to be scanned to belong to category A∈Θ in the recognition framework; Based on the evidence combination rules of Dempster-Shafer evidence theory, multi-dimensional features are fused to obtain the fused basic probability m. total (A); where, the fusion basic probability represents the comprehensive confidence level of determining whether the QR code scanning result of the object to be scanned belongs to category A after integrating the basic probabilities of multidimensional features; The credibility level Bel=m is calculated according to the formula. total (m1)+λ×m total (m2); where λ represents the weighting coefficient.

7. The QR code intelligent scanning method based on multimodal perception according to claim 6, characterized in that, The basic probability m i The formula for (A) is: ;in, This indicates that feature i is the preset baseline feature value of category A, and σ represents the scaling parameter; The fusion basic probability m total The formula for (A) is: ;in, To represent the empty set, The symbol represents the intersection.

8. A QR code intelligent scanning system based on multimodal perception, characterized in that, include: The data acquisition module is used to acquire visual images, depth images, infrared reflectance intensity maps, and radar point cloud data of the object to be scanned, thereby obtaining multi-dimensional data. The feature extraction module is used to extract features from multidimensional data to obtain multidimensional features, which include texture features extracted based on visual images, curvature features calculated based on infrared reflectance intensity maps and depth images, and normal vector features calculated based on radar point cloud data. The fusion decision module is used to input multi-dimensional features into the Dempster Shafer evidence theory fusion unit and output the confidence level; the fusion unit is used to perform fusion calculation on multi-dimensional features based on Dempster Shafer evidence theory. The decision on the scanning mode is based on the confidence level of the object to be scanned; the scanning modes include direct scanning mode, reflection suppression mode, occlusion compensation mode, and low-light enhancement mode; wherein... The decision-making process for the scanning mode of the object to be scanned based on credibility includes: If the confidence level is greater than T1, then execute the direct scan mode; If T2 ≤ confidence level ≤ T1, then pattern matching is performed based on multidimensional features, including: When the curvature feature of the target region exists K max >K th At that time, the reflection suppression mode is activated; K max K represents the maximum curvature eigenvalue. th Indicates the preset curvature threshold; When the peak ratio of the normal vector histogram of the target region is <H th At that time, the occlusion compensation mode is executed; the normal vector histogram is obtained based on the statistical analysis of the normal vectors of all points, H th Indicates the preset peak-to-peak ratio threshold; When the texture features of the target region exist F t_max <F t_th At that time, low-light enhancement mode is executed; F t_max F represents the maximum texture feature value. t_th This indicates a preset texture feature threshold; If the credibility is less than T2, then the manual verification mode is triggered; Wherein, the target area represents the QR code scanning area, which is obtained by edge detection or image recognition of the image through a visual sensor; T1 represents a preset first threshold, and T2 represents a preset second threshold; The automatic execution operation of the reflection suppression mode includes: turning off the light of the vision sensor, increasing the infrared emission power to a preset value, and deflecting the scanning angle by a preset angle along the direction of the curvature feature change rate. The automatic execution operation of the occlusion compensation mode includes: controlling the radar sensor to perform point cloud scanning within a preset pitch angle range; The automatic execution of the low-light enhancement mode includes: activating the light of the vision sensor and activating infrared auxiliary illumination.

Images