A visible light-infrared neuro-physical alignment visual semantic reconstruction and perception method and system for a warehouse robot

Abstract

The application discloses a visible light-infrared neuro-physical alignment visual semantic reconstruction and perception method and system for a warehouse body robot. The method comprises the following steps: synchronously collecting an RGB visible light image and an infrared image of a warehouse scene, and completing space registration; constructing an asymmetric double-flow feature extraction architecture, and respectively extracting semantic features of the visible light image and physical structure features of the infrared image; performing neuro-physical orthogonal fusion on the physical structure features, and retaining complementary structures orthogonal to the semantic features; taking the RGB semantic features as a query, taking the complementary structure features as keys and values, performing cross-modal structure injection, and obtaining structure injection features; calculating a dynamic inhibition conductance based on RGB semantic uncertainty, performing Shunting divisive inhibition fusion, and obtaining final fusion features; based on the final fusion features, outputting a warehouse perception result through a VLM guided closed set detection head; and converting the warehouse perception result into a warehouse business interface message, and driving a robot to operate.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligent visual perception technology in warehousing scenarios, and more specifically to a visible light-infrared neurophysical alignment visual semantic reconstruction and perception method and system for embodied warehousing robots. Background Technology

[0002] The first approach to existing visual perception technologies for warehouse robots is based on target detection and semantic understanding using pure RGB images. This type of method typically utilizes convolutional neural networks, Transformer visual backbones, or single-stage detectors to identify containers, pallets, shelves, personnel, forklifts, and aisle obstacles from images taken from onboard cameras. In warehouse areas with stable lighting, unobstructed targets, and clear packaging textures, the pure RGB solution offers advantages such as low hardware cost and simple deployment. However, when robots enter dark warehouses, narrow aisle shelves, reflective areas of plastic wrap, or experience motion blur due to high-speed turns, the texture, color, and edge information in the RGB image simultaneously degrades, causing a shift between visual features and the boundaries of real objects.

[0003] With the development of visual language models, open-vocabulary object detection technology has significantly improved general object detection capabilities by establishing a unified semantic space between visual regions and linguistic descriptions. Typical techniques utilize text encoders like CLIP to map category names to text embeddings, and then perform similarity matching between visual region features and text embeddings. Approaches such as GLIP, Grounding DINO, and YOLO-World transform the detection problem into region-text alignment or phrase localization. In warehousing scenarios, this technology allows the system to write business objects such as "pallets," "boxes," "personnel," "forklifts," and "shelves" into the target vocabulary, reducing the cost of retraining the complete classifier every time a new packaging appearance or business category is added.

[0004] However, the paper's introduction clearly points out that the effectiveness of visual language models highly depends on the stable correlation between visual region features and pre-computed text embeddings. Under conditions of low light, fog, strong reflection, and motion blur, RGB features degrade severely, making the region-text alignment relationship fragile. In these situations, although the detector still retains the text semantic prior, it cannot extract reliable visual features from the damaged RGB image sufficient to match the text space, ultimately resulting in missed detection of goods, false detection of the background, decreased confidence in personnel targets, or drift of passageway obstacle boundaries.

[0005] To compensate for the limitations of RGB vision in harsh conditions, multispectral object detection technology incorporates infrared images as auxiliary information. Infrared images can still reflect the outlines of people, vehicles, and goods, as well as some thermal structures, even in low-light environments, and are therefore often used for nighttime pedestrian detection, RGB-IR fusion detection, and low-light warehouse inspection. Existing fusion methods include feature stitching, weighted summation, cross-modal attention, illumination-aware weights, cross-modal correction, and uncertainty weighting. These methods improve visibility from an engineering perspective, but most treat RGB and infrared as equivalent inputs, focusing on "fusing more information" rather than preserving the semantic manifold already learned by the visual language model.

[0006] Therefore, the ability to repair low-quality RGB features of infrared structures without contaminating the semantic manifold through orthogonal decomposition, cross-modal structure injection, and Shunting suppression is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a visible light-infrared neurophysical alignment visual semantic reconstruction and perception method and system for embodied warehouse robots. This solution is designed for warehouse material storage scenarios. It regards RGB images as the neural mainstream carrying VLM semantic priors and infrared images as the physical auxiliary flow that provides illumination-invariant structures. Through orthogonal decomposition, cross-modal structure injection and shunting suppression, the infrared structure can repair low-quality RGB features without contaminating the semantic manifold.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A visible-infrared neurophysical alignment visual semantic reconstruction and perception method for embodied warehouse robots includes: Simultaneously acquire RGB visible light image (Iv) and infrared image (Iir) of the warehouse scene, and complete spatial registration; An asymmetric dual-stream feature extraction architecture is constructed to extract the RGB semantic features Zv of the visible light image Iv and the physical structural features Zir of the infrared image Iir, respectively. The physical structure feature Zir is subjected to neural-physical orthogonal fusion, which projects the physical structure feature Zir onto the RGB semantic feature Zv. Overlapping components are removed by vector rejection, and the complementary structure Zir_perp, which is orthogonal to the RGB semantic feature, is retained. Using RGB semantic feature Zv as the query vector and complementary structural feature Zir_perp as the key and value vectors, cross-modal structural injection is performed to obtain the structural injection feature Zinj. Dynamically suppressed conductance is calculated based on RGB semantic uncertainty, and Shunting suppression fusion is performed to obtain the final fused feature Zf. Based on the final fusion feature Zf, the closed-set detection head is guided by VLM to output warehouse perception results; The results of warehouse perception are converted into warehouse business interface messages to drive robot operations.

[0009] Preferably, when extracting the RGB semantic feature Zv, a VLM-aligned semantic expert module is set in the RGB semantic mainstream, pre-trained weights are kept unchanged or only the lightweight adaptation layer is fine-tuned, and the semantic response L of the visual backbone output is weighted, reconstructed, and normalized using a learnable prototype matrix P to obtain the RGB semantic feature Zv, as shown in the following formula: .

[0010] Where L represents the region-level or pixel-level semantic response tensor of the visual backbone output, P represents the learnable prototype matrix, Softmax(·) represents the normalized weight assignment operator, LayerNorm(·) represents the layer normalization operator, and Zv represents the output RGB semantic features.

[0011] Preferably, when extracting physical structure features (Zir), gradient injection is performed in the infrared physical auxiliary flow, and the horizontal and vertical gradients are extracted using a fixed Sobel operator to construct a nonparametric physical structure prior, as shown in the following formula: ; ; Where I_ir represents the input infrared image, M_grad represents the gradient magnitude map calculated by the Sobel operator, and I_enh represents the enhanced infrared image after injecting gradient prior; S_x and S_y are the horizontal and vertical Sobel operators, respectively, used to extract directional edges; epsilon is the stabilization term, lambda is the injection intensity coefficient, and Repeat(·) represents the expansion operation that copies the single-channel gradient magnitude to the same number of infrared input channels; The enhanced infrared image is input into a lightweight encoder to generate physical structure features (Zir).

[0012] Preferably, the formula for the neuro-physical orthogonal fusion is as follows: ; The orthogonalized Zir_perp represents the orthogonal infrared complementary structure feature after removing redundant semantic components; Where, <·,·> represents the inner product, ||·||_2 represents the L2 norm, epsilon is the stability term, and Proj_Z_v(Z_ir) represents the projection component of the infrared physical structure feature Z_ir onto the semantic basis spanned by the RGB semantic feature Z_v.

[0013] Preferably, the process of performing cross-modal structure injection includes: Using RGB semantic features Zv as the query vector and orthogonal infrared complementary structural features Zir_perp as the key and value vectors, the injection process employs a dual residual structure of multi-head cross-attention and a feedforward network, as shown in the following formula:

[0014] The intermediate state is used to obtain the structural injection feature Zinj through a nonlinear feedforward network and residual connections; Where W_q, W_k, and W_v are linear mapping matrices of query, key, and value, respectively; Q, K, and V are the projected query, key, and value, respectively; d_k represents the dimension of the key vector; Softmax(·) represents the attention normalization function; MHA(·) represents multi-head cross-attention; FFN(·) represents the feedforward network; LayerNorm(·) or LN(·) represents the layer normalization operation; Z_tilde_v represents the intermediate semantic state after cross-modal attention; and Zmid represents the intermediate state.

[0015] Preferably, the process of performing Shunting desiccation suppression fusion includes: The dynamic suppression conductance is calculated based on the RGB semantic uncertainty, and the formula is as follows: ; ; Where g_leak is the leakage conductance, g_inh is the dynamic suppression conductance, lambda_s is the suppression scaling factor, W_gate is the uncertainty mapping weight, and sigma(·) is the Sigmoid gate function.

[0016] Preferably, the warehouse sensing results output by the VLM-guided closed-set detection head include: The detection head includes both a decoupled classification branch and a bounding box regression branch. During inference, non-maximum suppression is applied to the candidate boxes, and the target box coordinates, category, confidence score, target source modality contribution, and low-confidence region markers are output, as shown in the following formula: ; Where S represents the category score of each candidate region to the target vocabulary, T^T represents the transpose of the category text matrix, tau is the temperature scaling factor, and cos(·,·) or the normalized inner product is used to measure the similarity between the final fused features and the embeddings of each category text.

[0017] Preferably, a visible light-infrared neurophysical alignment visual semantic reconstruction and perception system for embodied warehouse robots includes: A multispectral acquisition module is used to simultaneously acquire and register RGB visible light and infrared images of the warehouse scene; The VLM text semantic library module is used to store and update the category text embeddings corresponding to the warehouse target vocabulary, providing text semantic priors for the RGB semantic mainstream module and the VLM guided detection head; The RGB semantic mainstream module is used to extract semantic features aligned with the VLM semantic space; The infrared physical-assisted flow module is used to extract physical structure features that are invariant to illumination and to perform gradient enhancement; The neuro-physical orthogonal fusion module performs neuro-physical orthogonal fusion on the physical structure feature Zir, projects the physical structure feature Zir onto the RGB semantic feature Zv, and removes overlapping components through vector rejection, retaining the complementary structure Zir_perp that is orthogonal to the RGB semantic feature; The cross-modal structure injection and shunting fusion module is used to perform cross-modal structure injection with RGB semantic feature Zv as query vector and complementary structure feature Zir_perp as key vector and value vector to obtain structure injection feature Zinj, and to complete shunting suppression fusion based on RGB semantic uncertainty to output the final fused feature Zf. The VLM-guided detection head outputs warehouse perception results based on the final fused feature Zf. The warehouse task output interface is used to convert the output warehouse perception results into standardized messages to drive robot operations.

[0018] As can be seen from the above technical solutions, compared with the prior art, the present invention provides a visible light-infrared neurophysical alignment visual semantic reconstruction and perception method and system for embodied warehouse robots. The present invention focuses on the visual semantic reconstruction and perception problem under harsh working conditions of warehouse materials, and adopts an asymmetric neurophysical alignment architecture to unify the mainstream RGB semantics, infrared physical auxiliary stream and VLM text semantic library into a deployable detection system, which has at least the following beneficial effects.

[0019] 1. Preserve VLM semantic priors and reduce feature contamination. By using the RGB branch as the semantic mainstream and freezing or weakly updating its pre-trained representation, this invention avoids the infrared thermal response directly rewriting the VLM semantic space; by projecting visual features onto a latent space of the same dimension as the text embedding through a prototype adapter, the storage target category can continue to be stably distinguished through text semantics.

[0020] 2. Utilizing infrared structure to compensate for the lack of low-light information. Infrared physical aid flow reinforces the edges of containers, pallets, people, and obstacles through fixed Sobel gradient injection, providing usable structural evidence even in low-light, dark warehouses, shaded shelves, and foggy / dusty environments. In the experiments presented in this paper, this physical structure encoding resulted in a higher mAP compared to the original infrared input and achieved better performance than learnable edge filtering with fewer parameters.

[0021] 3. Complementary information injection is achieved through orthogonal decomposition. NPFusion first performs vector rejection on infrared features, retaining the unique physical structure orthogonal to the RGB semantic basis, reducing redundant information and thermal noise entering the semantic space, making the fusion closer to "gap filling" rather than "stack".

[0022] 4. Enhanced robustness against low signal-to-noise ratio through a shunting suppression mechanism. This invention uses visual uncertainty estimation to suppress conductance, performing degenerate suppression on degraded semantic components when RGB noise is strong, allowing the infrared structure to play a compensating role in the numerator term, while avoiding linear amplification of noise. Component ablation results show that NPFusion achieves better accuracy-delay compromise in the nighttime subset.

[0023] 5. Improve the detection accuracy of key targets in warehouses. In the experiments in the paper, NPSA achieved an mAP50 of 87.1% on the M3FD dataset and 97.2% on the LLVIP low-light dataset; the system ablation gradually improved from 90.5% in the basic RGB semantic flow to 97.2% in the complete model, indicating that each module makes a clear contribution to robust detection.

[0024] 6. It possesses advantages in edge deployment and engineering integration. The infrared branch adopts lightweight structure coding, gradient injection is a deterministic operator, and the detection head uses pre-computed text embedding, making it suitable for deployment in the edge computing unit of warehouse robots. It can also output standardized target information to WMS, RCS, navigation and obstacle avoidance, grasping control, and safety interlocking systems.

[0025] 7. Improve the continuity and safety of warehousing operations. In scenarios such as dark warehouses, reflective packaging, obscured stacking, and fog and dust, this invention can reduce missed material inspections, background mis-inspections, and human misjudgments, and reduce the risks of AMR emergency stops, detour failures, pallet recognition errors, and robotic arm grasping deviations. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the overall architecture of the present invention.

[0028] Figure 2 This is a schematic diagram illustrating the mainstream semantics of RGB provided by the present invention.

[0029] Figure 3 A schematic diagram of the infrared physical auxiliary flow provided by the present invention.

[0030] Figure 4 This is a schematic diagram of the neuro-physical orthogonal fusion provided by the present invention.

[0031] Figure 5 This is a schematic diagram of the closed-loop output of the warehousing task provided by the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] like Figure 1 As shown, this embodiment of the invention is based on RGB and infrared synchronous input, and outputs the detection and decision information required for warehousing tasks through the semantic mainstream, physical auxiliary stream, and NPFusion module. This embodiment of the invention discloses a visible light-infrared neurophysical alignment visual semantic reconstruction and perception method for equivocal warehousing robots, including: Simultaneously acquire RGB visible light image (Iv) and infrared image (Iir) of the warehouse scene, and complete spatial registration; An asymmetric dual-stream feature extraction architecture is constructed to extract the RGB semantic features Zv of the visible light image Iv and the physical structural features Zir of the infrared image Iir, respectively. The physical structure feature Zir is subjected to neural-physical orthogonal fusion, which projects the physical structure feature Zir onto the RGB semantic feature Zv. Overlapping components are removed by vector rejection, and the complementary structure Zir_perp, which is orthogonal to the RGB semantic feature, is retained. Using RGB semantic features as the query vector and complementary structural features Zir_perp as the key and value vectors, cross-modal structural injection is performed to obtain the structural injection feature Zinj. Dynamically suppressed conductance is calculated based on RGB semantic uncertainty, and Shunting suppression fusion is performed to obtain the final fused feature Zf. Based on the final fusion feature Zf, the closed-set detection head is guided by VLM to output warehouse perception results; The results of warehouse perception are converted into warehouse business interface messages to drive robot operations.

[0034] In a specific embodiment of the present invention, such as Figure 2 As shown, the mainstream RGB semantic model maps damaged RGB images into semantic features compatible with warehouse category text embeddings by freezing the semantic backbone and prototype adapter, establishing a multispectral input for warehouse material scenarios. During operation, the robot simultaneously acquires RGB visible light images (Iv) and infrared images (Iir), and spatially registers them based on camera intrinsic and extrinsic parameters, timestamps, and resolution. Target categories can be pre-defined according to warehouse operations as boxes, pallets, personnel, forklifts, shelves, aisle obstacles, lighting fixtures, or other objects requiring monitoring.

[0035]

[0036] Where S is the region-text matching score vector, sigma(·) is the similarity normalization function, F_vis is the candidate region visual feature matrix, T is the category text matrix, T^T represents the transpose of the category text matrix, and tau is the temperature scaling factor.

[0037] The target category embedding matrix is ​​obtained by freezing the text encoder, the visual region features are output by the detection network, and the temperature scaling factor is used to adjust the category similarity distribution. This formula reflects the foundation of VLM detection: visual features must be consistent with the text semantic space.

[0038] An asymmetric two-stream feature extraction method is constructed. The RGB semantic mainstream is responsible for preserving the semantic prior of the VLM, while the infrared physical auxiliary stream is responsible for providing illumination-invariant edges, contours, and thermal structures. The two are not concatenated with equal weight, but are functionally distinguished as the semantic mainstream and the physical auxiliary stream.

[0039]

[0040] Where Psi_sem(I_v;theta_v) represents the RGB semantic mainstream encoder, Phi_str(I_ir;theta_ir) represents the infrared physical auxiliary stream encoder, I_v represents the RGB input image, I_ir represents the infrared input image, theta_v and theta_ir represent the network parameters of the two branches respectively, and Z_v and Z_ir represent the output RGB semantic features and infrared physical structure features respectively.

[0041] Specifically, when extracting the RGB semantic feature Zv, a VLM-aligned semantic expert module is set in the RGB semantic mainstream. The pre-trained weights are kept unchanged or only the lightweight adaptation layer is fine-tuned. The semantic response L of the visual backbone output is weighted, reconstructed, and normalized using a learnable prototype matrix P to obtain the RGB semantic feature Zv, as shown in the following formula: .

[0042] Where L represents the region-level or pixel-level semantic response tensor of the visual backbone output, P represents the learnable prototype matrix, Softmax(·) represents the normalized weight assignment operator, LayerNorm(·) represents the layer normalization operator, and Zv represents the output RGB semantic features.

[0043] In one specific embodiment of the present invention, a VLM alignment semantic expert module is set in the RGB semantic mainstream. This expert can use SegFormer, CLIP visual backbone, SigLIP visual backbone, or other hierarchical Transformer as a visual encoder.

[0044] The aforementioned prototype matrix P can be set according to the category of warehouse target, background category, and suspected obstacle category, or it can be learned from training data. Its function is to pull the RGB region features back into a latent space isomorphic to the text embedding, preventing the detection head from being unable to interpret the fused features after subsequent infrared information injection.

[0045] Specifically, when extracting physical structure features (Zir), gradient injection is performed in the infrared physical auxiliary flow. A fixed Sobel operator is used to extract horizontal and vertical gradients, and a nonparametric physical structure prior is constructed, as shown in the following formula: ; ; Where I_ir represents the input infrared image, M_grad represents the gradient magnitude map calculated by the Sobel operator, and I_enh represents the enhanced infrared image after injecting gradient priors; S_x and S_y are the horizontal and vertical Sobel operators, respectively, used to extract directional edges; epsilon is a stabilization term used to avoid numerical anomalies caused by an excessively small denominator; lambda is the injection intensity coefficient used to control the degree of enhancement of the infrared input by the edge priors; Repeat(·) represents the expansion operation that copies the single-channel gradient magnitude to the same number of infrared input channels.

[0046] The enhanced infrared image is input into a lightweight encoder to generate physical structure features (Zir).

[0047] In one specific embodiment of the present invention, such as Figure 3As shown, the infrared-assisted flow uses a fixed gradient prior to highlight the boundaries of the goods, making pallet holes, cargo edges, and personnel contours, which are difficult to identify by RGB under low illumination, usable structural cues. Specifically, when performing gradient injection in the infrared physical-assisted flow, the infrared image still retains the thermal contours of objects under low illumination, but the original intensity distribution may be affected by thermal crossover and equipment noise. This invention uses a fixed Sobel operator to extract horizontal and vertical gradients, constructing a non-parametric physical structure prior. The enhanced infrared input is then processed by a lightweight ResNet or equivalent structure encoder to generate multi-scale physical features for subsequent fusion.

[0048] Specifically, the formula for the neuro-physical orthogonal fusion is as follows: ; The orthogonalized Zir_perp represents missing or unreliable structural information in RGB, such as the thermal outline of people in dark passages, the pallet boundary obscured by fog and dust, and the outline of real cargo boxes next to reflective packaging.

[0049] Where, <·,·> represents the inner product, ||·||_2 represents the L2 norm, epsilon is the stability term, and Proj_Z_v(Z_ir) represents the projection component of the infrared physical structure feature Z_ir onto the semantic basis spanned by the RGB semantic feature Z_v; in the current formula notation, Zir or Z_ir represents the infrared physical structure feature, Zv or Z_v represents the RGB semantic feature, and Zir_perp or Z_ir_perp represents the orthogonal infrared complementary structure after removing redundant semantic components.

[0050] In a specific embodiment of the present invention, such as Figure 4 As shown, NPFusion processes infrared structures in three stages: orthogonal decomposition, structural injection, and shunting suppression. This allows infrared structures to assist the mainstream of RGB semantics in a controlled manner and perform neuro-physical orthogonal fusion. To avoid infrared features directly polluting the VLM semantic space, this embodiment first projects infrared physical features onto the RGB semantic basis and removes overlapping components through vector rejection, retaining only complementary structures orthogonal to the RGB semantic features.

[0051] Specifically, the process of performing cross-modal structure injection includes: Using RGB semantic features Zv as the query vector and complementary structural features Zir_perp (orthogonal infrared structure) as the key and value vectors, the injection process employs a dual residual structure of multi-head cross-attention and a feedforward network, as shown in the following formula:

[0052] Where W_q, W_k, and W_v are linear mapping matrices of query, key, and value, respectively; Q, K, and V are the projected query, key, and value, respectively; d_k represents the dimension of the key vector; Softmax(·) represents the attention normalization function; Z_tilde_v represents the intermediate semantic state after cross-modal attention; and Z_inj represents the structure-injected feature after residual injection. In the current formula, Zv represents the query feature, and Zir_perp represents the orthogonal infrared complementary structure corresponding to the key and value.

[0053] The intermediate state is used to obtain the structural injection feature Zinj through a nonlinear feedforward network and residual connections.

[0054] In one specific embodiment of the present invention, cross-modal structural injection is performed. Using the RGB semantic feature Zv as the query vector and the complementary structural feature Zir_perp as the key and value vectors, the semantic mainstream queries physical structural details only when needed. This injection process can be implemented using a dual residual structure of multi-head cross-attention and a feedforward network.

[0055] Furthermore, the intermediate state is processed through a nonlinear feedforward network and residual connections to obtain the structure-injected feature Zinj. This design ensures that the infrared structure serves only as the basis for semantic repair, rather than as an independent source of semantic categories.

[0056] Specifically, the process of performing Shunting desaturation fusion includes: The dynamic suppression conductance is calculated based on the RGB semantic uncertainty, and the formula is as follows: ; ; Where Z_v or Zv represents RGB semantic features, Z_inj or Zinj represents structure injection features, Z_ir_perp or Zir_perp represents orthogonal infrared complementary structures, Z_f or Zf represents final fusion features; g_leak is leakage conductance, g_inh is dynamic suppression conductance, lambda_s is suppression scaling factor, W_gate is uncertainty mapping weight, sigma(·) is the Sigmoid gate function, ||·||_2 is the L2 norm, and epsilon is the stability term.

[0057] In one specific embodiment of the present invention, shunting suppression fusion is performed. Addressing the issue that RGB semantic noise may be amplified during the fusion process in low signal-to-noise ratio environments like warehouses, this embodiment introduces the concept of destructive suppression from bio-inspired neurodynamics, using visual uncertainty control to regulate suppression conductance. When low illumination, strong reflection, or blurring in the RGB image leads to increased uncertainty, the suppression conductance increases accordingly, suppressing degraded semantic components through destructive suppression. Simultaneously, the numerator term retains the orthogonal infrared structure and injected semantic information, achieving a balance between structural compensation and noise suppression.

[0058] Specifically, the warehouse perception results output by the VLM-guided closed-set detection head include: The detection head includes both a decoupled classification branch and a bounding box regression branch. During inference, non-maximum suppression is applied to the candidate boxes, and the target box coordinates, category, confidence score, target source modality contribution, and low-confidence region markers are output, as shown in the following formula: ; Where S represents the category score of each candidate region to the target vocabulary, Z_f or Zf represents the final fused feature, T is the category text embedding matrix, T^T represents the transpose of the category text matrix, tau is the temperature scaling factor, and cos(·,·) or normalized inner product is used to measure the similarity between the final fused feature and each category text embedding.

[0059] In a specific embodiment of the present invention, such as Figure 5 As shown, the NPSA output not only serves object detection but also forms a closed loop for warehouse scheduling, navigation, grasping, and security interlocking. Considering the relatively fixed warehouse categories but significant appearance variations, this embodiment does not use a conventional random initialization classifier. Instead, it pre-calculates the text embeddings of the target vocabulary and uses the temperature-scaled similarity between normalized visual features and text embeddings as the classification criterion. The detection head can simultaneously include a decoupled classification branch and a bounding box regression branch. During inference, non-maximum suppression is performed on candidate boxes, and the target box coordinates, category, confidence level, target source modality contribution, and low-confidence region markers are output.

[0060] Generate task interfaces for warehousing operations. Detection results (outputting warehousing perception results) can be converted into dynamic obstacle layers on the robot navigation map, candidate regions for robotic arm grasping poses, pallet / box inventory results, personnel intrusion alarms, and aisle congestion alerts. For low-confidence areas, the system can trigger robot deceleration, secondary observation, manual verification, or switch to a conservative obstacle avoidance strategy.

[0061] Specifically, a visible light-infrared neurophysical alignment visual semantic reconstruction and perception system for embodied warehouse robots includes: A multispectral acquisition module is used to simultaneously acquire and register RGB visible light and infrared images of the warehouse scene; The VLM text semantic library module is used to store and update the category text embeddings corresponding to the warehouse target vocabulary, providing text semantic priors for the RGB semantic mainstream module and the VLM guided detection head; The RGB semantic mainstream module is used to extract semantic features aligned with the VLM semantic space; The infrared physical-assisted flow module is used to extract physical structure features that are invariant to illumination and to perform gradient enhancement; The neuro-physical orthogonal fusion module performs neuro-physical orthogonal fusion on the physical structure feature Zir, projects the physical structure feature Zir onto the RGB semantic feature Zv, and removes overlapping components through vector rejection, retaining the complementary structure Zir_perp that is orthogonal to the RGB semantic feature Zv; The cross-modal structure injection and shunting fusion module is used to perform cross-modal structure injection with RGB semantic feature Zv as query vector and complementary structure feature Zir_perp as key vector and value vector to obtain structure injection feature Zinj, and to complete shunting suppression fusion based on RGB semantic uncertainty to output the final fused feature Zf. The VLM-guided detection head outputs warehouse perception results based on the final fused feature Zf. The warehouse task output interface is used to convert the output warehouse perception results into standardized messages to drive robot operations.

[0062] In one specific embodiment of the present invention, the system can be deployed in the edge computing unit of an AMR, AGV, automated forklift, warehouse inspection robot, or picking robotic arm. The system includes at least: a multispectral acquisition module, a VLM text semantic library module, an RGB semantic mainstream module, an infrared physical auxiliary stream module, a neuro-physical orthogonal fusion module, a VLM guided detection head, a warehouse task output interface, and an optional physical prior extension interface.

[0063] Furthermore, key handling strategies under warehousing conditions include: In darkened warehouses and shaded areas of shelves, RGB textures typically disappear while infrared outlines remain visible. This invention classifies such areas as regions of high visual uncertainty, improves the suppression conductivity during shunting fusion, and makes the detection results more dependent on orthogonal infrared structures. At the same time, it continues to use RGB semantic mainstreaming and text embedding to determine category names, avoiding misinterpreting thermal outlines as non-existent categories.

[0064] In scenarios involving densely stacked containers and pallet hole recognition, the boundaries of adjacent targets are often blurred due to occlusion or similar packaging. Infrared gradient injection can enhance contours that are not obvious in visible light, while cross-modal structure injection allows the mainstream RGB semantics to query infrared structures at suspected boundaries, thereby improving container separation, pallet positioning, and candidate region generation for robotic arm grasping.

[0065] In scenarios involving plastic wrapping, metal packaging, or highly reflective labels, RGB images may exhibit bright pseudo-textures, making it easy for traditional detectors to mistake reflective areas for targets. This invention removes redundant or conflicting physical responses from RGB images through orthogonal decomposition and reduces the semantic weight of reflective areas through noise gating, allowing the infrared structure and textual semantics to jointly constrain the final result.

[0066] When an AMR makes high-speed turns, stops abruptly, or traverses narrow alleyways, images exhibit motion blur and localized deformation. The system can perform post-processing by combining continuous frame confidence, infrared structure stability, and detection box change rate: if the same target is continuously present in the infrared structure and the RGB confidence drops briefly, the target trajectory is maintained; if the RGB confidence is high but the infrared structure is missing for a long period, the alarm level is reduced or a secondary observation is triggered.

[0067] In safety-related scenarios, this invention prioritizes the recall rate of personnel, forklifts, and obstacle courses. For candidate targets with low confidence but high risk, the results can be marked as targets requiring verification and sent to the navigation controller, enabling the robot to adopt strategies such as deceleration, detour, or brief stop.

[0068] Optionally, the mainstream RGB semantics can be SegFormer MiT-B2, CLIP-ViT, ConvNeXt, or other visual backbones; the infrared auxiliary stream can be ResNet-18, MobileNet, lightweight Transformer, or a dedicated infrared encoder; gradient injection can be replaced with a fixed Laplacian operator, oriented gradient groups, or multi-scale edge filtering. All of the above substitutions do not change the basic idea of ​​this invention to repair VLM semantics with orthogonal structures.

[0069] Optionally, when the AMR / AGV is equipped with a depth camera, LiDAR, or IMU, depth discontinuities, space occupancy, velocity vectors, and attitude stability can be included as additional physical priors, input into the physical aid flow along with infrared structural features. However, in drafting the claims, the neuro-physical fusion of RGB and infrared should be considered the core embodiment, with other sensors as subordinate extensions.

[0070] In one implementation, the training phase uses registered RGB-infrared repositories or publicly available multispectral data, with the input resolution set to 640. 640. The text encoder remains frozen, the visual backbone uses pre-trained weights, the optimizer can be AdamW, the learning rate can be set to 1e-4, the weight decay can be set to 0.05, the training cycle can be set to 100 epochs, and linear warm-up and cosine annealing are used. Data augmentation can include Mosaic, MixUp, low-light perturbation, occlusion simulation, and reflective area augmentation.

[0071] The optional joint optimization objective formula is as follows:

[0072] Wherein, L_total is the total loss, L_det is the detection loss, L_align is the text alignment loss (used to constrain the alignment of visual features with text embeddings), L_ortho is the orthogonal constraint loss (used to enhance the orthogonality between the infrared structure and the RGB semantic basis), L_phy is the infrared boundary consistency loss (used to constrain the consistency between the detection boundary and the infrared structure prior), and alpha, beta, and gamma are the weight coefficients of the three auxiliary loss terms. During deployment, only the trained RGB semantic main stream, infrared auxiliary stream, NPFusion, and detection head need to be loaded, and the target category text embeddings can be cached to reduce the computational load on the edge.

[0073] Table 1 shows a comparison of the detection results for the M3FD dataset: Table 1

[0074] As shown in Table 1, NPSA achieves an mAP50 of 87.1% on the M3FD dataset, which is better than the comparison methods, indicating that the present invention has strong cross-modal robustness in multi-class infrared-visible light cargo and personnel target detection.

[0075] The comparison of detection results for the LLVIP low-light dataset is shown in Table 2: Table 2

[0076] As shown in Table 2, NPSA achieves 97.2% mAP50 on the LLVIP low-light dataset, exceeding the existing RGB, IR, and RGB-IR fusion baselines, indicating that the present invention is suitable for scenarios such as nighttime warehousing, dark warehouses, and low-light inspections.

[0077] The system's stepwise ablation experiments are shown in Table 3: Table 3

[0078] Table 3 shows that from the basic RGB semantic stream to the complete NPSA system, mAP increased from 90.5% to 97.2%. Among them, VLM adaptation, gradient injection, structure injection, orthogonal decomposition, shunting suppression, and VLM detection head all brought considerable gains, proving that the various technical features are not arbitrarily spliced ​​together, but rather form a synergy around semantic alignment and physical compensation.

[0079] Component-level ablation experiments and accuracy-efficiency comparisons are shown in Table 4: Table 4

[0080] Table 4 shows that fixed gradient priors have better parameter efficiency than complex learnable or diffuse edge priors; NPFusion achieves higher AP with lower latency in the nighttime subset, making it suitable for real-time perception on the edge of warehouse robots.

[0081] Furthermore, the system output can be in the form of JSON, ROS messages, shared memory, or industrial bus messages. Each target must include at least the category, target bounding box, confidence level, timestamp, image coordinates, optional world coordinates, and low-confidence markers; for containers and pallets, the output can include edge orientation, center point, and accessibility status for gripping or handling; for personnel and forklifts, the output can include safe distance and alarm level.

[0082] When the detection confidence level falls below a threshold or the conflict between RGB and infrared structures exceeds a threshold, the system can perform actions such as slowing down, stopping, secondary acquisition, active supplemental lighting, switching sensors, or notifying manual verification. This mechanism makes this invention not only a model structure, but also a complete visual perception method and system that can be embedded into a holographic warehouse system.

[0083] Specifically, the following formulas are used to limit the target vocabulary, sample structure, infrared structure pyramid, orthogonal constraints, text alignment constraints, boundary consistency, and final output, without changing the aforementioned module flow.

[0084] Formula (1) Warehouse target vocabulary and text embedding:

[0085] This formula is used to define the target semantic library in closed-set repository detection. Here, V_target represents the repository target vocabulary set, c_k represents the name of the k-th target category, K represents the total number of target categories, E_txt(c_k) represents the original text embedding generated by the text encoder for the category name c_k, and T_k represents the normalized text embedding for the k-th category; the text embeddings are pre-computed and cached before inference.

[0086] Equation (2) Definition of multispectral input sample:

[0087] This formula defines each training sample as a set of registered RGB images, infrared images, and target annotations. Here, D represents the training dataset, n represents the sample index, N represents the total number of samples, I_v^n and I_ir^n represent the RGB and infrared images of the nth sample, respectively, Y^n represents the annotation set of the nth sample, b_j represents the bounding box of the jth target, c_j represents the category identifier of the target, and M_n represents the number of targets in the nth sample.

[0088] Equation (3) Infrared multi-scale structure pyramid:

[0089] This formula represents the physical structural features of the infrared-assisted stream output at multiple scales. Here, P_ir represents the infrared multi-scale structural pyramid, l represents the scale level index, Z_ir^(l) represents the infrared structural feature map of the l-th layer, Phi_str(·) represents the infrared physical-assisted stream encoder, and I_enh represents the enhanced infrared input. This structure is used to adapt to different sizes of containers, pallet holes, personnel, and obstacles.

[0090] Equation (4) Infrared redundant projection term:

[0091] This expression explicitly represents the infrared components that coincide with the RGB semantic basis. Here, Z_ir_parallel represents the redundant projection of the infrared features onto the RGB semantic basis, Z_ir represents the infrared physical structure features, Z_v represents the RGB semantic features, <·,·> represents the inner product, ||·||_2 represents the L2 norm, and epsilon represents the stable term; this redundant term is subsequently removed using vector rejection.

[0092] Equation (5) Structural injection residual output:

[0093] This formula represents the semantic state after cross-modal attention being processed by a feedforward network and residual connections to form structure-injected features. Here, Z_tilde_v represents the intermediate semantic state output by cross-modal attention, Z_inj represents the structure-injected features, FFN(·) represents the feedforward network, and LN(·) represents the layer normalization operation.

[0094] Equation (6) Visual uncertainty estimation:

[0095] This formula is used to estimate the classification uncertainty of the RGB semantic mainstream in local regions. Here, U_v represents visual uncertainty, S represents the class score vector, Softmax(S)_k represents the normalized probability of the k-th class, and max_k represents taking the maximum probability among all classes; the higher the uncertainty, the stronger the shunting suppression.

[0096] Equation (7) Orthogonal constraint loss:

[0097] This formula is used to constrain the preserved infrared structure and RGB semantic features Z_v to be as orthogonal as possible, reducing redundant structures and thermal noise contamination. Here, L_ortho represents the orthogonality constraint loss, mean(·) represents averaging over all locations or samples, |·| represents the absolute value, Z_ir_perp represents the orthogonal infrared complementary structure, Z_v represents the RGB semantic features, <·,·> represents the inner product, ||·||_2 represents the L2 norm, and epsilon represents the stabilization term.

[0098] Equation (8) Text alignment loss:

[0099] This formula is used to enhance the similarity between the fused features and the text embeddings of the real storage categories, while also increasing the distance between them and the text embeddings of other categories. Here, L_align represents the text alignment loss, Z_f represents the final fused feature, T_y represents the text embedding corresponding to the real category, T_k represents the text embedding of the k-th category, sim(·,·) represents the similarity function, tau represents the temperature scaling factor, sum_k represents the summation over all categories, and mean(·) and log(·) represent the average and logarithmic operations, respectively.

[0100] Equation (9) Infrared boundary consistency constraint:

[0101] This formula is used to constrain the predicted target boundary to be consistent with the infrared gradient structure, making the cargo outline more stable under low illumination. Here, L_edge represents the boundary consistency loss, B_pred represents the predicted boundary or predicted box edge map, Grad(·) represents the boundary gradient operator, Norm(·) represents the normalization operator, M_grad represents the infrared gradient magnitude map, and ||·||_1 represents the L1 norm.

[0102] Equation (10) Final category and bounding box output:

[0103] This expression indicates that the detection head determines the category based on text similarity and outputs the target bounding box through the regression branch, which can be further converted into a warehouse navigation, crawling, or security control interface. Here, c_hat represents the predicted category, argmax_k represents the category index corresponding to the maximum value among all category scores S_k, S_k represents the score of the k-th class, b_hat represents the predicted bounding box, R_head(·) represents the bounding box regression branch, and Z_f represents the final fused feature.

[0104] 9. Correspondence between formulas and protection points in claims

[0105] Equations (1) and (2) above correspond to the input construction steps and are used to define the data structure of the warehouse target vocabulary, text embedding cache, multispectral samples, and target annotations. This part can support the claim statement of "acquiring RGB images, infrared images, and target category text, and constructing multispectral sensing data".

[0106] Equations (3) to (5) correspond to the subordinate features of the infrared physical auxiliary flow and the NPFusion module, illustrating that the infrared structure is not simply spliced ​​into the RGB features, but rather participates in the fusion after undergoing multi-scale structure encoding, redundant projection recognition, orthogonal decomposition, and residual injection. This part is the key to distinguishing ordinary RGB-IR fusion schemes.

[0107] Equations (6) to (8) correspond to the training objectives of adaptive suppression and semantic alignment. Visual uncertainty is used to adjust the suppression intensity in low signal-to-noise ratio regions, orthogonal constraints are used to prevent infrared thermal response from contaminating the mainstream RGB semantics, and text alignment constraints are used to maintain the consistency between fused features and warehouse category text embeddings.

[0108] Equation (9) can correspond to the boundary consistency subordination scheme under low illumination and occlusion conditions, and is used to constrain the predicted boundary to fit the infrared gradient structure. For storage targets such as pallet holes, cargo box edges, personnel outlines and passageway obstacles, this constraint can reduce boundary drift and local missed detections.

[0109] Equation (10) corresponds to the system output interface features, namely, the detection head outputs the category based on text similarity, and the regression branch outputs the bounding box, which is then further converted into navigation obstacle avoidance, capture and positioning, inventory recognition or security alarm messages. Thus, the mathematical expression of this invention can cover the complete link from input, feature extraction, fusion, training constraints to business output.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0111] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A visible light-infrared neurophysical alignment visual semantic reconstruction and perception method for embodied warehouse robots, characterized in that, include: Simultaneously acquire RGB visible light image (Iv) and infrared image (Iir) of the warehouse scene, and complete spatial registration; An asymmetric dual-stream feature extraction architecture is constructed to extract the RGB semantic features Zv of the visible light image Iv and the physical structural features Zir of the infrared image Iir, respectively. The physical structure feature Zir is subjected to neural-physical orthogonal fusion, which projects the physical structure feature Zir onto the RGB semantic feature Zv. Overlapping components are removed by vector rejection, and the complementary structure Zir_perp, which is orthogonal to the RGB semantic feature Zv, is retained. Using RGB semantic feature Zv as the query vector and complementary structural feature Zir_perp as the key and value vectors, cross-modal structural injection is performed to obtain the structural injection feature Zinj. Dynamically suppressed conductance is calculated based on RGB semantic uncertainty, and Shunting suppression fusion is performed to obtain the final fused feature Zf. Based on the final fusion feature Zf, the closed-set detection head is guided by VLM to output warehouse perception results; The results of warehouse perception are converted into warehouse business interface messages to drive robot operations.

2. The visible light-infrared neurophysical alignment visual semantic reconstruction and perception method for embodied warehouse robots according to claim 1, characterized in that, When extracting the RGB semantic feature Zv, a VLM-aligned semantic expert module is set in the RGB semantic mainstream. The pre-trained weights are kept unchanged or only the lightweight adaptation layer is fine-tuned. The semantic response L of the visual backbone output is weighted, reconstructed, and normalized using a learnable prototype matrix P to obtain the RGB semantic feature Zv, as shown in the following formula: ； Where L represents the region-level or pixel-level semantic response tensor of the visual backbone output, P represents the learnable prototype matrix, Softmax(·) represents the normalized weight assignment operator, LayerNorm(·) represents the layer normalization operator, and Zv represents the output RGB semantic features.

3. The visible light-infrared neurophysical alignment visual semantic reconstruction and perception method for embodied warehouse robots according to claim 1, characterized in that, When extracting physical structure features (Zir), gradient injection is performed in the infrared physical auxiliary flow. A fixed Sobel operator is used to extract horizontal and vertical gradients, and a nonparametric physical structure prior is constructed, as shown in the following formula: ； ； Where I_ir represents the input infrared image, M_grad represents the gradient magnitude map calculated by the Sobel operator, and I_enh represents the enhanced infrared image after injecting gradient prior; S_x and S_y are the horizontal and vertical Sobel operators, respectively, used to extract directional edges; epsilon is the stabilization term; lambda is the injection intensity coefficient; Repeat(·) represents the expansion operation that copies the single-channel gradient magnitude to the same number of infrared input channels; The enhanced infrared image is input into a lightweight encoder to generate physical structure features (Zir).

4. The visible light-infrared neurophysical alignment visual semantic reconstruction and perception method for embodied warehouse robots according to claim 1, characterized in that, The formula for the neuro-physical orthogonal fusion is as follows: ； The orthogonalized Zir_perp represents the orthogonal infrared complementary structure feature after removing redundant semantic components; Where, <·,·> represents the inner product, ||·||_2 represents the L2 norm, epsilon is the stability term, and Proj_Z_v(Z_ir) represents the projection component of the infrared physical structure feature Z_ir onto the semantic basis spanned by the RGB semantic feature Z_v.

5. A visible light-infrared neurophysical alignment visual semantic reconstruction and perception method for embodied warehouse robots according to claim 1, characterized in that, The process of performing cross-modal structure injection includes: Using RGB semantic features Zv as the query vector and orthogonal infrared complementary structural features Zir_perp as the key and value vectors, the injection process employs a dual residual structure of multi-head cross-attention and a feedforward network, as shown in the following formula: The intermediate state is used to obtain the structural injection feature Zinj through a nonlinear feedforward network and residual connections; Where W_q, W_k, and W_v are linear mapping matrices of query, key, and value, respectively; Q, K, and V are the projected query, key, and value, respectively; d_k represents the dimension of the key vector; Softmax(·) represents the attention normalization function; MHA(·) represents multi-head cross-attention; FFN(·) represents the feedforward network; LayerNorm(·) or LN(·) represents the layer normalization operation; Z_tilde_v represents the intermediate semantic state after cross-modal attention; and Zmid represents the intermediate state.

6. The visible light-infrared neurophysical alignment visual semantic reconstruction and perception method for embodied warehouse robots according to claim 1, characterized in that, The process of performing Shunting desaturation fusion includes: The dynamic suppression conductance is calculated based on the RGB semantic uncertainty, and the formula is as follows: ； ； Where g_leak is the leakage conductance, g_inh is the dynamic suppression conductance, lambda_s is the suppression scaling factor, W_gate is the uncertainty mapping weight, and sigma(·) is the Sigmoid gate function.

7. A visible light-infrared neurophysical alignment visual semantic reconstruction and perception method for embodied warehouse robots according to claim 1, characterized in that, The warehouse perception results output by the VLM-guided closed-set detection head include: The detection head includes both a decoupled classification branch and a bounding box regression branch. During inference, non-maximum suppression is applied to the candidate boxes, and the target box coordinates, category, confidence score, target source modality contribution, and low-confidence region markers are output, as shown in the following formula: ； Where S represents the category score of each candidate region to the target vocabulary, T^T represents the transpose of the category text matrix, tau is the temperature scaling factor, and cos(·,·) or the normalized inner product is used to measure the similarity between the final fused features and the embeddings of each category text.

8. A visible light-infrared neurophysical alignment visual semantic reconstruction and perception system for embodied warehouse robots, characterized in that, include: A multispectral acquisition module is used to simultaneously acquire and register RGB visible light and infrared images of the warehouse scene; The VLM text semantic library module is used to store and update the category text embeddings corresponding to the warehouse target vocabulary, providing text semantic priors for the RGB semantic mainstream module and the VLM guided detection head; The RGB semantic mainstream module is used to extract semantic features aligned with the VLM semantic space; The infrared physical-assisted flow module is used to extract physical structure features that are invariant to illumination and to perform gradient enhancement; The neuro-physical orthogonal fusion module performs neuro-physical orthogonal fusion on the physical structure feature Zir, projects the physical structure feature Zir onto the RGB semantic feature Zv, and removes overlapping components through vector rejection, retaining the complementary structure Zir_perp that is orthogonal to the RGB semantic feature; The cross-modal structure injection and shunting fusion module is used to perform cross-modal structure injection with RGB semantic feature Zv as query vector and complementary structure feature Zir_perp as key vector and value vector to obtain structure injection feature Zinj, and to complete shunting suppression fusion based on RGB semantic uncertainty to output the final fused feature Zf. The VLM-guided detection head outputs warehouse perception results based on the final fused feature Zf. The warehouse task output interface is used to convert the output warehouse perception results into standardized messages to drive robot operations.

Images