Multi-spectral visual perception system, method and device

Abstract

This application provides a multi-spectral visual perception system, method, and device, relating to computer vision technology. The system includes: a data acquisition unit for acquiring multi-spectral image data from a task execution environment; the multi-spectral image data includes visible light images and infrared images; and a processing unit for, upon acquiring the multi-spectral image data, inputting it into a human visual model to obtain a multi-spectral fused image, and inputting the multi-spectral fused image into a machine vision model to execute a preset perception task to obtain a perception result. The human visual model and the machine vision model are obtained through joint training, which introduces a gradient matching regularization term to constrain the parameter update direction of both models, enabling synchronous parameter optimization. This application effectively improves the accuracy of the perception results.

Description

Technical Field

[0001] This application relates to computer vision technology, and more particularly to a multi-spectral visual perception system, method, and device. Background Technology

[0002] In visual perception systems such as security monitoring, assisted driving, and industrial inspection, overcoming the limitations of a single spectrum by fusing the rich texture details of visible light images with the stable thermal radiation information of infrared images has become a key requirement for improving system environmental adaptability and perception reliability. Generating images that simultaneously meet the needs of human visual observation and automatic machine recognition through fusion technology is an important development direction in this field.

[0003] Currently, to achieve the above goals, known technologies mainly follow two paths. The first focuses on algorithmic improvements, such as designing advanced image fusion algorithms to generate fused images with better visual effects, which are then input into object detection or segmentation models for machine recognition. The second involves integrated hardware and software system design, which involves fixing visible light and infrared cameras to a specific structure and combining them with conventional fusion and recognition algorithms to build solutions for specific application scenarios.

[0004] However, when fusing images through any of the above methods to obtain perceptual results, there is a drawback of low accuracy. Summary of the Invention

[0005] This application provides a multi-spectral visual perception system, method, and device to improve the accuracy of perception results.

[0006] In a first aspect, this application provides a multi-spectral visual perception system, the system comprising:

[0007] The acquisition unit is used to acquire multi-spectral image data in the task execution environment; the multi-spectral image data includes visible light images and infrared images.

[0008] The processing unit is configured to, upon acquiring the multispectral image data, input the multispectral image data into a human visual model to obtain a multispectral fused image, and input the multispectral fused image into a machine vision model to perform a preset perception task to obtain a perception result; wherein, the human visual model and the machine vision model are obtained through joint training, and the joint training introduces a gradient matching regularization term, which constrains the parameter update direction of the human visual model and the machine vision model, so that the parameters of the two models are optimized synchronously.

[0009] In one possible implementation, the system further includes a human-computer interaction unit, which receives training instructions input by the user and sends the training instructions to the processing unit; the training instructions are joint optimization instructions or non-joint optimization instructions.

[0010] Accordingly, the processing unit is further configured to: upon receiving the joint optimization instruction, jointly train the human visual model and the machine visual model based on the joint optimization instruction; and upon receiving the non-joint optimization instruction, independently train the human visual model and the machine visual model.

[0011] In one possible implementation, the processing unit is specifically used for:

[0012] The multispectral image data is preprocessed, and the preprocessed multispectral image data is input into the human visual model to obtain the multispectral fused image; wherein, the human visual model fuses the visible light color texture and infrared thermal radiation features in the preprocessed multispectral image data to obtain the multispectral fused image.

[0013] And / or, when jointly training the human visual model and the machine vision model, the visible light image and infrared image used for training are input into the human visual model to obtain the training fused image; the training fused image is input into the machine vision model to obtain the perception task prediction result;

[0014] Based on the training fused image and the task prediction result, the gradient matching regularization term is generated, and a joint loss function is constructed based on the gradient matching regularization term;

[0015] Based on the joint loss function, the parameters of the human vision model and the machine vision model are updated synchronously.

[0016] In one possible implementation, the processing unit is specifically used for:

[0017] Based on the training fused image and the visible light image and infrared image used for training, calculate the fusion loss; based on the perception task prediction result and the corresponding perception task ground truth, calculate the perception task loss.

[0018] The gradient of the first parameter is calculated based on the perception task loss, the gradient of the second parameter is calculated based on the fusion loss, and the gradient matching regularization term is generated by calculating the directional similarity between the gradient of the first parameter and the gradient of the second parameter.

[0019] The joint loss function is constructed based on the gradient matching regularization term and the perception task loss.

[0020] In one possible implementation, the processing unit is further configured to:

[0021] When generating the gradient matching regularization term, the weighting coefficients of the first parameter gradient and the second parameter gradient are dynamically determined based on the attention mechanism.

[0022] In one possible implementation, the acquisition unit includes at least two image acquisition devices and an image acquisition card; the input terminal of the image acquisition card is connected to the output terminal of the at least two image acquisition devices, and the output terminal is connected to the input terminal of the processing unit.

[0023] The at least two image acquisition devices are used to acquire the visible light image and the infrared image respectively, and are also used to change the field of view range by adjusting their own posture or position;

[0024] The image acquisition card is used to receive multi-spectral image data output by the at least two image acquisition devices and transmit the multi-spectral image data to the processing unit.

[0025] In one possible implementation, the image acquisition card and the processing unit are connected via a slot-type connection.

[0026] In one possible implementation, the system further includes a housing encapsulation assembly for integrating and encapsulating the acquisition unit and the processing unit; wherein the housing encapsulation assembly adopts a detachable connection structure and is provided with a ventilation structure for heat dissipation.

[0027] Secondly, this application provides a multi-spectral visual perception method, the method comprising:

[0028] Acquire multispectral image data in the task execution environment; the multispectral image data includes visible light images and infrared images;

[0029] The multispectral image data is input into a human visual model to obtain a multispectral fused image, and the multispectral fused image is input into a machine vision model to perform a preset perception task to obtain a perception result; wherein, the human visual model and the machine vision model are obtained through joint training, and the joint training introduces a gradient matching regularization term, which constrains the parameter update direction of the human visual model and the machine vision model, so that the parameters of the two models are optimized synchronously.

[0030] Thirdly, the application provides a multi-spectral visual sensing device, the device comprising:

[0031] The acquisition module is used to acquire multispectral image data in the task execution environment; the multispectral image data includes visible light images and infrared images;

[0032] The processing module is used to input the multispectral image data into a human visual model to obtain a multispectral fused image, and input the multispectral fused image into a machine vision model to perform a preset perception task to obtain a perception result; wherein, the human visual model and the machine vision model are obtained through joint training, and the joint training introduces a gradient matching regularization term, which constrains the parameter update direction of the human visual model and the machine vision model, so that the parameters of the two models are optimized synchronously.

[0033] Fourthly, this application provides an electronic device, including a processor and a memory communicatively connected to the processor;

[0034] The memory stores computer-executed instructions;

[0035] The processor executes computer execution instructions stored in the memory to implement the method as described in the second aspect.

[0036] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method described in the second aspect.

[0037] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the second aspect.

[0038] This application provides a multi-spectral visual perception system, method, and device. The multi-spectral visual perception system includes an acquisition unit and a processing unit. Specifically, the system acquires multi-spectral image data during task execution through the acquisition unit and transmits it to the processing unit. The processing unit then uses a trained human visual model to fuse the multi-spectral image data, obtaining a multi-spectral image. This multi-spectral image is then input into a trained machine vision model to execute a preset perception task and obtain the perception result. In this system, since the human visual model and the machine vision model are jointly trained, and the joint training introduces a gradient matching regularization term to constrain the parameter update direction of both models, the parameters of the two models are optimized synchronously. Therefore, the multi-spectral fused image can satisfy both the human visual requirement for detail and discernibility and the machine vision requirement for feature extraction and semantic recognition, avoiding a disconnect between the needs of the two models. This improves the collaborative adaptability and individual performance of the two models, thereby enhancing the accuracy of the final perception result output by the machine vision model. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0040] Figure 1 This is a schematic diagram illustrating an application scenario of a multi-spectral visual perception system provided in an embodiment of this application.

[0041] Figure 2 A schematic diagram of the structure of a multi-spectral visual perception system provided in this application embodiment. Figure 1 ;

[0042] Figure 3 A schematic diagram of the structure of a multi-spectral visual perception system provided in this application embodiment. Figure 2 ;

[0043] Figure 4A An example of a multi-spectral visual perception system provided in this application embodiment. Figure 1 ;

[0044] Figure 4B An example of a multi-spectral visual perception system provided in this application embodiment. Figure 2 ;

[0045] Figure 4C An example of a multi-spectral visual perception system provided in this application embodiment. Figure 3 ;

[0046] Figure 4D Figure 4 shows an example of a multi-spectral visual perception system provided in an embodiment of this application;

[0047] Figure 5 A flowchart illustrating a multi-spectral visual perception method provided in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of a multi-spectral visual sensing device provided in an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0050] Explanation of reference numerals in the attached figures:

[0051] 1. Top cover; 1-1. First annular protrusion; 1-2. Second annular protrusion; 2. Top cover plate; 2-1. First slot; 2-2. Limiting plate; 2-3. Annular groove; 3. Side plate; 3-1. First buckle; 3-2. Ventilation hole; 3-3. Right-angle limiting buckle; 3-4. Second buckle; 4. First visible light camera; 5. Infrared camera; 6. Second visible light camera; 7. Base plate; 7-1. Limiting side plate; 7-2. Mounting bracket; 8. Stud; 9. Display screen; 10. Camera mounting plate; 10-1. Parallel slide; 10-2. Connecting groove; 11. Image acquisition card; 12. Acquisition card connecting plate; 13. Support sleeve; 14. Pressure block; 15. Processing unit; 16. Battery.

[0052] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0053] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0054] With the development of computer vision and robotics technologies, visual sensors, characterized by their non-contact nature, low cost, and high measurement accuracy, are widely used in various production activities. Currently, in visual perception systems such as security monitoring, assisted driving, and industrial inspection, overcoming the limitations of a single spectrum by fusing the rich texture details of visible light images with the stable thermal radiation information of infrared images has become a key requirement for improving the system's environmental adaptability and perception reliability. Generating images that simultaneously meet the needs of human visual observation and automatic machine recognition through fusion technology is an important development direction in this field.

[0055] It should be understood that visible light, as the spectrum band through which humans perceive the environment, exhibits clear appearance and rich texture under sufficient ambient light. However, it often produces poor images when there are drastic changes in brightness or obstructions. In contrast, infrared cameras image based on infrared radiation emitted by objects themselves, which is invisible to the human eye. When there is a significant temperature difference between the object and its environment, the object in the infrared image often exhibits a significant high contrast characteristic. This characteristic is unaffected by changes in ambient light, effectively compensating for the shortcomings of visible light cameras in imaging due to the influence of lighting conditions.

[0056] In known technologies, to achieve the generation of images that simultaneously meet the needs of human visual observation and machine automatic recognition, there are two main paths: one is to focus on algorithm-level improvement, designing advanced image fusion algorithms to generate fused images with better visual effects, and then inputting them into target detection or segmentation models to complete machine recognition; the other is to design an integrated system that combines hardware and software, fixing visible light and infrared cameras to a specific structure, and combining them with conventional fusion and recognition algorithms to build a solution for specific scenarios.

[0057] However, when using any of the above methods to achieve multi-spectral image fusion and subsequent machine vision perception tasks, there are significant drawbacks such as low accuracy due to the disconnect between human vision and machine vision requirements or poor hardware compatibility.

[0058] Therefore, embodiments of this application provide a multi-spectral visual perception system, method, and device to solve the above-mentioned problems. Specifically, the multi-spectral visual perception system of this application proposes to achieve fusion perception by jointly training a human visual model and a machine vision model. Specifically, during joint training, a gradient matching regularization term is introduced to constrain the parameter update direction of the human visual model and the machine vision model, so that the parameters of the two models are optimized synchronously.

[0059] It is understood that the multi-spectral visual perception system of this application is applicable to any scenario in which a perception task is achieved by combining multi-spectral images. For example, Figure 1 This is a schematic diagram illustrating an application scenario of a multi-spectral visual perception system provided in an embodiment of this application, such as... Figure 1 As shown, the system of this application can be used in battery manufacturing scenarios.

[0060] Currently, in the battery manufacturing process, it is necessary to conduct quality inspections on key links such as battery electrodes, cell packaging, and casing welding. Single visible light images are difficult to identify hidden defects such as micro-cracks inside the electrodes and bubbles in the packaging, while infrared images can capture such hidden problems through differences in thermal radiation. Therefore, it is necessary to integrate the two spectrum images to achieve comprehensive quality perception.

[0061] like Figure 1 As shown, the multi-spectral visual perception system of this application is deployed in a battery manufacturing scenario. This multi-spectral visual perception system is used to interact and communicate with the central control system, quality inspection terminal, and production management platform of the battery production line. The multi-spectral visual perception system outputs the collected multi-spectral image data and processed perception results, such as electrode defect type, packaging pass rate, and welding quality level, to the central control system and production management platform to report manufacturing progress and provide feedback on quality issues in real time. At the same time, it pushes detailed defect analysis data to the quality inspection terminal to assist inspection personnel in verification.

[0062] By applying the multi-spectral visual perception system of this application, both visible and latent defects in battery production can be accurately identified by simultaneously utilizing the texture details of visible light images and the thermal radiation characteristics of infrared images. Compared with traditional single-spectrum detection schemes, this effectively improves the defect detection rate. Furthermore, through joint model training and optimization, the accuracy of the perception results is higher, reducing misjudgments and missed judgments caused by human inspection errors, thus lowering the risk of defective products flowing into the next stage. Simultaneously, the real-time output of manufacturing progress and quality data can assist the production management platform in dynamically adjusting production parameters, improving battery production efficiency and finished product yield.

[0063] It should be understood that the multi-spectral visual perception system of this application can also interact with user terminals, such as mobile tablets and computers used by management personnel, supporting users to remotely view perception results and retrieve historical detection data. The specific interaction objects can be flexibly set according to actual production needs, and this embodiment does not limit this. In addition, the multi-spectral visual perception system of this application is also applicable to scenarios such as industrial inspection, assisted driving, security monitoring, and intelligent robots that require the combination of multi-spectral images to achieve perception tasks, and this embodiment does not limit this.

[0064] The following detailed description, with reference to the accompanying drawings and taking an application in a battery manufacturing scenario as an example, illustrates some implementations of the multi-spectral visual perception system of this application. Where the embodiments do not conflict, the following embodiments and features thereof can be combined with each other.

[0065] This application provides a multi-spectral visual perception system. Figure 2 A schematic diagram of the structure of a multi-spectral visual perception system provided in this application embodiment. Figure 1 ,like Figure 2 As shown in the figure, a multi-spectral visual perception system provided in this application includes an acquisition unit and a processing unit, with the output of the acquisition unit connected to the input of the processing unit.

[0066] In this embodiment, the acquisition unit is used to acquire multispectral image data in the task execution environment and transmit the multispectral image data to the processing unit. Upon acquiring the multispectral image data, the processing unit inputs the multispectral image data into a human visual model to obtain a multispectral fused image, and then inputs the multispectral fused image into a machine vision model to execute a preset perception task and obtain a perception result.

[0067] The multispectral image data includes visible light images and infrared images. The task execution environment refers to the specific spatial setting where the system actually operates and performs visual perception tasks, such as an electrode processing workshop in battery production, an equipment room in industrial inspection, or a road driving environment in assisted driving—that is, the physical space where target monitoring, quality inspection, or environmental perception needs to be achieved through multispectral images. During task execution, the system acquires visible light and infrared images through its acquisition unit.

[0068] Specifically, in this embodiment, the acquisition unit includes at least two image acquisition devices and an image acquisition card. The input terminal of the image acquisition card is connected to the output terminals of the at least two image acquisition devices, and the output terminal is connected to the input terminal of the processing unit. Based on this, the at least two image acquisition devices are used to acquire visible light images and infrared images respectively, and are also used to change the field of view by adjusting their own posture or position. The image acquisition card is used to receive multispectral image data output by the at least two image acquisition devices and transmit the multispectral image data to the processing unit.

[0069] It should be understood that an image acquisition card is a hardware component used to receive and convert the output signals of an image acquisition device, and it needs to be compatible with the signal type and data format of the image acquisition device. For example, if the acquisition device outputs analog signals, an analog image acquisition card is required; if it outputs digital signals, a digital image acquisition card is required. Therefore, when the model of the acquisition device changes, resulting in a change in signal / format, the image acquisition card usually needs to be replaced accordingly to ensure data transmission compatibility.

[0070] In this embodiment, the input terminal of the image acquisition card is physically connected to the output terminals of at least two acquisition devices via a compatible signal transmission cable. Furthermore, as a preferred design, the image acquisition card and the processing unit employ a slot-type connection. More specifically, a standardized slot-type connection (such as a PCIe slot) is used, and this slot supports compatible interfacing with image acquisition cards of different models and adapted to different signal types.

[0071] This setting enables quick installation, removal, and replacement of the image acquisition card. When it is necessary to adapt to different models of image acquisition devices, or when the acquisition card malfunctions, there is no need to disassemble the entire processing unit; simply plug and unplug the acquisition card to replace it, greatly improving the convenience of system hardware maintenance and function expansion.

[0072] In this embodiment, the field of view is adjusted by mounting a motorized rotating gimbal and a linear guide rail assembly at the bottom of the image acquisition device. Specifically, the motorized rotating gimbal can rotate the acquisition device around the horizontal and vertical directions to adjust the acquisition angle; the linear guide rail assembly can drive the acquisition device to translate along the rail to change the acquisition distance; the two work together to flexibly adjust the coverage of the acquisition field of view and adapt to the image acquisition needs of different task scenarios.

[0073] In this embodiment, visible light and infrared images are acquired by at least two independent acquisition devices, which fully leverages the spectral acquisition advantages of both devices and avoids the loss of spectral information caused by acquisition by a single device. This provides comprehensive raw data for the subsequent generation of high-quality fused images by the human visual model, laying the foundation for the accuracy of the perception results. Simultaneously, the acquisition devices support adjusting the field of view by changing their posture or position, breaking the limitations of traditional fixed-field-of-view acquisition. This allows them to adapt to acquisition target scenarios of different sizes and distances. For example, in battery production, the field of view can be flexibly adjusted to cover different areas of the electrode sheet; in industrial inspection, it can adapt to the inspection needs of equipment at different heights, significantly improving the system's scenario adaptability.

[0074] In addition, the image acquisition card enables unified reception and transmission of data from multiple devices, eliminating the need for a processing unit to connect directly to multiple acquisition devices, thus reducing the hardware interface burden on the processing unit. At the same time, the acquisition card can adapt to the data formats of different acquisition devices, providing flexibility for subsequent replacement or upgrades of acquisition devices and reducing the overall system transformation cost.

[0075] It should be understood that in practical applications, multi-spectral image data can also include image data of other spectral dimensions such as ultraviolet images and millimeter-wave images. The specific choice can be flexibly made according to the needs of the sensing task, such as surface discharge detection of ultraviolet equipment or millimeter-wave imaging through obstructions. This embodiment does not limit this. Furthermore, in practical applications, at least two image acquisition devices can adjust their posture or position to change the field of view by manually adjusting the bracket angle, remotely controlling the motor-driven gear transmission mechanism, or automatically adjusting the position of the robotic arm based on a preset program. This embodiment also does not limit this.

[0076] In this embodiment, the human vision model and the machine vision model are obtained through joint training. The joint training introduces a gradient matching regularization term, which constrains the parameter update direction of the human vision model and the machine vision model, so that the parameters of the two models are optimized synchronously.

[0077] It should be understood that the human visual model is used to fuse visible light and infrared images transmitted by the acquisition device to obtain a multi-spectral fused image. As a preferred design, in this embodiment, the processing unit first preprocesses the multi-spectral image data, and then inputs the preprocessed multi-spectral image data into the human visual model to obtain the multi-spectral fused image. Specifically, the human visual model fuses the visible light color texture and infrared thermal radiation features in the preprocessed multi-spectral image data to obtain the multi-spectral fused image.

[0078] In this embodiment, the human eye vision model is a feature fusion model built on a deep learning architecture, which includes a color and texture feature extraction subnetwork for visible light images, a thermal radiation feature extraction subnetwork for infrared images, and a feature attention fusion subnetwork.

[0079] Building upon this foundation, when the human visual model receives multispectral image data, it first preprocesses it. Then, two extraction sub-networks separate texture features such as object edges and color distribution from the preprocessed visible light image, and thermal features such as object temperature differences and thermal radiation intensity from the infrared image. An attention fusion sub-network then dynamically assigns weights to the two types of features based on scene requirements. For example, in low-light scenarios, infrared thermal features are given higher weight to highlight object outlines; in well-lit scenarios, the focus is on preserving visible light texture features to restore details. Finally, the weighted features are integrated dimensionally and reconstructed at the pixel level to generate a multispectral fused image.

[0080] In this embodiment, preprocessing includes image denoising, size unification, viewpoint alignment, and pixel value normalization. Image denoising eliminates noise caused by equipment interference and ambient light fluctuations during acquisition, preventing noise from affecting the accuracy of subsequent feature extraction. Size unification uses interpolation algorithms to adjust visible light and infrared images to the same resolution, resolving the issue of inconsistent image sizes caused by differences in parameters of different acquisition devices. Viewpoint alignment corrects spatial misalignment between the two types of images due to shooting angle shifts using image registration technology. Pixel value normalization maps the color brightness values ​​of visible light and the thermal radiation intensity values ​​of infrared to the [0,1] interval, reducing the interference of dimensional differences on feature fusion.

[0081] It should be understood that the human eye vision model in this embodiment is trained on a large dataset containing visible light images, infrared images and corresponding fusion labels. It can learn the complementary rules of the two spectral features, simulate the human eye’s integration logic of multi-dimensional visual information, and ensure that the output fused image not only conforms to the human eye’s observation habits, but also retains key features for subsequent machine recognition.

[0082] In this embodiment, the machine vision model is used to perform a preset perception task on the multi-spectral fusion image output by the human eye vision model, and outputs a perception result corresponding to the task. The preset perception task can be flexibly set according to the application scenario. For example, in a battery production scenario, it could be "detection of micro-cracks in electrode sheets" or "identification of air bubbles in cell packaging"; in an industrial inspection scenario, it could be "judgment of loose bolts in equipment" or "assessment of pipeline corrosion"; and in an assisted driving scenario, it could be "classification of pedestrians and obstacles" or "recognition of traffic signs." The perception result is specific information matched to the task, such as defect type (crack, air bubble), defect location coordinates (specific values ​​on the X / Y axis), target category (pedestrian, vehicle, speed limit sign), and target risk level (e.g., "high risk" or "low risk" for loose bolts).

[0083] It should be understood that the machine vision model in this embodiment is built on mature target detection / recognition architectures such as convolutional neural networks (CNN) and YOLO. After being trained on a large number of multi-spectral fusion images labeled with the ground truth of the perception task, it has the ability to accurately extract key features of the target from the fusion image, match feature categories and output judgment results. Moreover, due to joint training with the eye vision model, it has a stronger adaptability to the feature distribution of the fusion image, further reducing recognition errors.

[0084] In this embodiment, when the processing unit receives multi-spectral image data, it first preprocesses the data before fusing it. This process eliminates image quality issues caused by device interference and environmental noise through denoising. It also resolves spatial misalignment issues between visible light and infrared images caused by differences in acquisition device parameters and shooting angle shifts through size unification and viewing angle alignment. Furthermore, it reduces the interference of the difference in the dimensions of the two types of image data on feature fusion through pixel value normalization. This lays the foundation for the subsequent accurate extraction of visible light color texture and infrared thermal radiation features by the human visual model, thereby improving the clarity, feature integrity, and spatial consistency of the multi-spectral fused image. It avoids poor fusion results due to defects in the original data and ultimately provides high-quality input data for the machine vision model to perform perception tasks, helping to improve the accuracy of perception results.

[0085] The multi-spectral visual perception system provided in this embodiment acquires multi-spectral image data, including visible light and infrared images, through an acquisition unit and transmits it to a processing unit. The processing unit inputs the multi-spectral image data into a human eye vision model obtained through joint training to generate a multi-spectral fused image. The fused image is then input into a machine vision model to perform a perception task. Furthermore, the joint training introduces gradient matching regularization terms to constrain the synchronous optimization of the parameters of the two models, thereby achieving multi-spectral image fusion and accurate perception.

[0086] The multi-spectral visual perception system in this embodiment effectively solves the problems of disconnect between human eye and machine vision needs and low perception accuracy in known technologies. It generates fused images that conform to observation habits through human eye vision models, and improves the accuracy of perception results by using jointly trained machine vision models. At the same time, the flexible field of view adjustment and hardware adaptation design of the acquisition unit greatly enhance the system's adaptability to different scenarios and reduce hardware maintenance and upgrade costs.

[0087] This application also provides an embodiment of a multi-spectral visual perception system for detailing the training method of the two models. Figure 3 A schematic diagram of the structure of a multi-spectral visual perception system provided in this application embodiment. Figure 2 The following is combined with Figure 3 The multi-spectral visual perception system of this embodiment will be further described.

[0088] like Figure 3 As shown, the multi-spectral visual perception system in this embodiment also includes a human-computer interaction unit, the output of which is connected to the input of the processing unit. Based on this, the human-computer interaction unit receives training instructions input by the user and sends these instructions to the processing unit. The training instructions can be joint optimization instructions or non-joint optimization instructions.

[0089] Accordingly, when the processing unit receives a joint optimization instruction, it performs joint training on the human vision model and the machine vision model based on the joint optimization instruction; when it receives a non-joint optimization instruction, it performs independent training on the human vision model and the machine vision model.

[0090] Specifically, the human-computer interaction unit can provide the user with a first button for instructing joint optimization training and a second button for instructing non-joint optimization training, allowing the user to choose. When the user selects the first button, it confirms that the training instruction received from the user is a joint optimization instruction; when the user selects the second button, it confirms that the training instruction received from the user is a non-joint optimization instruction.

[0091] It should be understood that in practical applications, the human-computer interaction unit can also receive user feedback in other ways, such as providing a touch screen input interface for users to manually input training command types, or supporting voice recognition function to receive users' voice training commands, or obtaining commands by reading training configuration files that users have pre-uploaded to a specified storage path, or even supporting remote terminals (such as computers, mobile APPs) to connect and send training commands, etc. This embodiment does not limit this.

[0092] like Figure 3As shown, in this embodiment, the input end of the human-computer interaction unit is also connected to the output end of the processing unit, used to receive the perception results output by the processing unit through the machine vision model, and to feed back the perception results to the user. Specifically, the perception results can be displayed on the screen set in the human-computer interaction unit. In practical applications, the perception results can also be fed back through the voice broadcaster set in the human-computer interaction unit, or by automatically triggering the human-computer interaction unit to send messages to the user terminal. This embodiment does not limit this method.

[0093] In this embodiment, when the processing unit jointly trains the human vision model and the machine vision model, it first inputs the visible light image and infrared image used for training into the human vision model to obtain the training fused image; then it inputs the training fused image into the machine vision model to obtain the perception task prediction result; subsequently, based on the training fused image and the task prediction result, it generates a gradient matching regularization term and constructs a joint loss function based on the gradient matching regularization term; finally, based on the joint loss function, it synchronously updates the parameters of the human vision model and the machine vision model.

[0094] The processing unit constructs the joint loss function through the following process: First, it calculates the fusion loss based on the training fused image and the visible light and infrared images used for training; then, it calculates the perception task loss based on the perception task prediction result and the corresponding perception task ground truth; subsequently, it calculates the gradient of the first parameter based on the perception task loss, calculates the gradient of the second parameter based on the fusion loss, and generates a gradient matching regularization term by calculating the directional similarity between the gradient of the first parameter and the gradient of the second parameter; finally, it constructs the joint loss function based on the gradient matching regularization term and the perception task loss.

[0095] Specifically, in this embodiment, the output of the human vision model is concatenated with the input of the machine vision model. Based on this, the joint training process first requires constructing a training set containing massive amounts of labeled data. Each sample in the dataset includes a visible light image, an infrared image, and the corresponding ground truth for the perception task (such as defect annotations, target category labels, etc.). After training begins, the processing unit reads the training set samples in batches, inputting the visible light and infrared images of each batch into the human vision model. The model extracts key features from the two types of images through its internal feature extraction subnetwork, and then dynamically weights and fuses them through an attention fusion subnetwork, outputting a training fused image. Next, the training fused image is input into the machine vision model, which extracts semantic features from the fused image through operations such as convolution and pooling, and outputs the perception task prediction results (such as defect probability, target coordinates, etc.) in conjunction with a fully connected layer.

[0096] Then, the fusion loss (using the structural similarity index SSIM or mean squared error MSE to measure the feature consistency between the training fused image and the original image) and the perceptual task loss (using cross-entropy loss or IOU loss to measure the deviation between the prediction result and the true value) are calculated. Then, the gradient of the first parameter of the machine vision model is calculated by backpropagation based on the perceptual task loss, and the gradient of the second parameter of the human visual model is calculated by backpropagation based on the fusion loss. The directional similarity between the two gradients is calculated by using the cosine similarity formula and converted into a gradient matching regularization term.

[0097] Finally, the gradient matching regularization term and the perception task loss are weighted and summed according to a preset ratio to obtain the joint loss function. Using stochastic gradient descent (SGD) or adaptive moment estimation (Adam) optimizer, the weights, biases and other parameters of the two models are updated synchronously based on the joint loss function. The above batch training process is repeated until the joint loss function value converges to the preset threshold, thus completing the joint training.

[0098] More specifically, assuming x, y, and u represent a visible light image, an infrared image, and a fused image of the same size, respectively, θ f and θ d These are the parameters to be optimized for the visible light and infrared image fusion model Φ under human vision and the perception and recognition task model Ψ under machine vision, respectively. Based on this, the process of generating a visually fused image from the input visible light and infrared images can be represented as: .

[0099] Furthermore, joint optimization training aims to leverage semantic information from downstream perceptual tasks such as detection to optimize visual fusion results and improve the detection performance of visually fused images. Therefore, the relationship between the human eye perception model and the machine perception model satisfies a nested optimization problem, the objective of which can be modeled as: This formula embodies the joint optimization process of the two model parameters, where L d To detect task loss.

[0100] To effectively solve the above problem, this embodiment models it as a meta-learning process. Specifically, the optimal parameters θ of the downstream machine vision model... d Depends on the parameters θ of the upstream human eye visual model d ,Right now , where t is the truth value of the perception task.

[0101] Based on this, in this embodiment, the processing unit first calculates the perception task loss L. d Regarding the machine vision model parameter θ d gradient of the first parameter Then calculate the fusion loss L. f Regarding the machine vision model parameter θ d The gradient of the second parameter Furthermore, by calculating the directional difference between the gradients of the first and second parameters, specifically the negative cosine similarity, a gradient matching regularization term is generated, as follows: .

[0102] Finally, the joint loss function L total It consists of a weighted sum of the perceptual task loss and the aforementioned gradient matching regularization term, specifically expressed as: Here, λ is a hyperparameter ranging from 0.1 to 1.0. Its optimal value can be determined using a grid search method (e.g., iterating through values ​​of 0.1, 0.2…1.0 before training and selecting the λ value that minimizes the validation set loss), or it can be dynamically adjusted during training based on the loss convergence speed (e.g., setting λ to 0.3 initially to prioritize the accuracy of the perception task, and later adjusting it to 0.6 to enhance the synergy between the two models). By minimizing this joint loss function, the gradient direction generated during the fusion process can be kept consistent with the direction of the perception task optimization while optimizing the perception task, thus synchronously updating θ. f and θ d This enables joint training of the two models.

[0103] In the above process, the processing unit updates the parameters of the human vision model and the machine vision model synchronously based on the joint loss function, ensuring that the parameters θ of the human vision model are updated accordingly. f When adjusting towards generating a better fused image, the machine vision model parameter θ d Simultaneously, adjustments are made towards a direction that can more accurately identify the fused image. This co-evolutionary optimization process breaks the limitations of independent parameter updates and fragmented optimization objectives in traditional serial design, enabling the final model combination to achieve globally optimal collaborative perception performance.

[0104] In the above process, the joint loss function is constructed by first calculating the fusion loss and the perception task loss, then generating a gradient matching regularization term, and finally weighting and summing it with the perception task loss. By minimizing this joint loss function, the parameters of the human visual model and the machine vision model can be optimized simultaneously, ensuring that the fused image retains key spectral features and adapts to the requirements of the perception task. At the same time, it constrains the gradient directions of the two model parameters to be consistent, avoiding the decoupling of optimization directions. It can also flexibly adapt to different scenarios through hyperparameters, ultimately improving the system's perception accuracy and scene adaptability.

[0105] As a preferred example, the processing unit is also used to: dynamically determine the weighting coefficients of the first parameter gradient and the second parameter gradient based on the attention mechanism when generating the gradient matching regularization term.

[0106] Specifically, in this embodiment, the processing unit first analyzes the numerical proportions of fusion loss and perceptual task loss in the current training batch. If the proportion of fusion loss is high (e.g., greater than 50%), it indicates that the fusion effect of the human visual model has a large deviation, and its parameter update priority needs to be increased. In this case, a higher weighting coefficient (e.g., 0.6-0.8) is assigned to the gradient of the second parameter through an attention mechanism, and a lower coefficient (e.g., 0.2-0.4) is assigned to the gradient of the first parameter. If the proportion of perceptual task loss is high (e.g., greater than 50%), it indicates that the recognition accuracy of the machine vision model is insufficient, and optimization needs to be emphasized. Therefore, a higher coefficient (0.6-0.8) is assigned to the gradient of the first parameter, and a lower coefficient (0.2-0.4) is assigned to the gradient of the second parameter. If the proportions of the two losses are close, the coefficients are both set to around 0.5.

[0107] Meanwhile, in this embodiment, the processing unit statistically analyzes the loss change trend of the first 10-20 batches in real time. If one loss continues to decrease while the other loss stagnates, the coefficients will be dynamically adjusted to balance the optimization progress of the two models and ensure that the weighting coefficients always adapt to the current training state.

[0108] In practical applications, the processing unit can also dynamically determine the weighting coefficients based on the scene features of the training samples. For example, for samples from battery production scenarios, if the electrode texture is blurry (poor visible light image quality), a higher coefficient is assigned to the gradient of the second parameter corresponding to the thermal radiation features of the infrared image, thereby enhancing the extraction of thermal features by the fusion model. Alternatively, a pre-trained feature importance evaluation model can be used to score the contribution of different spectral features in the perception task and assign gradient weighting coefficients according to the score ratio. In addition, users can manually set coefficient adjustment rules through the human-computer interaction unit (such as specifying that "the gradient coefficient of the first parameter in the defect detection scenario is not less than 0.7"). The processing unit will execute the coefficient allocation according to the preset rules. This embodiment does not limit this.

[0109] This setup allows for flexible adjustments to optimization priorities based on real-time training status (loss percentage, trend) or sample characteristics. For example, when the fusion loss remains high, the gradient weight of the second parameter can be rapidly increased to accelerate the performance improvement of the fusion model. Furthermore, it ensures that the two models are always in a state of collaborative optimization, preventing over-optimization of one model from degrading the performance of the other. For instance, it avoids the loss of key texture features in the fused image due to excessive focus on perception task optimization. Ultimately, this allows the jointly trained system to achieve better perception accuracy in multiple scenarios compared to the fixed-coefficient scheme, while also reducing the risk of model overfitting and enhancing robustness in complex scenarios (such as sudden changes in lighting or target occlusion).

[0110] In this embodiment, when the processing unit trains the human vision model and the machine vision model independently, it needs to construct independent training datasets and optimization objectives for each model. Specifically, for the human vision model, a dataset containing visible light images, infrared images, and standard fused image labels is used. The fusion loss (such as SSIM loss) is used as the sole optimization objective. Backpropagation is used to update only the parameters of the human vision model. The training objective is to ensure that the fused image output by the model is consistent with the standard label in terms of texture, color, and thermal feature preservation.

[0111] For machine vision models, a dataset containing multi-spectral fused images (which can be pre-generated by mature fusion algorithms) and corresponding ground truth values ​​for perception tasks is used. The perception task loss (such as cross-entropy loss) is the sole optimization objective, and only the parameters of the machine vision model are updated. The training objective is to improve the model's accuracy in recognizing and detecting targets in the fused images. During training, the parameter updates of the two models do not interfere with each other. The processing unit can choose to complete the training of one model first and then start the other, or use multi-threaded parallel processing to train simultaneously, according to user needs. After independent training is completed, the parameters of the two models can be loaded into the system for subsequent image fusion and perception tasks.

[0112] In the multi-spectral visual perception system provided in this embodiment, the human-computer interaction unit supports receiving joint optimization instructions or non-joint optimization instructions, providing users with a flexible training mode selection space. When users need to quickly adapt to new scenarios (such as the first application to battery electrode detection), they can choose joint optimization instructions to improve the overall perception adaptability of the system through collaborative training of two models; when users only need to optimize a single model (such as only updating the defect recognition algorithm of the machine vision model), they can choose non-joint optimization instructions to avoid unnecessary repeated training of the fusion model, significantly saving training time and computing power costs.

[0113] In this application, the multi-spectral visual perception system further includes a housing encapsulation component, which is used to integrate and encapsulate the acquisition unit and the processing unit; wherein, the housing encapsulation component adopts a detachable connection structure, and the housing encapsulation component is provided with a ventilation structure for heat dissipation. It should be understood that the housing encapsulation component is also used to integrate and encapsulate the human-computer interaction unit.

[0114] In this embodiment, the detachable connection structure is a detachable snap-fit ​​and slot-fit connection structure. The housing encapsulation assembly includes a base plate, a top cover, a top cover plate, and side plates. Using the base plate as the assembly reference, the top cover and top cover plate are connected by an interference fit between a boss and a groove. The top structure formed by the top cover and top cover plate is then connected to the side plates by snap-fit ​​through slots. The side plates are detachably connected to the base plate by snap-fit ​​through slots. Ventilation holes are provided on the side plates for heat dissipation. In this embodiment, evenly distributed ventilation holes are provided on both opposite side plates, which can meet the heat dissipation requirements of components such as the processing unit and battery, while preventing a large amount of dust from entering the housing.

[0115] It should be understood that in practical applications, the housing packaging component can also be designed as a split structure, with the upper and lower shells detachably connected via a hidden magnetic structure. An airflow channel can be incorporated inside the housing as a ventilation structure, complemented by a built-in small cooling fan to enhance heat dissipation. Alternatively, a modular splicing structure can be adopted, with the side panels joined together using mortise and tenon joints to form a complete housing, and louvered ventilation openings on the back panel as a ventilation structure. This application does not limit the specific structure of the housing packaging component, as long as it can achieve integrated packaging of the acquisition unit, processing unit, and human-machine interaction unit, and possesses detachable assembly and effective heat dissipation capabilities.

[0116] As an example, Figure 4A An example of a multi-spectral visual perception system provided in this application embodiment. Figure 1 , Figure 4B An example of a multi-spectral visual perception system provided in this application embodiment. Figure 2 , Figure 4C An example of a multi-spectral visual perception system provided in this application embodiment. Figure 3 , Figure 4D Figure 4 shows an example of a multi-spectral visual perception system provided in an embodiment of this application.

[0117] like Figure 4A and Figure 4B As shown in (1) and (2), the housing encapsulation assembly includes a top cover 1, a top cover plate 2, four side plates 3, and a bottom plate 7. The top cover 1 is provided with a first annular protrusion 1-1 and a second annular protrusion 1-2, with the second annular protrusion 1-2 located within the ring of the first annular protrusion 1-1. Correspondingly, the top cover plate 2 is provided with a groove, and an annular groove 2-3 adapted to the second annular protrusion 1-2 is provided within the groove. The second annular protrusion 1-2 and the annular groove 2-3 are interference-fitted to connect the top cover 1 and the top cover plate 2, and at this time, the outer wall of the first annular protrusion 1-1 abuts against the inner wall of the groove.

[0118] like Figure 4B As shown in (2), a first slot 2-1 is also provided in the groove of the top cover plate 2, correspondingly, as Figure 4CAs shown, each side plate 3 has a first buckle 3-1 at its top that matches the first slot 2-1. The first slot 2-1 and the first buckle 3-1 are interference-fitted to connect the side plate 3 to the top cover plate 2. A second buckle 3-4 is also provided at the bottom of the side plate 3. Correspondingly, the bottom plate 7 has a slot (not shown in the figure) that is interference-fitted with the second buckle 3-4. Furthermore, in this example, elongated ventilation holes 3-2 are evenly distributed on two opposite side plates 3, and one of the remaining side plates 3 has a reserved hole for the acquisition device.

[0119] like Figure 4C and 4D As shown, in this example, the side plate 3 is also provided with a right-angle limiting buckle 3-3, which is used to cooperate with the display screen 9 included in the human-machine interaction unit. Specifically, the right-angle limiting buckle 3-3 has a hole that connects to the copper pillar on the display screen 9. In addition, in this example, the display screen 9 is set on the side plate 3 on the side away from the acquisition unit, and the side plate 3 has a reserved display hole adapted to the display screen 9. The upper surface of the base plate 7 is also provided with a limiting side plate 7-1, and the display screen 9 is connected to the limiting side plate 7-1 through the copper pillar 8. Correspondingly, as Figure 4B As shown, a limiting plate 2-2 adapted to the display screen 9 is also provided on the top cover plate 2.

[0120] like Figure 4D As shown, in this example, the acquisition unit includes two visible light cameras (first visible light camera 4 and second visible light camera 6), one infrared camera 5, and an image acquisition card 11. The input end of the image acquisition card 11 is connected to the output end of the three cameras, and the output end is connected to the processing unit 15. In this example, the dynamic field of view of the acquisition device is achieved through a dynamic field of view adjustment structure. This dynamic field of view adjustment structure includes a camera mounting plate 10 fixed on a base plate. The camera mounting plate 10 has three equally spaced parallel sliding grooves 10-1. The two visible light cameras 4 and 6 and the infrared camera 5 are respectively mounted on the parallel sliding grooves 10-1 via 360° rotating gimbals. By translating the camera along the parallel sliding grooves 10-1, the horizontal position of the acquisition device can be adjusted. By rotating the gimbal, the acquisition angle of the camera can be adjusted. The two work together to achieve dynamic adjustment of the acquisition field of view, adapting to the acquisition needs of different scenarios.

[0121] Specifically, in this example, a mounting bracket 7-2 is provided on the upper surface of the base plate 7. The processing unit 15 and the battery 16 are both housed within this mounting bracket 7-2. The camera mounting plate 10 is fixedly mounted above the mounting bracket 7-2 via a slot and a buckle, achieving a fixed connection with the base plate 7 and positioned above the processing unit 15 and the battery 16. The image acquisition card 11 is located outside the mounting bracket 7-2 near the processing unit. The upper surface of the camera mounting plate 10 also has a connecting groove 10-2. The image acquisition card 11 is connected to the connecting groove 10-2 on the camera mounting plate 10 via an acquisition card connecting plate 12 and a support sleeve 13, and is secured with nuts and washers. In this example, the acquisition card connecting plate 12 is arc-shaped and rotatably connected to the image acquisition card 11, supporting the movement of the support sleeve 13 along the connecting groove 10-2.

[0122] In this example, the base plate 7 is provided with a limiting groove (not shown in the figure) and a fixing hole (not shown in the figure) for fixing the processing unit 15 and the battery 16. In addition, in this example, a pressure block 14 is provided above the processing unit 15 to further limit the processing unit 15 and prevent it from vibrating. The top of the pressure block 14 abuts against the bottom of the camera mounting plate 10.

[0123] This application also provides a multi-spectral fusion sensing method. Figure 5 This is a flowchart illustrating a multi-spectral visual perception method provided in an embodiment of this application. The following is a summary of the process. Figure 5 The method of this embodiment will be described in detail. For example... Figure 5 As shown, the method in this embodiment includes:

[0124] S501 acquires multispectral image data in the task execution environment.

[0125] The multispectral image data includes visible light images and infrared images.

[0126] Specifically, in this embodiment, the acquisition unit uses acquisition devices to acquire visible light and infrared images of the task execution environment, respectively. The acquired image data is then transmitted to the processing unit via an image acquisition card. Before acquisition, the location of the acquisition device must be determined, system parameters initialized, and the acquisition device ensured to be in an initial state adapted to the current task scenario.

[0127] For more specific details regarding the task execution environment, multispectral image data, etc., please refer to the aforementioned system implementation examples, which will not be repeated here.

[0128] S502, input multispectral image data into the human eye vision model to obtain a multispectral fused image, and input the multispectral fused image into the machine vision model to perform a preset perception task to obtain a perception result.

[0129] The human vision model and the machine vision model are obtained through joint training. The joint training introduces a gradient matching regularization term, which constrains the parameter update direction of the human vision model and the machine vision model, so that the parameters of the two models are optimized synchronously.

[0130] In this embodiment, the multispectral image data is first preprocessed by the processing unit. Preprocessing includes image denoising, size unification, viewpoint alignment, and pixel value normalization. After preprocessing, the preprocessed multispectral image data is input into a human visual model. This model extracts texture features from the visible light image and thermal radiation features from the infrared image through a feature extraction subnetwork, and then dynamically weights and fuses them through an attention fusion subnetwork to output a multispectral fused image. The fused image is then input into a machine vision model. The machine vision model extracts semantic features through operations such as convolution and pooling, performs a preset perception task, and outputs the results.

[0131] It should be understood that the joint training process requires first constructing a training set containing a large number of visible light images, infrared images, and corresponding ground truth values ​​for the perception task. During training, specifically, the training data is first input into the human visual model to generate a training fused image, and then the training fused image is input into the machine vision model to obtain the prediction result. The fusion loss is calculated based on the training fused image and the original image, and the perception task loss is calculated based on the prediction result and the ground truth value. The gradients of the first parameter of the machine vision model and the second parameter of the human visual model are calculated according to the two types of losses, respectively. The difference in the directions of the two gradients is calculated using negative cosine similarity to generate a gradient matching regularization term. The gradient matching regularization term and the perception task loss are weighted and summed to construct a joint loss function. An optimizer is then used to synchronously update the parameters of both models based on this function until the loss converges.

[0132] It should be understood that the preset perception tasks can be flexibly set according to the application scenario. For example, in the battery electrode defect detection task, the preset perception task is "electrode microcrack and bubble recognition". The machine vision model will locate and classify the defect area in the multi-spectral fusion image. The perception results include the defect type (microcrack / bubble), defect location coordinates, and defect area size. In the industrial equipment inspection task, the preset perception task is "equipment component damage and overheating detection". The perception results include the name of the damaged component (such as bearing / line), the degree of damage (mild / moderate / severe), the location of the overheated component, and the temperature value.

[0133] Furthermore, the human vision model and the machine vision model can use any preset algorithm model. For example, the human vision model can use a CNN-based fusion model, and the machine vision model can use a YOLO object detection model or a U-Net semantic segmentation model.

[0134] In the multi-spectral fusion sensing method provided in this embodiment, multi-dimensional multi-spectral image data is acquired, and the data quality is improved by preprocessing. Then, the fusion and perception are achieved by using a jointly trained dual model. The gradient matching regularization term ensures that the two models are optimized in coordination, taking into account both the human visual effect and machine perception accuracy of the fused image. It also supports flexible selection of training mode and model algorithm to adapt to different task scenarios.

[0135] It should be understood that the method of this application relies on the hardware support of a multi-spectral visual perception system to realize a closed loop of the entire process from data acquisition, processing, model calculation to result output, reducing the dependence on external equipment and computing power. It can be widely used in scenarios such as inspection robots, assisted driving, drone monitoring, and production line quality inspection, providing an effective solution for collaborative perception in complex environments.

[0136] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0137] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0138] The above embodiments introduce a multi-spectral visual perception method from the perspective of process flow. The following embodiments introduce a multi-spectral visual perception device from the perspective of virtual module or virtual unit. For details, please refer to the following embodiments.

[0139] This application also provides a multi-spectral visual sensing device for implementing the methods described in the above method embodiments. Figure 6 This is a schematic diagram of the structure of a multi-spectral visual sensing device provided in an embodiment of this application, as shown below. Figure 6 As shown, in this embodiment, the multi-spectral visual sensing device may include:

[0140] Acquisition module 61 is used to acquire multispectral image data in the task execution environment; the multispectral image data includes visible light images and infrared images;

[0141] The processing module 62 is used to input the multispectral image data into the human visual model to obtain a multispectral fused image when the multispectral image data is acquired, and to input the multispectral fused image into the machine vision model to perform a preset perception task to obtain a perception result. The human visual model and the machine vision model are obtained through joint training. The joint training introduces a gradient matching regularization term, which constrains the parameter update direction of the human visual model and the machine vision model, so that the parameters of the two models are optimized synchronously.

[0142] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0143] This application provides an electronic device. Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 7 As shown, Figure 7 The illustrated electronic device includes at least one processor 71 and a memory 72. The processor 71 and the memory 72 are connected, for example, via a bus 73. Optionally, the electronic device may also include a transceiver 74. It should be noted that in practical applications, the transceiver 74 is not limited to one, and the structure of this electronic device does not constitute a limitation on the embodiments of this application.

[0144] Processor 71 may be a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 71 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0145] Bus 73 may include a pathway for transmitting information between the aforementioned components. Bus 73 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 73 may be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0146] The memory 72 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0147] The memory 72 stores computer execution instructions for implementing the scheme of this application, and the processor 71 controls the execution. The processor 71 executes the computer execution instructions stored in the memory 72 to implement the content shown in the foregoing method embodiments.

[0148] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Specifically, the computer-readable storage medium stores computer-executable instructions, which are used to implement the methods in the above embodiments.

[0149] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the technical solution of the above method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0150] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0151] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0152] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A multi-spectral visual perception system, characterized in that, The system includes: The acquisition unit is used to acquire multi-spectral image data in the task execution environment; the multi-spectral image data includes visible light images and infrared images. The processing unit is configured to, upon acquiring the multispectral image data, input the multispectral image data into a human visual model to obtain a multispectral fused image, and input the multispectral fused image into a machine vision model to perform a preset perception task to obtain a perception result; wherein, the human visual model and the machine vision model are obtained through joint training, and the joint training introduces a gradient matching regularization term, which constrains the parameter update direction of the human visual model and the machine vision model, so that the parameters of the two models are optimized synchronously.

2. The system according to claim 1, characterized in that, The system further includes a human-computer interaction unit, which is used to receive training instructions input by the user and send the training instructions to the processing unit; the training instructions are joint optimization instructions or non-joint optimization instructions. Accordingly, the processing unit is further configured to: upon receiving the joint optimization instruction, jointly train the human visual model and the machine visual model based on the joint optimization instruction; and upon receiving the non-joint optimization instruction, independently train the human visual model and the machine visual model.

3. The system according to claim 1 or 2, characterized in that, The processing unit is specifically used for: The multispectral image data is preprocessed, and the preprocessed multispectral image data is input into the human visual model to obtain the multispectral fused image; wherein, the human visual model fuses the visible light color texture and infrared thermal radiation features in the preprocessed multispectral image data to obtain the multispectral fused image. And / or, when jointly training the human visual model and the machine vision model, the visible light image and infrared image used for training are input into the human visual model to obtain the training fused image; the training fused image is input into the machine vision model to obtain the perception task prediction result; Based on the training fused image and the task prediction result, the gradient matching regularization term is generated, and a joint loss function is constructed based on the gradient matching regularization term; Based on the joint loss function, the parameters of the human vision model and the machine vision model are updated synchronously.

4. The system according to claim 3, characterized in that, The processing unit is specifically used for: Based on the training fused image and the visible light image and infrared image used for training, calculate the fusion loss; based on the perception task prediction result and the corresponding perception task ground truth, calculate the perception task loss. The gradient of the first parameter is calculated based on the perception task loss, the gradient of the second parameter is calculated based on the fusion loss, and the gradient matching regularization term is generated by calculating the directional similarity between the gradient of the first parameter and the gradient of the second parameter. The joint loss function is constructed based on the gradient matching regularization term and the perception task loss.

5. The system according to claim 4, characterized in that, The processing unit is also used for: When generating the gradient matching regularization term, the weighting coefficients of the first parameter gradient and the second parameter gradient are dynamically determined based on the attention mechanism.

6. The system according to claim 1 or 2, characterized in that, The acquisition unit includes at least two image acquisition devices and an image acquisition card; the input terminal of the image acquisition card is connected to the output terminal of the at least two image acquisition devices, and the output terminal is connected to the input terminal of the processing unit. The at least two image acquisition devices are used to acquire the visible light image and the infrared image respectively, and are also used to change the field of view range by adjusting their own posture or position; The image acquisition card is used to receive multi-spectral image data output by the at least two image acquisition devices and transmit the multi-spectral image data to the processing unit.

7. The system according to claim 6, characterized in that, The image acquisition card and the processing unit are connected via a slot.

8. The system according to claim 1 or 2, characterized in that, The system further includes a housing encapsulation assembly for integrating and encapsulating the acquisition unit and the processing unit; wherein the housing encapsulation assembly adopts a detachable connection structure and is provided with a ventilation structure for heat dissipation.

9. A multi-spectral visual perception method, characterized in that, The method includes: Acquire multispectral image data in the task execution environment; the multispectral image data includes visible light images and infrared images; The multispectral image data is input into a human visual model to obtain a multispectral fused image, and the multispectral fused image is input into a machine vision model to perform a preset perception task to obtain a perception result; wherein, the human visual model and the machine vision model are obtained through joint training, and the joint training introduces a gradient matching regularization term, which constrains the parameter update direction of the human visual model and the machine vision model, so that the parameters of the two models are optimized synchronously.

10. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory to implement the method as described in claim 9.

Images