ISP (Internet Service Provider) autonomous selection aid decision-making method and device for multi-sensor data source

Abstract

The invention provides a multi-sensor data source-oriented ISP autonomous selection aid decision-making method and device, and the method comprises the steps: an initial step: receiving multi-modal image data collected by various types of sensors; extracting image features of each modal image data through a convolutional neural network adopting a weight sharing mechanism; the classifier identifies and obtains the sensor type of each modal image data according to the image features; a selection and optimization step: selecting an ISP module corresponding to the sensor type to process modal image data under the sensor type to obtain a processing result output by each ISP module, and adjusting the ISP module according to the type of the input data and the current environment to optimize the processing result to obtain an optimization result; and screening an image conforming to a specific application scene type from the optimization result as an image processing result.

Description

Technical Field

[0001] The present invention relates to the technical field of combining image signal processing (ISP) and AI-assisted decision-making, and particularly relates to an ISP autonomous selection and AI-assisted decision-making method, device, electronic device, computer-readable storage medium, and computer program product for multi-sensor data sources. Background Art

[0002] With the increasing dependence of industrial intelligent devices in more and more application scenarios on multi-sensor technology, different types of sensors, such as visible light sensors, infrared sensors, and 3D sensors, can provide complementary sensing information and provide more comprehensive and accurate environmental data for the system in harsh environments, low-light conditions, or complex scenarios. However, there are significant differences in the data characteristics output by different sensors. Traditional image signal processors (ISPs) are usually designed for a single type of sensor and cannot adaptively process data from multiple sensors. For example, a visible light ISP module is usually responsible for operations such as demosaicking, white balance, and noise suppression, and is specifically used to process data from visible light sensors. Infrared ISP, on the other hand, focuses more on dynamic range adjustment and suppression of specific noise. However, in a multi-sensor scenario, each sensor requires an independent ISP module to process its data, which greatly increases the complexity and cost of the system.

[0003] In the existing technology, a single image signal processor (ISP) module is usually used to process data from multi-source sensors. For example, one ISP processes multiple visible light sensors and synchronously processes the raw data collected by multiple visible light sensors. The core idea of this method is to synchronously process the raw data collected by different sensors through a unified ISP processing path, so as to achieve synchronous output of images from multiple sensors. This type of method solves the problem of parallel processing of multi-sensor data to a certain extent and can reduce the redundancy of various data processing paths. However, since these ISP modules are usually designed with hardware fixation, they can only provide limited image processing algorithms, such as black level compensation, lens correction, white balance adjustment, color interpolation, noise removal, gamma correction, etc. Their processing capabilities and flexibility are relatively low, and it is difficult to dynamically adjust according to the data characteristics of different sensors. Especially in complex application scenarios, they lack automation capabilities and cannot meet the diverse requirements of different sensor data sources for processing algorithms.

[0004] To solve this problem, the prior art has proposed an ISP adaptive adjustment control method, which adjusts the ISP according to image analysis parameters until the parameters of the ISP meet the preset conditions, so as to achieve real-time and accurate face image acquisition. However, this method mainly relies on a single visible light sensor data source and is difficult to comprehensively support multiple sensor scenarios. In addition, with the development of artificial intelligence (AI) technology, AI-assisted ISP optimization technology has gradually been applied to multi-sensor data processing. Through deep learning algorithms, AI can intelligently analyze the sensor input data and dynamically adjust the parameters of the ISP to optimize the image quality. For example, AI can perform adaptive optimization on processing processes such as noise suppression, color correction, and image enhancement. However, most of the existing AI-assisted ISP optimization schemes are for single-sensor data, and their application in multi-sensor adaptive ISP selection is still in its infancy.

[0005] Although multi-sensor technology and image signal processors (ISPs) have been widely used in the fields of image processing and intelligent perception, there are still many problems in the prior art for processing multi-sensor data sources, which cannot meet the application requirements of diverse sensor data sources and complex environments. Specifically, the prior art has the following main problems and disadvantages:

[0006] 1) Unable to automatically identify and match sensor types: Existing multi-sensor systems usually require manual configuration of the corresponding ISP module for each type of sensor, lacking automation capabilities. In practical applications, the device may be connected to different types of sensors (such as visible light, infrared light, 3D sensors, etc.), but the prior art cannot automatically identify and dynamically select the appropriate ISP module for processing according to the sensor type. For example, traditional multi-sensor systems can only manually select the ISP in a predefined manner and cannot be automatically recognized by the adaptive processor. This manual configuration method lacking flexibility significantly increases the complexity of the system and is difficult to meet the requirements of efficient processing in real-time scenarios.

[0007] 2) Single ISP module design and lack of adaptability: Existing ISP systems often optimize for a single type of sensor. For example, an ISP module specifically designed for visible light image processing is difficult to be compatible with the data processing of infrared or 3D sensors. This means that the system cannot achieve effective adaptive processing when dealing with multiple sensor inputs. Although some solutions attempt to integrate multiple ISP modules to process different types of sensor data respectively, most of these modules are statically configured and cannot be dynamically adjusted according to the different characteristics of the sensor inputs, which limits the flexibility and scalability of the system in a multi-sensor environment.

[0008] 3) Insufficient application of AI-assisted optimization technology: In recent years, although AI technology has been gradually introduced into the field of image signal processing, such as for image enhancement, noise suppression, and color correction, most of these AI technologies are limited to the data optimization of a single sensor and cannot be comprehensively applied to ISP selection and processing in multi-sensor scenarios. Existing AI-assisted optimization schemes mainly process fixed sensor inputs and lack the flexibility to handle heterogeneous data sources (such as visible light and infrared light data), and have not achieved multi-sensor adaptive selection and full-process automation processing. Summary of the Invention

[0009] The present invention relates to the comprehensive processing of multiple sensor data sources, specifically including the data source processing of visible light sensors, non-visible light sensors (such as infrared), and 3D sensors. The invention aims to achieve efficient and intelligent processing of multiple sensor data through adaptive ISP selection and AI-assisted decision optimization, meeting the different requirements of multi-application scenarios for image processing effects.

[0010] The present invention aims to solve the technical problems in the current multi-sensor data source processing system, especially the deficiencies in sensor automatic identification, ISP module selection, and multi-sensor heterogeneous data processing. When accessing multiple different types of sensors (such as visible light, infrared, 3D sensors), the existing technology cannot automatically identify the sensor type and dynamically match a suitable ISP module, resulting in poor flexibility and real-time performance of the system. The sensor automatic identification mechanism proposed by the present invention can achieve dynamic detection and adaptive ISP selection. In addition, the traditional ISP module design usually targets a single type of sensor and cannot flexibly handle the data characteristics of different sensors. The present invention designs a multi-mode ISP architecture that can dynamically select a suitable ISP module according to the type of sensor, solving the limitation of single processing ability in the existing technology. Further, the present invention also introduces an AI-assisted decision-making engine, which uses deep learning technology to automatically predict and select the optimal ISP processing path, achieving intelligent optimization at the system level, solving the problem that the application of AI technology is limited to a single task, and thus significantly improving the automation level and processing efficiency of the system. This invention greatly enhances the intelligent, automated, and efficient processing capabilities of multi-sensor systems in complex environments, providing technical support for their wide application in scenarios such as autonomous driving, intelligent monitoring, and intelligent detection.

[0011] Aiming at the deficiencies of the existing technology, such as Figure 3 The present invention proposes an ISP autonomous selection and auxiliary decision-making method for multi-sensor data sources, which includes:

[0012] Initial step: Receive multi-modal image data collected by multiple types of sensors; extract image features of each modal image data through a convolutional neural network adopting a weight sharing mechanism; the classifier identifies the sensor types of each modal image data based on the image features.

[0013] Selection and optimization step: Select the ISP module corresponding to the sensor type to process the modal image data of this sensor type, obtain the processing results output by each ISP module, adjust the ISP module according to the type of input data and the current environment to optimize the processing results, and obtain the optimized results; screen the images that meet the specific application scenario type from the optimized results as the image processing results.

[0014] The ISP autonomous selection and auxiliary decision-making method for multi-sensor data sources, wherein the multi-modal image data includes: visible light image data collected by an RGB sensor, and / or infrared image data collected by an infrared sensor, and / or depth point cloud image data collected by a 3D sensor.

[0015] The ISP autonomous selection and auxiliary decision-making method for multi-sensor data sources, wherein in the convolutional neural network, the weight sharing mechanism is used to extract the basic features of each modal data; after extracting the basic features, convolutional kernels are designed respectively for each sensor modality to further extract the high-level features of each sensor type as the image features.

[0016] The ISP autonomous selection and auxiliary decision-making method for multi-sensor data sources, wherein the ISP module includes a visible light ISP module, an infrared ISP module, and an AI ISP module;

[0017] In the selection and optimization step, optimizing the processing results specifically includes:

[0018] According to the image feature F of the sensor input data i and the current environmental characteristic C env , optimize the processing parameter P of the current ISP module init , and obtain the optimized processing parameter P that can more accurately adapt to the current working environment opt ; P opt = Optimize(P init , F i , C env ).

[0019] As Figure 4 shown, the present invention also proposes an ISP autonomous selection and auxiliary decision-making device for multi-sensor data sources, which includes:

[0020] Initial module, which receives multi-modal image data collected by multiple types of sensors; extracts image features of each modal image data through a convolutional neural network adopting a weight sharing mechanism; and a classifier identifies the sensor types of each modal image data according to the image features.

[0021] Selection and optimization module, which selects an ISP module corresponding to the sensor type to process the modal image data under the sensor type, obtains the processing results output by each ISP module, adjusts the ISP module according to the type of input data and the current environment to optimize the processing results, and obtains optimized results; filters out images that conform to the specific application scenario type from the optimized results as the image processing results.

[0022] The ISP autonomous selection and auxiliary decision-making device for multi-sensor data sources, wherein the multi-modal image data includes: visible light image data collected by an RGB sensor, and / or infrared image data collected by an infrared sensor, and / or depth point cloud image data collected by a 3D sensor.

[0023] The ISP autonomous selection and auxiliary decision-making device for multi-sensor data sources, wherein in the convolutional neural network, the weight sharing mechanism is used to extract the basic features of each modal data; after extracting the basic features, convolutional kernels are designed for each sensor modality respectively to further extract the high-level features of each sensor type as the image features.

[0024] The ISP module includes a visible light ISP module, an infrared ISP module, and an AI ISP module.

[0025] In the selection and optimization module, optimizing the processing results specifically includes:

[0026] According to the image feature F of the sensor input data i and the current environmental characteristic C env , optimize the current ISP module

[0027] 's processing parameter P init , and obtain optimized processing parameter P that can more accurately adapt to the current working environment opt ; P opt =Optimize(P init ,F i ,C env ).

[0028] The present invention also proposes an electronic device, which includes the ISP autonomous selection and auxiliary decision-making device for multi-sensor data sources described above. The electronic device is either connected to an information display device, and the information display device is used to display the image processing results with display parameters, attributes set by the user, or through an artificial intelligence model.

[0029] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the ISP autonomous selection and auxiliary decision-making method for multi-sensor data sources are implemented.

[0030] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the ISP autonomous selection and auxiliary decision-making method for multi-sensor data sources are implemented.

[0031] As can be seen from the above solutions, the advantages of the present invention are as follows:

[0032] 1) Improvement in system automation level: By introducing a sensor automatic recognition mechanism, the present invention realizes the autonomous recognition and adaptive ISP selection after the access of multi-sensor data sources, reduces the complexity of manual configuration and operation, and greatly improves the automation level of the system.

[0033] 2) Substantial improvement in processing efficiency: The prior art usually adopts static ISP configuration and optimizes the processing for a single sensor, and cannot flexibly handle multi-sensor input. The multi-mode ISP architecture of the present invention can dynamically switch different ISP modules according to the sensor type, so as to achieve the optimal data processing path selection in the multi-sensor scenario, and significantly improve the processing efficiency of the system.

[0034] 3) Improvement in intelligence level and system performance: By introducing an AI-assisted decision-making engine, the present invention enables the system to have the ability of self-learning and optimization. The AI engine can automatically select the best ISP module according to the type of sensor input and environmental conditions, and continuously adjust the processing parameters to achieve global optimization decision-making.

[0035] 4) Reduction in system complexity and cost: In the prior art, it is usually necessary to integrate multiple independent ISP modules to process different types of sensor data respectively, which not only increases the hardware complexity of the system, but also significantly increases the system cost. Through the adaptive ISP architecture design of the present invention, the need for redundant configuration of multiple modules is avoided, and the processing of multiple sensor inputs through a unified platform is realized. This design greatly simplifies the hardware architecture, reduces the waste of hardware resources, reduces the system cost, and at the same time improves the overall energy efficiency ratio, making the system more competitive in application scenarios with limited resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is an architecture diagram of ISP autonomous selection and AI-assisted decision-making for multi-sensor data sources;

[0037] Figure 2 is a flowchart;

[0038] Figure 3 This is the flowchart of the method of the present invention;

[0039] Figure 4 This is the module diagram of the device of the present invention;

[0040] Figure 5 This is the schematic structural diagram of the first electronic device of the present invention;

[0041] Figure 6 This is the schematic structural diagram of the application environment of the first electronic device of the present invention;

[0042] Figure 7 This is the schematic structural diagram of the second electronic device of the present invention.

[0043] Reference numerals:

[0044] A - The first electronic device;

[0045] B - ISP independent selection assistance decision-making device for multi-sensor data sources;

[0046] C - Data acquisition device;

[0047] D - Information display device;

[0048] 1000 - The second electronic device;

[0049] Ⅰ - Computing unit;

[0050] Ⅱ - ROM;

[0051] Ⅲ - RAM;

[0052] Ⅳ - Bus;

[0053] Ⅴ - Interface;

[0054] Ⅵ - Input unit;

[0055] Ⅶ - Output unit;

[0056] Ⅷ - Storage medium;

[0057] Ⅸ - Communication unit. Detailed implementation manners

[0058] It should be noted that in this application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0059] Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0060] The processor described in the present invention is the control center of the electronic device, which may be a single processor or a collective term for multiple processing elements. For example, it may be one or more central processing units (CPUs), or it may be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as: one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0061] Optionally, the processor can execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0062] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, the processor may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). The electronic device may include: servers, desktop computers, laptop computers, smart phones, tablet computers, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0063] The memory is used to store the software program for implementing the solution of the present invention and is controlled by the processor for execution. The specific implementation manner may refer to the above method embodiments and will not be elaborated here.

[0064] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not limit it. The actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or combine some components, or have different component arrangements.

[0065] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access, or a data storage device such as a server or data center that contains one or more collections of available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0066] It should also be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be understood specifically with reference to the context.

[0067] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0068] It should also be understood that in various embodiments of the present invention, the magnitudes of the serial numbers of the above processes do not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0069] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0070] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0071] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0072] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs and other various media that can store program codes.

[0073] When conducting research on intelligent processing methods for multi-sensor data sources, the inventor discovered three major defects in the existing technology, namely the inability to automatically identify and match sensor types, the single and inflexible design of the ISP module, and the insufficient application of AI-assisted optimization technology. These problems limit the flexibility and processing efficiency of multi-sensor systems. Through in-depth research, the inventor found that the root cause of these defects lies in the fact that most existing technologies only consider the processing requirements of a single sensor during design, without fully considering the data heterogeneity and complexity in a diverse sensor environment. In addition, the existing ISP system architecture is too fixed and lacks the ability to adaptively process dynamic inputs, while existing AI technologies are limited to single-task optimization and are difficult to play a role in global processing. To solve the above technical problems, the inventor proposed an ISP autonomous selection and AI-assisted decision-making method for multi-sensor data sources. This design includes the following key parts:

[0074] Technical Difficulty 1: Inability to Automatically Identify and Match Sensor Types

[0075] The identification of sensor types and ISP selection in existing technologies mostly rely on manual configuration or static system design. This approach has low processing efficiency when faced with diverse sensor data. The inventor found through research that the root cause of this problem is the lack of an automatic identification mechanism for diverse inputs in the system. Existing systems are usually designed for a specific type of sensor and are difficult to dynamically adapt to the data inputs of other sensors. This limitation stems from the fixed thinking in the design architecture, and traditional ISP modules fail to consider the automatic adaptability after multi-sensor access. To solve this problem, the inventor proposed a method based on an automatic identification and matching mechanism. By introducing an identification layer for multi-sensor interfaces in the system, it can automatically select the corresponding ISP module according to the accessed sensor type, thus achieving fully autonomous sensor data processing. The key to this identification layer lies in that its design needs to consider the output characteristics of different sensors (such as signal frequency, data format, dynamic range, etc.) and use an adaptive algorithm to determine the optimal ISP module.

[0076] Technical Difficulty 2: The ISP Module Design is Single and Lacks Adaptability

[0077] The architectures of existing ISP systems are often optimized for single tasks, that is, only one type of sensor data processing is considered during design. Through research, the inventors found that the difficulty in solving this problem lies in how to design a multi-sensor adaptive ISP architecture that can handle the data processing requirements of different sensors. In the prior art, multi-sensor processing usually relies on the integration of multiple independent ISP modules, which increases the complexity of the system and has low processing efficiency. The inventors realized that the key lies in breaking the traditional static ISP design concept and proposing a multi-mode ISP architecture that can dynamically adjust according to the input characteristics of sensors. By introducing different types of ISP modules (such as visible light ISP, non-visible light ISP, AI ISP, etc.), each module can automatically load and unload the corresponding processing flow according to the sensor type, thus achieving adaptive processing of multiple sensor data at the system level. Through a large number of experiments, the inventors found that this adaptive architecture can significantly improve the processing flexibility and scalability of multi-sensor systems.

[0078] Technical Difficulty 3: Insufficient application of AI-assisted optimization technology

[0079] Although AI technology has been applied in the field of image processing optimization, most of the existing systems only use AI for single-task optimization and do not effectively apply it to ISP selection and global processing in multi-sensor systems. Through in-depth research, the inventors found that the potential of AI lies not only in improving the processing ability of a single ISP, but more importantly, in achieving dynamic matching and global optimization of multi-sensor data through the decision-making ability of AI. However, the existing AI technology is mainly used for image enhancement and optimization in single tasks and has not been widely applied to the entire process of ISP selection. The inventors proposed that the key role of AI is to predict and select the optimal ISP processing flow by learning the characteristics of different sensor data. For this purpose, the inventors designed an AI-assisted decision-making engine, which can monitor the input characteristics of different sensors in real time and make the best ISP selection decision by combining machine learning algorithms and big data analysis, thereby improving the intelligence level and processing efficiency of the entire system. This engine can make decisions at the system level rather than being limited to single ISP optimization tasks, thus significantly improving the automation level of the system.

[0080] The present invention achieves the above technical effects by proposing the following key technical points:

[0081] Key Point 1: Sensor Autonomous Recognition and Adaptive ISP Selection Mechanism. An important innovation of the present invention is the proposal of a sensor autonomous recognition and adaptive ISP selection mechanism. This mechanism designs different processing paths for different types of sensor data (such as RGB images, grayscale images, and 3D depth maps, etc.) through a multi-modal feature extraction model based on convolutional neural network (CNN), extracts the feature information of various sensors, realizes the automatic recognition of sensor types, and at the same time improves the efficiency of feature extraction through a weight sharing mechanism. Secondly, according to this feature information, it automatically, quickly, and accurately identifies the sensor type (such as visible light sensor, infrared sensor, or 3D sensor) and adaptively selects the ISP processing path that matches it. For example, visible light data uses the visible light ISP processing path, infrared data uses the non-visible light ISP, and 3D data uses the AI ISP module, etc. The whole process is completely automated without manual intervention, which can significantly improve the flexibility and automation level of the system, and ensure that various sensor data can be optimally processed in complex environments. The sensor data is preprocessed through feature extraction, and a suitable ISP module is dynamically selected for subsequent processing. Its corresponding technical effect is that the automation level of the multi-sensor system is greatly improved, the complexity of manual configuration and switching is reduced, and it can adapt to the data processing requirements of diverse sensors in real time, especially suitable for complex and changing application scenarios, such as occasions that require real-time response like autonomous driving, intelligent monitoring, and intelligent detection, etc.

[0082] Key Point 2: Multi-Mode ISP Architecture Design. The present invention also designs a multi-mode ISP architecture. As Figure 1 shown, independent ISP processing modules are set up respectively for different types of sensor data sources (such as visible light, non-visible light, 3D data). These modules are specifically optimized according to the data characteristics of the sensors. For example, the visible light ISP module mainly processes denoising, white balance, gamma correction, color correction, etc., while the infrared ISP module focuses on infrared correction, depth map generation, timing control, etc. In addition, the system also designs an AI ISP module, which is specifically used for AI inference and processing optimization in specific scenarios, including AI white balance, AI color correction, AI denoising, AI dynamic adjustment, and AI edge enhancement. When a sensor is connected, the system can automatically select a suitable data processing path according to the type of the sensor and switch different ISP modules when needed. This design greatly enhances the adaptability of the system to the multi-sensor environment. Its technical effect is that by dynamically selecting and switching different ISP modules, the system can efficiently process data from different sensors, ensure that each sensor can obtain the most optimized processing result, thereby improving the overall image quality and data processing performance.

[0083] Key Point 3: AI-Assisted Decision Engine. The present invention introduces an AI-assisted decision engine. This part utilizes deep learning and big data technologies to analyze and predict the data characteristics of different sensors. The AI engine can automatically select the most suitable ISP module based on the data type input by the sensor, environmental conditions, and historical processing records, and adjust and optimize the processing parameters. This approach not only improves the processing efficiency of the system but also enables the system to self-regulate according to environmental changes, thus adapting to different working scenarios. In addition, the AI engine can continuously learn and improve, further optimizing the performance of the system during subsequent use. Its technical effect is to achieve intelligent ISP selection and parameter optimization in a multi-sensor environment, ensuring that the system can make the fastest and optimal processing decisions in complex and changing environments, greatly enhancing the real-time performance and processing efficiency of the system.

[0084] To make the above features and effects of the present invention more clearly and understandably described, specific embodiments are hereinafter given and will be described in detail in conjunction with the accompanying drawings of the specification. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely for illustrative purposes. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the appended claims.

[0085] The overall technical solution of the present invention realizes the intelligent processing of multi-sensor data sources through multi-modal feature extraction based on convolutional neural network (CNN), automatic sensor identification, adaptive selection of ISP modules, and AI-assisted optimization processing. The overall architecture diagram is as Figure 1 shown, and the following is the detailed implementation process based on this method:

[0086] Step 1, Obtain the input of sensor data:

[0087] Obtain input data from multiple sensors, which may include RGB sensors, infrared sensors, and 3D sensors. The data is transmitted to the system through a standard interface (such as MIPI D-PHY interface) or LVDS interface for processing. The input data is as follows:

[0088] RGB image: (Three channels: red, green, blue)

[0089] Infrared image: (Single-channel grayscale image)

[0090] 3D depth map or point cloud: or (Three-dimensional points)

[0091] Step 2, Multi-modal feature extraction based on convolutional neural network (CNN):

[0092] In a convolutional neural network, the weight sharing mechanism is used to extract the basic features (such as edges, simple textures, etc.) of different modality data. At this time, RGB, infrared, and 3D data are all processed through the same convolutional kernel to extract the shared basic features. This mechanism effectively reduces the number of parameters and improves the computational efficiency. The weight sharing convolution formula is as follows:

[0093]

[0094] Among them, D sensor (i, j, c) represents the pixel value of the input data, K shared (i, j) is the convolutional kernel, σ is the activation function (ReLU), and F shared is the feature map of the convolutional output.

[0095] After extracting the basic features, convolutional kernels are designed for each sensor modality respectively to further extract the advanced features of each sensor type. These convolutional kernels focus on processing the data characteristics of specific modalities, such as the color information of RGB images, the brightness information of infrared images, and the spatial structure information of 3D data.

[0096] RGB image convolution:

[0097] F RGB = σ(F shared * K RGB )

[0098] Among them, the RGB convolutional kernel k RGB extracts features related to color and details.

[0099] Infrared image convolution:

[0100] F IR = σ(F shared * K IR )

[0101] Among them, the infrared convolutional kernel F IR processes the brightness and contrast features.

[0102] 3D image convolution:

[0103] F 3D = σ(F shared * K 3D )

[0104] Among them, the 3D convolutional kernel K 3D is dedicated to extracting the spatial structure and depth information.

[0105] Step 3, sensor type recognition:

[0106] After feature extraction, according to the feature data of different sensors, the classifier C is used to automatically identify the sensor type. The classifier can be a deep learning-based model that takes different feature maps as input and outputs the type label T of the sensor. i . The sensor identification formula is as follows:

[0107] T i = Classifier(F RGB , F IR , F 3D )

[0108] Step 4, Adaptive ISP Module Selection:

[0109] According to the sensor type identified in Step 3, the system needs to select a suitable ISP module. Common ISP modules include visible light ISP, infrared ISP, and AI ISP. Through MCM, the resource allocation, task switching, and parameter sharing between different ISP paths are managed and coordinated, and the data is transferred to the dedicated ISP module for processing.

[0110] 1) ISP Module Selection

[0111] Based on the sensor type T i , the system automatically selects a suitable ISP module using the selection logic. This selection can be represented by a decision function:

[0112] ISP i = SelectISP(T i )

[0113] Among them, ISP i represents the ISP module selected for the current sensor, and SelectISP is a function that dynamically allocates ISP modules according to the sensor type. Visible light data selects the visible light ISP, infrared data selects the non-visible light ISP, and 3D data selects the AI ISP module.

[0114] 2) Selection Conditions

[0115] To ensure that the correct ISP module is selected, the selection function makes a determination based on the following conditions:

[0116]

[0117] Ensure that different sensor types correspond to different ISP module processing paths.

[0118] Step 5, ISP Data Processing:

[0119] After selecting the appropriate ISP module, the system processes the sensor data D in .

[0120] 1) Data processing formula

[0121] Different ISP modules have different processing logics. For example, the visible light ISP module processes demosaicing and white balance adjustment, while the infrared ISP module focuses on dynamic range adjustment and noise reduction. The processing formula is as follows:

[0122] D out = ISP i (D in )

[0123] where D in is the data input by the sensor, ISP i represents the currently selected ISP module, and D out is the processed output data.

[0124] 2) Specific processing steps of ISP

[0125] Taking the visible light ISP module as an example, its processing process includes demosaicing, white balance adjustment, and color enhancement. The formula is as follows:

[0126] Demosaicing:

[0127] D demosaic = Demosaic(D in )

[0128] White balance adjustment:

[0129] D white = D demosaic × WB(C wb )

[0130] where WB is the white balance adjustment function and C wb is the white balance coefficient.

[0131] Color enhancement:

[0132] D enhanced = Enhance(D white )

[0133] The final output data D out optimizes the image quality through multiple processing steps.

[0134] = ISP i (D sensor )

[0135] Step 6, AI-assisted optimization:

[0136] In step 6, the AI-assisted decision-making engine processes the result D outFor further optimization, the parameter settings of the ISP module will be optimized in real time according to the type of input data and the current environment to ensure the adaptive processing of the system in different environments and the quality and processing efficiency of the output results. The specific optimization process is as follows:

[0137] AI model optimization:

[0138] The AI engine dynamically optimizes the processing parameters of the ISP according to the feature vector F of the sensor input data i , the initial processing parameter P init and the current environmental characteristics C env (such as lighting, noise, etc.). The optimized parameter P opt can more accurately adapt to the current working environment.

[0139] P opt = Optimize(P init , F i , C env )

[0140] Among them, P opt is the optimized processing parameter, ensuring that the ISP can make optimal processing for different environments and input data.

[0141] Adaptive adjustment process:

[0142] Through the AI engine, the system can adjust the processing parameters in real time according to environmental changes and data characteristics. For example, in low-light environments, the system will automatically increase the exposure adjustment:

[0143] P exposure = AdjustExposure(C env )

[0144] In addition to exposure adjustment, the AI engine can also dynamically adjust noise reduction, white balance, and sharpening parameters to ensure that the output data reaches the best quality. Ensure that the system performs intelligent optimization under different environmental conditions, improving the image processing quality and efficiency.

[0145] Parameter feedback and optimization loop:

[0146] The system evaluates the adjusted parameters through a continuous feedback mechanism and further optimizes them to ensure that the processing parameters can automatically adapt to new environmental or data changes after each operation.

[0147] The adjusted parameters can be evaluated by the following methods:

[0148] Image quality assessment: Objective image quality evaluation metrics (such as Peak Signal-to-Noise Ratio PSNR, Structural Similarity SSIM) are adopted. The images after AI optimization processing are compared with the images processed by traditional ISP to quantify the improvement in image quality.

[0149] Processing efficiency assessment: Processing latency (Latency) and throughput are used as evaluation metrics to verify the improvement effect of the ISP processing speed after AI-assisted optimization.

[0150] Adaptability assessment: Multiple typical environmental scenarios are designed for testing (such as low light, high dynamic range, high noise scenarios). The output quality and stability of the ISP before and after AI-assisted optimization are compared respectively to quantify the improvement in the system's ability to adapt to complex environments.

[0151] Resource utilization efficiency assessment: By calculating the resource consumption rate of the ISP module (such as memory occupancy rate, computing resource occupancy rate), the resource utilization before and after AI-assisted optimization is compared to reflect the technical advantage of this application in reducing system resource consumption.

[0152] Subjective user evaluation: A subjective user evaluation experiment (MOS) is adopted to score and analyze the image quality and visual comfort that users actually feel, reflecting the positive impact of the AI optimization scheme on users' actual applications.

[0153] Step 7, output the processing result:

[0154] Step 7 is responsible for outputting the data D after passing through the ISP module and AI-assisted optimization. out And determine the type of the output data according to the specific application scenario. For RGB data, it is output in image format; for 3D sensor data, it is output as point cloud or depth map; for infrared data, it is output as an enhanced thermal image. It can be used for subsequent applications such as image display, video processing, or 3D modeling.

[0155] Output data format:

[0156] The data format output by the system depends on the specific application scenario. For visible light sensors, the output is usually an RGB image; for 3D sensors, the output is a point cloud or depth map; for infrared sensors, the output is an enhanced thermal image.

[0157] D output = FormatOutput(D out )

[0158] Where FormatOutput is a formatted output function to ensure that the data meets the requirements of subsequent processing.

[0159] The following is a system embodiment corresponding to the method embodiment above. This embodiment can be implemented in cooperation with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. To avoid repetition, they will not be elaborated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0160] As Figure 4 shown, the present invention also proposes an ISP autonomous selection auxiliary decision-making device for multi-sensor data sources, which includes:

[0161] An initial module that receives multi-modal image data collected by multiple types of sensors; extracts image features of each modal image data through a convolutional neural network adopting a weight sharing mechanism; and a classifier identifies the sensor types of each modal image data based on the image features.

[0162] A selection and optimization module that selects an ISP module corresponding to the sensor type to process the modal image data of the sensor type, obtains the processing results output by each ISP module, adjusts the ISP module according to the type of input data and the current environment to optimize the processing results, and obtains an optimized result; screens out images that meet the specific application scenario type from the optimized result as the image processing result.

[0163] In the ISP autonomous selection auxiliary decision-making device for multi-sensor data sources, the multi-modal image data includes: visible light image data collected by an RGB sensor, and / or infrared image data collected by an infrared sensor, and / or depth point cloud image data collected by a 3D sensor.

[0164] In the ISP autonomous selection auxiliary decision-making device for multi-sensor data sources, in the convolutional neural network, the weight sharing mechanism is used to extract the basic features of each modal data; after extracting the basic features, convolutional kernels are designed for each sensor modality respectively to further extract the high-level features of each sensor type as the image features.

[0165] The ISP module includes a visible light ISP module, an infrared ISP module, and an AI ISP module.

[0166] In the selection and optimization module, optimizing the processing result specifically includes:

[0167] According to the image feature F of the sensor input data i and the current environmental characteristic C env , optimize the processing parameter P of the current ISP module init to obtain an optimized processing parameter P that can more accurately adapt to the current working environment opt ; P opt = Optimize(Pinit , F i , C env )。

[0168] As Figure 5 shown, in another embodiment of the present invention, a first electronic device A is further proposed, which includes the ISP autonomous selection auxiliary decision-making device for multi-sensor data sources described above.

[0169] As Figure 6 shown, the first electronic device A can also be connected to a data acquisition device C and an information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to acquire multi-modal image data, and the information display device D is used to display the image processing results obtained by the analysis of the present invention.

[0170] Among them, the information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data output by the first electronic device A. This information display mechanism can be preset manually. For example, the data output by the first electronic device A is visually displayed, and it can display according to the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. Present the key information specified by the user to the user, so that the user can understand this information more timely without having to access the secondary page or scroll the page, saving the user's operation. Or this information display mechanism can be an artificial intelligence AI display model, which can learn the key information of the user according to the user's previous usage habits, such as viewing duration, click times, editing times, etc., and then automatically present rich and necessary key information to the user.

[0171] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the ISP autonomous selection auxiliary decision-making method for multi-sensor data sources provided by the above various methods.

[0172] In another embodiment, the present invention further provides a storage medium VIII for storing a computer program for executing the ISP autonomous selection and auxiliary decision-making method for multi-sensor data sources. It should be understood that the storage medium in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0173] Figure 7 FIG. shows a schematic block diagram of a second electronic device 1000 that can be used to implement the embodiments of the present invention. The second electronic device 1000 is intended to represent various forms of digital computers, such as, for example, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 may also represent various forms of mobile devices, such as, for example, personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the present invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0174] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In the RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, the ROM II, and the RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0175] Multiple components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a magnetic disk, an optical disc, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0176] The computing unit I can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit I include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit I executes the various methods and processes described above, such as method steps S1 - S2. For example, in some embodiments, the method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM II and / or the communication unit IX. When the computer program is loaded into the RAM III and executed by the computing unit I, one or more steps of the method described above can be executed. Alternatively, in other embodiments, the computing unit I can be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0177] Although the embodiments of the present invention have been disclosed as above, it is not limited to only the applications listed in the specification and the embodiments. It can be fully applied to various fields suitable for the present invention. For those familiar with the field, additional modifications can be easily made. Therefore, without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to specific details and the illustrations shown and described herein.

Claims

1. An ISP autonomous selection auxiliary decision-making method for multi-sensor data sources, characterized in that Including: An initial step of receiving multi-modal image data collected by the multiple types of sensors; Extracting image features of each modal image data through a convolutional neural network adopting a weight sharing mechanism; The classifier identifying the sensor types of each modal image data according to the image features; A selection and optimization step of selecting an ISP module corresponding to the sensor type to process the modal image data under the sensor type, obtaining the processing results output by each ISP module, adjusting the ISP module according to the type of input data and the current environment to optimize the processing results, obtaining an optimized result; screening images conforming to the specific application scenario type from the optimized result as the image processing result.

2. The ISP autonomous selection and auxiliary decision-making method for multi-sensor data sources according to claim 1, wherein The multi-modal image data includes: visible light image data collected by an RGB sensor, and / or infrared image data collected by an infrared sensor, and / or depth point cloud image data collected by a 3D sensor.

3. The ISP autonomous selection assistance decision-making method for multi-sensor data sources as described in claim 1 or 2, characterized in that, In the convolutional neural network, the weight sharing mechanism is used to extract the basic features of each modal data; after extracting the basic features, convolutional kernels are designed for each sensor modality respectively to further extract the advanced features of each sensor type as the image features.

4. The ISP autonomous selection auxiliary decision-making method for multi-sensor data sources according to claim 1 or 2, characterized in that, The ISP module includes a visible light ISP module, an infrared ISP module and an AI ISP module; In the selection and optimization step, optimizing the processing result specifically includes: According to the image feature F of the sensor input data i and the current environmental characteristic C env , optimize the processing parameter P of the current ISP module init , and obtain the optimized processing parameter P that can more accurately adapt to the current working environment opt ; P opt = Optimize(P init , F i , C env ).

5. An ISP autonomous selection auxiliary decision-making device for multi-sensor data sources, characterized in that, Including: An initial module of receiving multi-modal image data collected by multiple types of sensors; Extracting image features of each modal image data through a convolutional neural network adopting a weight sharing mechanism; The classifier identifying the sensor types of each modal image data according to the image features; A selection and optimization module of selecting an ISP module corresponding to the sensor type to process the modal image data under the sensor type, obtaining the processing results output by each ISP module, adjusting the ISP module according to the type of input data and the current environment to optimize the processing results, obtaining an optimized result; screening images conforming to the specific application scenario type from the optimized result as the image processing result.

6. The ISP autonomous selection and auxiliary decision-making device for multi-sensor data sources according to claim 5, characterized in that, The multi-modal image data includes: visible light image data collected by an RGB sensor, and / or infrared image data collected by an infrared sensor, and / or depth point cloud image data collected by a 3D sensor.

7. The ISP autonomous selection and auxiliary decision-making device for multi-sensor data sources according to claim 5 or 6, characterized in that In the convolutional neural network, the weight sharing mechanism is used to extract the basic features of each modal data; after extracting the basic features, convolutional kernels are designed for each sensor modality respectively to further extract the advanced features of each sensor type as the image features; The ISP module includes a visible light ISP module, an infrared ISP module and an AI ISP module; In the selection and optimization module, optimizing the processing result specifically includes: According to the image feature F of the sensor input data i and the current environmental characteristic C env , optimize the processing parameter P of the current ISP module init , and obtain the optimized processing parameter P that can more accurately adapt to the current working environment opt ; P opt = Optimize(P init , F i , C env ).

8. An electronic device, characterized in that, Including an ISP independent selection and auxiliary decision-making device for multi-sensor data sources as described in claims 5-7, the electronic device is connected to an information display device, and the information display device is used to display the image processing result with display parameters, attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the ISP autonomous selection auxiliary decision-making method for multi-sensor data sources described in any one of claims 1-4 are implemented.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, the steps of the ISP autonomous selection auxiliary decision-making method for multi-sensor data sources described in any one of claims 1-4 are implemented.

Images