Data processing and interaction method and system of multifunctional image terminal

Abstract

The invention provides a data processing and interaction method and system for a multifunctional image terminal, and relates to the technical field of image data processing.The method comprises the steps that a cooperative relation is determined, input signals are dynamically scheduled through a priority queue algorithm and a heartbeat detection mechanism, and an output mapping rule base is constructed; constructing an image integrated control center based on a multi-modal image fusion framework and an image intention decision engine; a multi-source heterogeneous training data set in a complex noise environment is obtained, and parameters are optimized in stages through a multi-task joint loss function end-to-end training control center; inputting a real-time multi-modal input signal into the trained center to obtain an optimized output; the system comprises a multi-modal collaboration and mapping configuration module, an image control center construction module and the like. According to the invention, end-to-end delay time is reduced through a hardware direct connection architecture, image quality and intention recognition accuracy are improved, and the problems of high delay in multi-mode cooperative processing and poor robustness in a noise environment are solved.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a data processing and interaction method and system for a multifunctional image terminal. Background Technology

[0002] With the increasing prevalence of imaging terminals in security and medical fields, the market demands higher levels of data processing and collaborative interaction from multi-functional imaging terminals. Taking security monitoring as an example, under low-light conditions at night, the terminal needs to simultaneously process infrared video streams, ambient audio, and user touch commands. Currently, most commercially available terminals use discrete processing modules, lacking an integrated collaborative mechanism, leading to high signal scheduling latency and inaccurate intent recognition. Specifically, when a user selects a target via the touchscreen and issues a voice command to "zoom in on the license plate area," existing technologies often experience response delays exceeding 500ms due to the reliance on software buses for inter-module communication. Furthermore, low-light noise interference degrades image quality, causing target detection failures or misjudgments. In addition, fixed mapping rules cannot adapt to scene changes (e.g., high-resolution streams need to be routed to the editing scene), resulting in wasted resources. Existing methods, such as single-modal processing or simple rule engines, struggle to achieve efficient fusion and real-time decision-making of multi-modal signals in complex noisy environments, limiting the terminal's intelligence level. Therefore, a trajectory reconstruction method for missing single-side video data is urgently needed to improve the practicality and accuracy of intelligent transportation systems. Summary of the Invention

[0003] The purpose of this invention is to provide a data processing and interaction method and system for a multifunctional image terminal, in order to solve the problems of low efficiency in multimodal signal collaborative processing, poor image quality and inaccurate user intent recognition in complex noise environments, and high terminal response latency, unintelligent interaction and unbalanced resource utilization caused by the lack of scene adaptation capability in service output in the prior art. The specific technical solution is as follows: This invention provides a data processing and interaction method for a multifunctional image terminal, comprising the following steps: S1: Determine the collaborative relationship between the multimodal image interaction module and the image service processing module, and determine the output mapping relationship of the image service processing module to different image service scenarios; S2: Based on the multimodal image fusion framework corresponding to the cooperative relationship and the image intent decision engine corresponding to the mapping relationship, an integrated image control center for controlling multifunctional image terminals is constructed. S3: Obtain the training dataset, which contains multi-source heterogeneous image-related signals collected under complex noise environments; S4: Use the training dataset to train the integrated image control center and jointly optimize each algorithm module; S5: Input the multimodal image input signal to be processed into the trained image integrated control center to obtain optimized output.

[0004] Furthermore, step S1 also includes the following steps: S101: Parse the hardware interface protocol of the multimodal image interaction module and initialize the data channel between each interaction unit and the image service processing module; S102: Construct a collaborative relationship matrix based on the module functional dependency graph, and allocate data processing tasks using a consistent hashing algorithm; S103: Generate an output mapping rule table based on the dynamic demand characteristics of the image service scenario.

[0005] Furthermore, the multimodal image interaction module in S101 includes an image acquisition unit, a voice recognition unit, and a touch sensing unit; wherein the image acquisition unit transmits RAW format image data through the MIPI CSI-2 interface, the voice recognition unit transmits PCM audio streams through the I2S interface, and the touch sensing unit reports coordinate data through the SPI interface; the image service processing module automatically detects the connection status of each interface upon startup and allocates a memory buffer for each interface, with the buffer size dynamically adjusted according to the data type; in S102, the rows of the collaboration relationship matrix represent the functional units of the interaction module, the columns represent the sub-modules of the service processing module, and the matrix element values ​​represent the response time weights of cross-module calls; the task allocation path is optimized through Dijkstra's shortest path algorithm to ensure that high-priority tasks are routed to the NPU acceleration sub-module first.

[0006] Furthermore, the mapping rule table in S103 includes output port identifier, scene type, data format constraints, and routing strategy; when the business processing module outputs a high frame rate video stream, it automatically matches the encoding parameters of the image acquisition scene; when the output contains metadata containing semantic segmentation results, it maps to the post-processing pipeline of the image recognition scene.

[0007] Furthermore, step S2 also includes the following steps: S201: Configure the algorithm pipeline for the multimodal image fusion framework and initialize hardware acceleration parameters; S202: Construct a multimodal fusion model for the image intent decision engine and load pre-trained weights; S203: Deploy an integrated system on a hardware direct-connect collaborative architecture and establish inter-module communication protocols.

[0008] Furthermore, the multimodal image fusion framework in S201 includes an environment-adaptive multimodal fusion component, which consists of an environment perception algorithm, a FastViTHD high-efficiency image coding algorithm, and an image scene adaptation algorithm; the environment perception algorithm adopts an improved Retinex theoretical model, and its mathematical expression is: , where I inv For the input image, G k For the k-th Gaussian convolution kernel, ω k Here, ε is the weighting coefficient, and R is the constant for preventing division by zero. target The target illumination parameters are given; the image intent decision engine in S202 includes a semantically enhanced intent decision component, which contains a multimodal intent recognition model that combines image semantic features. It employs a cross-modal Transformer architecture, and its input feature fusion formula is: Where I represents image features, A represents voice features, T represents touch features, and W represents [missing information]. f To integrate the weight matrix, the hardware direct-connect collaborative architecture in S203 supports direct collaboration between the ISP and NPU. By configuring the ISP output interface to directly connect to the NPU input buffer, image data is transmitted using the AXI-Stream protocol, and the transmission latency is controlled within 5ms. At the same time, the physical address space is shared through device tree overlap mapping technology.

[0009] Furthermore, step S3 also includes the following steps: S301: Construct an image data acquisition platform under real noise environment to obtain noise-clean image pairs from the SIDD real noise image dataset; S302: Synchronously acquire multimodal interaction signals to construct time-aligned training samples; S303: Injecting simulated transmission noise and compression noise to expand the diversity of the training dataset; The expression for acquiring noisy images in S301 is: , where N shot The noise level during filming follows a Poisson distribution; N read To read the noise, it follows a Gaussian distribution; N quant To quantize the noise, it follows a uniform distribution.

[0010] Furthermore, step S4 also includes the following steps: S401: Construct a joint loss function for multiple tasks and define the weight balance of each optimization objective; S402: Employs a phased, progressive training strategy to gradually optimize each algorithm component; S403: Establish a real-time verification platform; Step S5 also includes the following steps: S501: Receives multimodal input signals and performs real-time preprocessing and feature extraction; S502: Multimodal fusion and intent decision-making are performed through an integrated image control center; S503: Output target files, operation instructions, and interactive responses based on the decision results.

[0011] Furthermore, the joint loss function in S401 is: L denoise Using the Charbonnier loss function: L perceptual Using LPIPS as a metric: Ll atency An S-shaped penalty function is used; L intent Focus loss is used; λ1 is 1.0, λ2 is 0.8, λ3 is 0.5, and λ4 is 1.2.

[0012] The present invention also provides a data processing and interaction system for a multifunctional image terminal, used to implement the method described above, the system comprising: The multimodal collaboration and mapping configuration module is used to determine the collaboration relationship between the multimodal image interaction module and the image service processing module, and to construct the output mapping relationship from the image service processing module to different image service scenarios. The image control center construction module is used to construct an integrated image control center for controlling the multi-functional image terminal based on a multi-modal image fusion framework corresponding to the collaborative relationship and an image intent decision engine corresponding to the mapping relationship. The training data acquisition and augmentation module is used to acquire a training dataset containing related signals from multi-source heterogeneous images collected under complex noise environments, and to perform data augmentation processing. The model joint training and optimization module is used to perform end-to-end training of the image integrated control center using the training dataset, and jointly optimize each algorithm module through a multi-task loss function. The multimodal signal processing and output module is used to input the multimodal image input signal to be processed into the trained image integrated control center to obtain the optimized output result.

[0013] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, performs the steps of the method described herein.

[0014] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method.

[0015] The beneficial effects of this invention are as follows: This invention achieves collaborative processing of image signals by adopting a multimodal fusion framework, combines environmental perception algorithms to adapt to complex lighting conditions to improve image quality, utilizes the FastViTHD efficient coding algorithm to reduce computational complexity while ensuring visual quality, and accurately understands user operation intentions through a multimodal intent recognition model to achieve natural human-computer interaction; optimizes inference latency based on hardware direct connection architecture to meet real-time requirements, and achieves intelligent scheduling and scene-adaptive routing of input signals through modular collaboration and scene-based mapping mechanisms; improves system robustness through training data augmentation methods that combine real noise environment with simulated noise; and simultaneously improves image quality, inference speed, and intent recognition accuracy by adopting multi-task joint optimization and phased progressive training strategies. Thus, it achieves efficient data processing and intelligent interaction of multifunctional image terminals in complex environments, significantly improving the intelligence level of terminal devices and user experience.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram illustrating the steps of a data processing and interaction method for a multifunctional image terminal according to the present invention; Figure 2 This is a schematic diagram of the data processing and interaction system of a multifunctional image terminal according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] In an embodiment of the present invention, a data processing and interaction method for a multifunctional image terminal is provided. Please refer to [link to relevant documentation]. Figure 1 The method includes the following steps: S1: Determine the collaborative relationship between the multimodal image interaction module and the image service processing module, and determine the output mapping relationship of the image service processing module to different image service scenarios.

[0020] In this embodiment, an integrated control architecture for a multi-functional imaging terminal is provided, which is based on a modular collaboration and scene-based mapping mechanism. The data processing and interaction methods of the multi-functional imaging terminal are implemented based on this architecture. This architecture first identifies the input features of the multi-modal imaging interaction module through a protocol parsing unit. The multi-modal imaging interaction module includes an image acquisition unit, a voice recognition unit, and a touch sensing unit. The image acquisition unit supports real-time capture and lossless compression of 4K ultra-high-definition video streams; the voice recognition unit uses AAC audio compression encoding technology to sample ambient audio; and the touch sensing unit collects user gesture operation signals through a capacitive high-precision sensor. The imaging service processing module includes a signal scheduling submodule, an encoding / decoding submodule, and a scene routing submodule, which establishes a collaborative relationship with the interaction module through an event-driven message bus. The process of determining the collaborative relationship includes: First, assigning an independent data channel identifier to each interaction unit and dynamically scheduling input signals to the business processing module based on a priority queue algorithm (such as a weighted fair queue algorithm); Second, maintaining the real-time status of the communication link between modules through a heartbeat detection mechanism. If the interaction module does not respond to the query command of the business module within a preset timeout threshold (e.g., 500ms), the redundant link switching logic is automatically triggered.

[0021] The determination of the output mapping relationship specifically involves: constructing a mapping rule base indexed by scene identifiers, where image service scenes include image acquisition scenes, image editing scenes, image recognition scenes, image transmission scenes, and image storage scenes. The mapping rule base is generated by: fitting the mapping weights between the output and scenes using statistical learning algorithms (such as Hidden Markov Models) based on the correlation between the business module output results and scene triggering conditions in historical operation logs; simultaneously, administrators can customize mapping strategies through a visual configuration interface, for example, mapping high-resolution image streams (resolution ≥ 3840×2160) to image editing scenes and low-latency streams (end-to-end latency ≤ 400ms) to image transmission scenes.

[0022] Specifically, in one implementation of this embodiment, step S1 includes the following sub-steps: S101 parses the hardware interface protocol of the multimodal image interaction module and initializes the data channel between each interaction unit and the image service processing module. As an example, the image acquisition unit transmits RAW format image data via the MIPI CSI-2 interface, the speech recognition unit transmits PCM audio streams via the I2S interface, and the touch sensing unit reports coordinate data via the SPI interface. Upon startup, the service processing module automatically detects the connection status of each interface and allocates a memory buffer for each interface. The buffer size is dynamically adjusted based on the data type (e.g., the image buffer is set to 8MB, and the audio buffer is set to 512KB).

[0023] S102 constructs a collaborative relationship matrix based on the module functional dependency graph and allocates data processing tasks through a consistent hashing algorithm. In this matrix, rows represent functional units of the interaction module, columns represent sub-modules of the business processing module, and matrix element values ​​represent response time weights for cross-module calls. The task allocation path is optimized using Dijkstra's shortest path algorithm to ensure that high-priority tasks (such as real-time object detection in image recognition scenarios) are routed preferentially to the NPU-accelerated sub-module.

[0024] S103, generate an output mapping rule table based on the dynamic demand characteristics of the image service scenario.

[0025] The mapping rule table includes output port identifiers, scene types, data format constraints, and routing strategies. For example, when the business processing module outputs a high frame rate video stream (frame rate ≥ 60fps), it automatically matches the encoding parameters of the image acquisition scene; when the output contains metadata containing semantic segmentation results, it maps to the post-processing pipeline of the image recognition scene.

[0026] In a specific application example of this embodiment, a security monitoring terminal is used as an example: The multimodal image interaction module captures low-light video streams at night via an infrared camera and simultaneously acquires ambient audio via a microphone array. The business processing module detects low-light conditions using an environmental perception algorithm, automatically activates the image enhancement submodule, and routes the output stream simultaneously to both the image storage scenario (full-resolution archiving) and the image transmission scenario (low-bitrate real-time upload to the cloud) based on mapping rules. The collaborative relationship is achieved through a hardware interrupt mechanism. When the touch unit receives a user-defined area of ​​interest, it immediately triggers the ROI (Region of Interest) encoding algorithm of the business module to reduce the bitrate of non-interested areas to save bandwidth.

[0027] S2: Based on the multimodal image fusion framework corresponding to the cooperative relationship and the image intent decision engine corresponding to the mapping relationship, an integrated image control center for controlling the multifunctional image terminal is constructed.

[0028] In this embodiment, a core processing architecture for a multi-functional image terminal is provided, which is based on a collaborative mechanism of multimodal fusion and intent decision-making. The construction process of the integrated image control center includes three core components: 1) Environment Adaptive Multimodal Fusion Component: This component consists of an environment perception algorithm, a FastViTHD high-efficiency image coding algorithm, and an image scene adaptation algorithm. It achieves real-time alignment and feature-level fusion of multimodal signals through a hardware-accelerated parallel computing pipeline. 2) Semantic Enhancement Intent Decision Component: This component includes a multimodal intent recognition model that combines image semantic features. It uses a cross-modal attention mechanism to perform joint semantic parsing on multi-source inputs such as images, voice, and touch, and generates accurate operation intent vectors. 3) Hardware Direct Connection Cooperative Execution Component: This component supports a hardware direct connection architecture between the ISP (Image Signal Processor) and the NPU (Neural Processing Unit), and realizes dynamic scheduling of computing resources through shared memory mapping and DMA transfer mechanism.

[0029] Based on the above three components, this embodiment requires heterogeneous integration of the multimodal image fusion framework and the image intent decision engine to construct a complete integrated image control center.

[0030] Specifically, in one implementation of this embodiment, step S2 includes the following sub-steps: S201 configures the algorithm pipeline for the multimodal image fusion framework and initializes the hardware acceleration parameters.

[0031] The environmental perception algorithm employs an improved Retinex theoretical model, achieving adaptive illumination correction through bilateral filtering and color constrained behavior. Its mathematical expression is as follows: , among which, I in For the input image, G k For the k-th Gaussian convolution kernel, ω k Here, ε is the weighting coefficient, and R is the constant for preventing division by zero. target The target illumination parameters are defined as follows. The FastViTHD high-efficiency image coding algorithm employs a hierarchical attention mechanism, reducing computational complexity to 30% of the traditional ViT while maintaining visual fidelity. Its key operations are defined as follows: Where M is the local receptive field mask matrix, used to limit the scope of self-attention calculation. The image scene adaptation algorithm dynamically adjusts the encoding parameters through a multi-task learning framework to support differentiated processing needs in different scenarios such as acquisition, editing, and recognition.

[0032] S202, construct a multimodal fusion model for the image intent decision engine and load pre-trained weights. The multimodal intent recognition model that combines image semantic features adopts a cross-modal Transformer architecture, and its input feature fusion formula is as follows: Where I represents image features, A represents voice features, T represents touch features, and W represents [missing information]. f The weight matrix is ​​used for fusion. The intent classification output is calculated using the softmax function: Where C represents the total number of intent categories, including operational intents such as image acquisition, editing, and recognition.

[0033] S203 deploys an integrated system on a hardware direct-connect collaborative architecture and establishes an inter-module communication protocol. The direct connection and collaboration between the ISP and NPU are achieved through the following method: the ISP output interface is configured to directly connect to the NPU input buffer, and image data is transmitted using the AXI-Stream protocol, with transmission latency controlled within 5ms. Simultaneously, physical address space is shared through device tree overlay mapping technology to avoid memory copy overhead. As an example, this embodiment uses the Rockchip RK3588S chip platform, whose ISP output resolution supports a maximum of 48MP and NPU computing power reaches 6TOPS.

[0034] In a specific application example of this embodiment, a smart law enforcement recorder is used as an example: When the device simultaneously receives the voice command "zoom in on the license plate area" and a touch selection operation, the environment-adaptive multimodal fusion component first performs super-resolution processing on the video stream using the FastViTHD algorithm. The semantic enhancement intent decision component then jointly analyzes the voice semantics and touch coordinates to generate a "local enhancement" operation intent. The hardware direct-connection collaboration component immediately schedules NPU resources to execute the license plate recognition algorithm, while simultaneously adjusting the local contrast in real time via the ISP. The entire processing flow has an end-to-end latency of less than 200ms, meeting real-time interaction requirements.

[0035] This embodiment achieves efficient fusion of multimodal data and accurate intent recognition through the above technical solution, providing low-latency and high-precision integrated control capabilities for multifunctional imaging terminals.

[0036] S3: Obtain the training dataset, which contains multi-source heterogeneous image correlation signals collected under complex noise environments.

[0037] This embodiment provides a method for acquiring and constructing diversified training data for practical application scenarios. The construction process of the training dataset includes the following three data acquisition modules: 1) Visual signal acquisition and enhancement module: specifically responsible for acquiring optical image data, injecting noise, and enhancing quality; 2) Interactive signal synchronization acquisition module: responsible for the synchronous acquisition and timing alignment of interactive signals such as voice and touch; 3) Auxiliary acquisition module for environmental signals: responsible for the acquisition and calibration of auxiliary signals such as ambient light sensors.

[0038] Based on the above three modules, this embodiment needs to construct a multi-source heterogeneous training dataset that includes real noise environments and simulated noise environments.

[0039] Specifically, in one implementation of this embodiment, step S3 includes the following sub-steps: S301, Construct an image data acquisition platform under real noise environment to obtain noise-clean image pairs from SIDD real noise image dataset. The acquisition platform comprises multiple image sensors of different models, simultaneously acquiring RAW format images of the same scene under complex lighting conditions such as low light (1-10 lux) and high light (>1000 lux). Noise-clean image pairs are obtained through multiple samplings of a fixed scene, where the direct acquisition expression for the noisy image is: , where N shot For the shooting noise (Poisson distribution), N read To read the noise (Gaussian distribution), N quant To quantify the noise (uniform distribution). As an example, this embodiment acquired more than 10,000 image pairs in 50 different indoor and outdoor scenes, each pair containing short-exposure (noisy) and long-exposure (clean) images.

[0040] S302 synchronously acquires multimodal interaction signals to construct time-aligned training samples. The image operation-related voice command signal is acquired via a directional microphone array at a sampling rate of 48kHz, using AAC-LC encoding; the touch operation signal is acquired via a capacitive touchscreen at a reporting rate of 120Hz, including a coordinate (x,y) pressure value p triplet; the ambient light sensing signal is acquired via a TSL2591 high-precision light sensor at a sampling rate of 10Hz. Multi-signal synchronization is achieved through hardware timestamps, with a synchronization error of less than 5ms. The timing alignment formula is: , where Δt sync This is the compensation value for inherent synchronization deviation in the hardware.

[0041] S303 injects simulated transmission noise and compression noise to expand the diversity of the training dataset. For transmission noise, the Gilbert-Elliott channel model is used to simulate wireless packet loss. , where P GB P GG Let denoted as , and PB and PG be the packet loss rates in the bad and good states, respectively. For noise compression, a combination of variable-quality JPEG compression and H.265 encoding is used. The QP (quantization parameter) is randomly selected within the range of 20-40 to simulate mass loss under different compression ratios.

[0042] S4: Use the training dataset to train the integrated image control center and jointly optimize each algorithm module.

[0043] In this embodiment, an end-to-end joint training method is provided, which simultaneously optimizes the environment perception algorithm, the FastViTHD high-efficiency image coding algorithm, the image scene adaptation algorithm, and the multimodal intent recognition model through a multi-task loss function.

[0044] Specifically, in one implementation of this embodiment, step S4 includes the following sub-steps: S401, construct a multi-task joint loss function and define the weight balance of each optimization objective.

[0045] The joint loss function consists of four components: The denoising loss uses the Charbonnier loss function: Perceptual loss is measured using LPIPS (Learned Perceptual Image Patch Similarity): The delay constraint loss uses an S-shaped penalty function: The intention is to identify the loss using focus loss. (Focal Loss): In this embodiment, λ1=1.0, λ2=0.8, λ3=0.5, and λ4=1.2 are set.

[0046] S402 employs a phased, progressive training strategy to gradually optimize each algorithm component. The first stage involves training the environment awareness algorithm separately, using the SIDD dataset to optimize denoising performance. The second stage involves jointly training the FastViTHD encoding algorithm and the scene adaptation algorithm, focusing on optimizing computational efficiency. The third stage involves end-to-end training of the entire system, jointly optimizing all parameters. Training uses the AdamW optimizer with an initial learning rate of 3e-4 and cosine annealing scheduling.

[0047] S403 establishes a real-time verification platform to ensure that the terminal-side inference response time is less than 400ms.

[0048] The trained model was deployed on the Rockchip RK3588S development board, and end-to-end latency was measured using timestamps. Optimization measures included operator fusion, inter-layer quantization (INT8), and memory reuse. The final performance achieved on the test set was an average latency of 378ms and an accuracy of 95.3%.

[0049] S5: Input the multimodal image input signal to be processed into the trained image integrated control center to obtain optimized output.

[0050] Specifically, in one implementation of this embodiment, step S5 includes the following sub-steps: The S501 receives multimodal input signals and performs real-time preprocessing and feature extraction. Image signals undergo black level correction, de-mosaicing, and white balance processing via the ISP pipeline; audio signals undergo endpoint detection and MFCC feature extraction; and touch signals undergo gesture trajectory recognition and feature encoding. The preprocessing process meets real-time requirements, with a single frame processing time of less than 33ms (corresponding to a 30fps video stream).

[0051] S502 performs multimodal fusion and intent decision-making through an integrated image control center. The fused features are input into the intent decision engine to generate an operation intent probability distribution: Where Wcls is the classification layer weight matrix. When the maximum probability exceeds the threshold of 0.8, the corresponding business operation is triggered.

[0052] S503 outputs target files, operation instructions, and interactive responses based on the decision results. The optimized target image file supports multiple output formats and embeds processing parameters in the metadata; the directional image service operation instructions are sent to the corresponding service modules through a message queue; the image service interaction response includes visual feedback (interface update), auditory feedback (voice prompts) and tactile feedback (vibration prompts).

[0053] In one example of a medical endoscope application in this embodiment: The system receives 1080p endoscopic video streams, doctor's voice commands to "zoom in on the lesion area," and touch selection signals in real time. The control center completes noise suppression, feature enhancement, and intent recognition within 320ms, outputs a 4K super-resolution image file, generates a "local zoom-in" operation command, and triggers an interface focusing animation and confirmation sound prompt, effectively assisting doctors in making accurate diagnoses.

[0054] This embodiment, through the above technical solution, realizes intelligent data processing and natural interaction of the multifunctional imaging terminal in complex environments, providing a complete solution for fields such as medical care, security, and industrial inspection.

[0055] In one specific embodiment, S3: Obtaining a training dataset, the training dataset containing multi-source heterogeneous image correlation signals collected under complex noise environments, further includes the following steps: S311, build a multi-source signal synchronous acquisition platform, and configure image sensor, microphone array, touch screen and ambient light sensor; The S312 acquires raw image data under complex lighting conditions such as low light (1-10 lux) and high light (>1000 lux), and simultaneously records voice command signals; S313 ensures the timing consistency of multimodal signals through a timestamp alignment mechanism, with alignment error controlled within 5ms; S314, inject simulated noise, including shooting noise based on the Poisson-Gaussian mixture model, transmission packet loss noise based on the Gilbert-Elliott channel, and compression noise based on variable quality parameters; S315, annotates the collected data, including noise-clean image pair matching, voice command text transcription, and touch gesture classification labels; S316 divides the training set, validation set, and test set into a 7:2:1 ratio to ensure a balanced data distribution.

[0056] This embodiment constructs a multi-source heterogeneous training dataset containing real environmental noise and simulated noise through the above technical solution, providing sufficient data support for model training.

[0057] In one specific embodiment, S4: Training the integrated image control center using the training dataset and jointly optimizing each algorithm module further includes the following steps: S411, initialize the parameters of each component of the integrated image control center, the environmental perception algorithm is initialized using the Kaiming normal distribution, and the FastViTHD encoding algorithm loads the ImageNet pre-trained weights. S412, construct the multi-task joint loss function, which includes four parts: denoising loss, perceptual loss, delay constraint loss, and intent recognition loss: The denoising loss uses the Charbonnier loss function: .

[0058] S413 adopts a phased progressive training strategy. First, the environment awareness algorithm is trained separately for 50 epochs with a learning rate of 3e-4. S414, with fixed environment perception algorithm parameters, jointly trains FastViTHD encoding algorithm and scene adaptation algorithm for 30 epochs, reducing the learning rate to 1e-4; S415, end-to-end training of the entire integrated image control center for 20 epochs, using a cosine annealing strategy to adjust the learning rate; S416: Evaluate model performance on the validation set and save the optimal model when the denoised PSNR is greater than 35dB and the intent recognition accuracy is greater than 95%. S417 was used for latency testing on the Rockchip RK3588S development board. Inference speed was optimized through techniques such as operator fusion and inter-layer quantization. The S418 model achieved an average latency of 378ms, a denoised PSNR of 36.2dB, and an intent recognition accuracy of 95.3% on the test set.

[0059] This embodiment enables the integrated image control center to adapt to complex shooting environments, achieve low-latency inference, and accurately identify user operation intentions through the above training scheme.

[0060] S5: Input the multimodal image input signal to be processed into the trained image integrated control center to obtain optimized output.

[0061] In the practical inference application of this embodiment, the multimodal signal processing flow includes the following steps: The S511 receives real-time input signals. Image signals are acquired in RAW format through the MIPI CSI-2 interface, voice signals are acquired in PCM audio stream through the I2S interface, and touch signals are reported in coordinate data through the SPI interface. S512 monitors ambient light sensor data in real time and automatically triggers low light enhancement mode when the ambient light illuminance is below 50 lux. S513: Multimodal signals are processed in parallel through a hardware direct-connect architecture, the ISP performs image signal preprocessing, and the NPU accelerates neural network inference. S514, the environment perception algorithm performs noise suppression and image quality enhancement on the input image, and the FastViTHD encoding algorithm extracts visual features; The S515 multimodal intent recognition model fuses image features, voice features, and touch features to generate an operation intent probability distribution: .

[0062] S516, when the maximum intent probability exceeds the threshold of 0.8, the corresponding business operation instruction is triggered; The S517 outputs optimized target image files, supports multiple formats such as JPEG, PNG, and HEIC, and embeds processing parameters in the metadata; S518 generates targeted business operation instructions, sends them to the corresponding business modules through a message queue, and provides multimodal interactive feedback.

[0063] In a security monitoring application example of this embodiment: the system receives 1080p video streams, voice commands "track suspicious persons," and touch selection signals in real time. The control center completes target detection, feature extraction, and intent recognition within 350ms, outputs a 4K super-resolution image file, generates a "target tracking" operation command, and triggers the display of the tracking box on the interface and voice confirmation prompts, effectively improving the intelligence level of security monitoring.

[0064] This embodiment achieves the following technical effects through the above technical solution: It employs a multimodal fusion framework to achieve collaborative processing of image signals, adapts to complex lighting conditions through an environmental perception algorithm, and improves image quality; it utilizes the FastViTHD efficient coding algorithm to significantly reduce computational complexity while ensuring visual quality; it combines a multimodal intent recognition model to accurately understand user operation intentions and achieve natural human-computer interaction; it optimizes inference latency based on a hardware direct connection architecture to meet real-time requirements; and it fully considers noise interference, computational resource constraints, and interaction needs in practical applications, realizing efficient data processing and intelligent interaction of a multifunctional image terminal, thereby improving the intelligence level of the terminal device and the user experience.

[0065] like Figure 2 As shown, the present invention also provides a data processing and interaction system for a multi-functional imaging terminal, used to implement the aforementioned data processing and interaction method for a multi-functional imaging terminal. The system includes: The multimodal collaboration and mapping configuration module is used to determine the collaboration relationship between the multimodal image interaction module and the image service processing module, and to construct the output mapping relationship from the image service processing module to different image service scenarios. The image control center construction module is used to construct an integrated image control center for controlling the multi-functional image terminal based on a multi-modal image fusion framework corresponding to the collaborative relationship and an image intent decision engine corresponding to the mapping relationship. The training data acquisition and augmentation module is used to acquire a training dataset containing related signals from multi-source heterogeneous images collected under complex noise environments, and to perform data augmentation processing. The model joint training and optimization module is used to perform end-to-end training of the image integrated control center using the training dataset, and jointly optimize each algorithm module through a multi-task loss function. The multimodal signal processing and output module is used to input the multimodal image input signal to be processed into the trained image integrated control center to obtain the optimized output result.

[0066] This embodiment achieves the following technical effects through the above technical solution: This embodiment employs a modular collaboration and scenario-based mapping mechanism to establish an efficient collaborative relationship between the multimodal interaction module and the business processing module, achieving intelligent scheduling and scenario-adaptive routing of input signals. Through the heterogeneous integration of an environment-adaptive multimodal fusion framework and a semantically enhanced intent decision engine, a low-latency, high-precision integrated control center is constructed. A training data augmentation method combining real and simulated noise environments enhances the system's robustness in complex environments. A multi-task joint optimization and phased progressive training strategy simultaneously improves image quality, inference speed, and intent recognition accuracy. Computational resource scheduling is optimized based on a hardware direct-connect architecture, ensuring end-to-end latency meets real-time interaction requirements. This embodiment fully considers the data processing and natural interaction needs of multi-functional imaging terminals in complex environments, achieving efficient fusion of multimodal signals and intelligent decision-making, significantly improving the intelligence level of terminal devices and user experience.

[0067] Based on the above embodiments, the present invention also provides a terminal, which includes: a processor, a memory, a multimodal interface, a display unit, and a communication module connected via a system bus; wherein, the processor of the terminal is used to provide computing and control capabilities and support NPU neural network acceleration; the memory of the terminal includes a computer-readable storage medium and internal memory; the computer-readable storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the computer-readable storage medium; the multimodal interface is used to connect external devices such as an image acquisition unit, a voice recognition unit, and a touch sensing unit; the display unit is used to display image processing results and an interactive interface; and the communication module is used to transmit image data with a cloud server or other devices.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0069] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A data processing and interaction method for a multi-functional image terminal, characterized in that, include: S1: Determine the collaborative relationship between the multimodal image interaction module and the image service processing module, and determine the output mapping relationship of the image service processing module to different image service scenarios; S2: Based on the multimodal image fusion framework corresponding to the cooperative relationship and the image intent decision engine corresponding to the mapping relationship, an integrated image control center for controlling multifunctional image terminals is constructed. S3: Obtain the training dataset, which contains multi-source heterogeneous image-related signals collected under complex noise environments; S4: Use the training dataset to train the integrated image control center and jointly optimize each algorithm module; S5: Input the multimodal image input signal to be processed into the trained image integrated control center to obtain optimized output.

2. The method as described in claim 1, characterized in that, Step S1 also includes the following steps: S101: Parse the hardware interface protocol of the multimodal image interaction module and initialize the data channel between each interaction unit and the image service processing module; S102: Construct a collaborative relationship matrix based on the module functional dependency graph, and allocate data processing tasks using a consistent hashing algorithm; S103: Generate an output mapping rule table based on the dynamic demand characteristics of the image service scenario.

3. The method as described in claim 2, characterized in that, The multimodal image interaction module in S101 includes an image acquisition unit, a voice recognition unit, and a touch sensing unit. The image acquisition unit transmits RAW format image data via a MIPI CSI-2 interface, the voice recognition unit transmits PCM audio streams via an I2S interface, and the touch sensing unit reports coordinate data via an SPI interface. The image service processing module automatically detects the connection status of each interface upon startup and allocates a memory buffer for each interface, with the buffer size dynamically adjusted according to the data type. In S102, the rows of the collaboration relationship matrix represent the functional units of the interaction module, the columns represent the sub-modules of the service processing module, and the matrix element values ​​represent the response time weights of cross-module calls. The task allocation path is optimized by using Dijkstra's shortest path algorithm to ensure that high-priority tasks are routed to the NPU acceleration submodule first.

4. The method as described in claim 2, characterized in that, The mapping rule table in S103 includes output port identifier, scene type, data format constraints, and routing strategy; when the business processing module outputs a high frame rate video stream, it automatically matches the encoding parameters of the image acquisition scene; when the output contains metadata containing semantic segmentation results, it maps to the post-processing pipeline of the image recognition scene.

5. The method as described in claim 1, characterized in that, Step S2 also includes the following steps: S201: Configure the algorithm pipeline for the multimodal image fusion framework and initialize hardware acceleration parameters; S202: Construct a multimodal fusion model for the image intent decision engine and load pre-trained weights; S203: Deploy an integrated system on a hardware direct-connect collaborative architecture and establish inter-module communication protocols.

6. The method as described in claim 5, characterized in that, The multimodal image fusion framework in S201 includes an environment-adaptive multimodal fusion component, which consists of an environment perception algorithm, a FastViTHD high-efficiency image coding algorithm, and an image scene adaptation algorithm. The environment perception algorithm adopts an improved Retinex theoretical model, and its mathematical expression is as follows: , where I inv For the input image, G k For the k-th Gaussian convolution kernel, ω k Here, R is the weighting coefficient, ε is the constant for preventing zero, and R is the weighting coefficient. target The target illumination parameters are given; the image intent decision engine in S202 includes a semantically enhanced intent decision component, which contains a multimodal intent recognition model that combines image semantic features. It employs a cross-modal Transformer architecture, and its input feature fusion formula is: Where I represents image features, A represents voice features, T represents touch features, and W represents [missing information]. f To integrate the weight matrix, the hardware direct-connect collaborative architecture in S203 supports direct collaboration between the ISP and NPU. By configuring the ISP output interface to directly connect to the NPU input buffer, image data is transmitted using the AXI-Stream protocol, and the transmission latency is controlled within 5ms. At the same time, the physical address space is shared through device tree overlap mapping technology.

7. The method as described in claim 1, characterized in that, Step S3 also includes the following steps: S301: Construct an image data acquisition platform under real noise environment to obtain noise-clean image pairs from the SIDD real noise image dataset; S302: Synchronously acquire multimodal interaction signals to construct time-aligned training samples; S303: Injecting simulated transmission noise and compression noise to expand the diversity of the training dataset; The expression for acquiring noisy images in S301 is: , where N shot The noise level during filming follows a Poisson distribution; N read To read the noise, it follows a Gaussian distribution; N quant To quantize the noise, it follows a uniform distribution.

8. The method as described in claim 1, characterized in that, Step S4 also includes the following steps: S401: Construct a joint loss function for multiple tasks and define the weight balance of each optimization objective; S402: Employs a phased, progressive training strategy to gradually optimize each algorithm component; S403: Establish a real-time verification platform; Step S5 also includes the following steps: S501: Receives multimodal input signals and performs real-time preprocessing and feature extraction; S502: Multimodal fusion and intent decision-making are performed through an integrated image control center; S503: Output target files, operation instructions, and interactive responses based on the decision results.

9. The method as described in claim 8, characterized in that, The joint loss function in S401 is: L denoise Using the Charbonnier loss function: L perceptual Using LPIPS as a metric: Ll atency An S-shaped penalty function is used; L intent Focus loss is used; λ1 is 1.0, λ2 is 0.8, λ3 is 0.5, and λ4 is 1.

2.

10. A data processing and interaction system for a multi-functional image terminal, used to implement the method described in any one of claims 1-9, characterized in that, The system includes: The multimodal collaboration and mapping configuration module is used to determine the collaboration relationship between the multimodal image interaction module and the image service processing module, and to construct the output mapping relationship from the image service processing module to different image service scenarios. The image control center construction module is used to construct an integrated image control center for controlling the multi-functional image terminal based on a multi-modal image fusion framework corresponding to the collaborative relationship and an image intent decision engine corresponding to the mapping relationship. The training data acquisition and augmentation module is used to acquire a training dataset containing related signals from multi-source heterogeneous images collected under complex noise environments, and to perform data augmentation processing. The model joint training and optimization module is used to perform end-to-end training of the image integrated control center using the training dataset, and jointly optimize each algorithm module through a multi-task loss function. The multimodal signal processing and output module is used to input the multimodal image input signal to be processed into the trained image integrated control center to obtain the optimized output result.

Images