A method and system for real-time processing of end-side multimodal large models
By synchronously segmenting video and audio streams into multimodal interactive data blocks, detecting sound and image anomalies, dynamically allocating computing power, and combining high- and low-precision computing modules, the problem of inference delay and anomaly localization difficulties caused by improper computing power allocation in existing technologies is solved, achieving efficient and accurate anomaly identification and localization.
Patent Information
- Application Number
- CN202611585592.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing edge processing solutions struggle to allocate computing power according to the importance of abnormal areas in industrial mobile inspection equipment, leading to increased inference latency, loss of key abnormal features, and difficulty in mapping abnormal locations to actual space.
By synchronously segmenting continuous video and audio streams into multimodal interactive data blocks, abrupt changes in sound energy and image contours are detected. Computational power is dynamically allocated, and features are extracted by combining high-precision and low-precision computing modules. Contextual correlation analysis is then performed to determine the type of abnormal events and their spatial coordinates.
It enables efficient identification and location of abnormal events under limited memory bandwidth and power consumption conditions, reduces the false positive rate, and provides accurate abnormal event types and spatial coordinates.
Smart Images

Figure CN123116101A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and system for real-time processing of edge-side multimodal large models. Background Technology
[0002] Industrial mobile inspection equipment typically identifies anomalies such as unusual noises, leaks, smoke, sparks, and unauthorized entry through video and audio. Existing edge-side processing solutions often process complete video frames and audio data with fixed precision, making it difficult to allocate computing power based on the importance of the anomaly area. When edge-side memory bandwidth, power consumption, and heat dissipation capabilities are limited, problems such as increased inference latency, loss of key anomaly features, and difficulty in mapping anomaly locations to actual space can easily occur. Summary of the Invention
[0003] This application provides a real-time processing method and system for edge-side multimodal large models to overcome the problem that existing edge-side inspection schemes, when using fixed computational accuracy and fixed computing power to process multimodal data, are unable to simultaneously ensure anomaly identification accuracy, spatial positioning accuracy, and real-time performance.
[0004] Firstly, this application provides a real-time processing method for edge-side multimodal large models, including: Acquire the continuous video stream and accompanying audio stream of the current inspection environment, and synchronously cut the continuous video stream and the accompanying audio stream into multiple multimodal interactive data blocks according to a preset time length; Detect the degree of abrupt change in sound energy and the degree of change in image contours in the multimodal interactive data block; Based on the degree of change in sound energy and the degree of change in image contour, the location of sound and image anomalies is determined, and the multimodal interaction data block is divided into target region data and non-target region data. A dynamic computing power allocation strategy is generated based on the target region data and the non-target region data, and the dynamic computing power allocation strategy is executed by calling the preset edge multimodal large model. Based on the dynamic computing power allocation strategy, the high-precision floating-point calculation module is called to extract features from the target area data to obtain target feature representation, and the low-precision integer calculation module is called to extract features from the non-target area data to obtain background feature representation; The target feature representation and the background feature representation are subjected to contextual correlation analysis, and combined with the abnormal location of the sound and image, the abnormal event type and spatial coordinates of the abnormal event in the inspection environment are determined.
[0005] Optionally, a continuous video stream and accompanying audio stream of the current inspection environment are acquired, and the continuous video stream and accompanying audio stream are synchronously segmented into multiple multimodal interactive data blocks according to a preset time length, including: The acoustic sensor array and vision sensor mounted on the industrial mobile inspection equipment synchronously collect the accompanying audio stream and the continuous video stream of the current inspection environment; On the same timeline of the accompanying audio stream and the continuous video stream, a sliding time window with the preset time length is established; The sliding time window is controlled to slide step by step on the same time axis; The time-aligned audio segments and video frame sequences are extracted using the sliding time window. The audio clips and the video frame sequence are combined to construct the multimodal interactive data block.
[0006] Optionally, detecting the degree of abrupt changes in sound energy and the degree of changes in image contours in the multimodal interactive data block includes: Calculate the short-time energy variation rate and audio spectral flux of the audio segment in the multimodal interactive data block; The weighted fusion value of the short-time energy variation rate and the audio spectral flux is determined as the degree of sound energy mutation. Extract the optical flow anomaly index and edge matching residual of the video frame sequence from the multimodal interactive data block; The weighted fusion value of the optical flow anomaly index and the edge matching residual is determined as the degree of image contour change.
[0007] Optionally, the location of sound and image anomalies is determined based on the degree of change in sound energy and the degree of change in image contour, and the multimodal interaction data block is divided into target region data and non-target region data, including: Extract the audio timestamps of sound energy mutations that exceed a preset threshold, and calculate the azimuth angle of the corresponding abnormal sound source using a multi-channel phase difference algorithm; Extract the video timestamps and corresponding image anomaly bounding boxes whose image contour changes are greater than a preset threshold. The audio timestamp and the video timestamp are time-aligned, and the spatial consistency of the sound source azimuth angle and the image anomaly bounding box is matched to generate the sound and image anomaly locations. The video data located within the image anomaly bounding box and the audio data within the time period corresponding to the audio timestamp are merged into the target region data; The remaining background data after stripping the target region data from the multimodal interactive data block is combined into the non-target region data.
[0008] Optionally, a dynamic computing power allocation strategy is generated based on the target region data and the non-target region data, and a preset edge-side multimodal large model is invoked to execute the dynamic computing power allocation strategy, including: The current edge status parameters of the industrial mobile inspection device are obtained, including available memory bandwidth, chip temperature, and remaining power. Based on the terminal state parameters and the preset target end-to-end delay requirements, calculate the current available computing power limit allocated to the multimodal interactive data block; Prioritize allocating a first computing power quota that meets the preset floating-point precision operation requirements to the data in the target area; The remaining computing power budget after deducting the first computing power quota from the current available computing power limit is determined as the second computing power quota allocated to the non-target area data; The dynamic computing power allocation strategy is generated by combining the first computing power quota and the second computing power quota.
[0009] Optionally, based on the dynamic computing power allocation strategy, a high-precision floating-point calculation module is invoked to extract features from the target region data to obtain a target feature representation, and a low-precision integer calculation module is invoked to extract features from the non-target region data to obtain a background feature representation, including: According to the first computing power quota in the dynamic computing power allocation strategy, the high-precision floating-point computing module with the corresponding number of threads is activated in the multimodal large model on the edge side; The high-precision floating-point calculation module performs floating-point tensor operations on the target region data and outputs the target feature representation. According to the second computing power quota in the dynamic computing power allocation strategy, the corresponding number of low-precision integer calculation modules are activated in the multimodal large model on the edge side; The low-precision integer calculation module performs quantization-level tensor operations on the non-target region data to output the background feature representation.
[0010] Optionally, the target feature representation and the background feature representation are subjected to contextual association analysis, and combined with the abnormal sound and image locations, the abnormal event type and spatial coordinates of the abnormal event in the inspection environment are determined, including: The target feature representation and the background feature representation are concatenated across modalities to generate global scene features; Calculate the similarity score between the global scene features and various event reference features in the pre-set industrial abnormal event knowledge base; If the highest similarity score is not less than the preset recognition threshold, the event category corresponding to the highest similarity score is determined as the abnormal event type; otherwise, an unknown anomaly warning operation is triggered. The camera intrinsic parameter matrix, synchronous positioning and mapping pose data of the equipment, and depth measurement values corresponding to the abnormal locations of sound and image are acquired by the industrial mobile inspection equipment. Based on the camera intrinsic parameter matrix, the device synchronous positioning and mapping pose data, and the depth measurement value, the two-dimensional coordinates of the abnormal sound and image location in the pixel coordinate system are projected and transformed to the three-dimensional world coordinate system to obtain the spatial coordinates of the abnormal event.
[0011] Secondly, this application provides an edge-side multimodal large model real-time processing system, including: The acquisition module is used to acquire the continuous video stream and the accompanying audio stream of the current inspection environment, and synchronously cut the continuous video stream and the accompanying audio stream into multiple multimodal interactive data blocks according to a preset time length. The detection module is used to detect the degree of abrupt change in sound energy and the degree of change in image contour in the multimodal interactive data block; The segmentation module is used to determine the abnormal locations of sound and image based on the degree of change in sound energy and the degree of change in image contour, and to divide the multimodal interactive data block into target region data and non-target region data. The generation execution module is used to generate a dynamic computing power allocation strategy based on the target area data and the non-target area data, and to call a preset edge multimodal large model to execute the dynamic computing power allocation strategy. The extraction module is used to, based on the dynamic computing power allocation strategy, call the high-precision floating-point calculation module to extract features from the target area data to obtain target feature representation, and call the low-precision integer calculation module to extract features from the non-target area data to obtain background feature representation; The analysis and determination module is used to perform contextual correlation analysis between the target feature representation and the background feature representation, and combine the abnormal location of the sound and image to determine the type of abnormal event and the spatial coordinates of the abnormal event in the inspection environment.
[0012] Thirdly, this application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement a real-time processing method for edge-side multimodal large models as described in the first aspect above.
[0013] Fourthly, this application provides a computer storage medium storing a computer program, which, when executed by a computer, implements a real-time processing method for edge-side multimodal large models as described in the first aspect.
[0014] This application enables the correlation between sound anomalies and image anomalies by synchronously cutting continuous video streams and accompanying audio streams on the same time axis, thereby reducing the possibility of mismatch between sound changes and image changes occurring at different times.
[0015] This application combines the degree of abrupt change in sound energy, the degree of change in image contour, the azimuth of the sound source, and the bounding box of image anomalies to determine the target region data, so that high-precision computing resources are concentrated on data directly related to the abnormal event.
[0016] This application dynamically determines the computing power quota based on available memory bandwidth, chip temperature, remaining power, and target end-to-end latency requirements. It uses high-precision floating-point calculation for data in the target area and low-precision integer calculation for data in non-target areas, thereby reducing background data processing overhead while preserving anomaly details.
[0017] This application determines the type of abnormal event through contextual correlation analysis of target feature representation and background feature representation, and can use the inspection environment and equipment operating status to help interpret local abnormal features.
[0018] This application calculates the spatial coordinates of abnormal events based on the camera intrinsic parameter matrix, equipment synchronous positioning and mapping pose data, and depth measurement values. The actual location of abnormal events can be marked on the inspection map, providing a location basis for subsequent review and maintenance.
[0019] These or other aspects of this application will become more apparent from the description of the following embodiments. Attached Figure Description
[0020] Figure 1 This is a flowchart illustrating the real-time processing method for multimodal large models on the edge side of this application.
[0021] Figure 2 This is a schematic diagram of the structure of the real-time processing system for multimodal large models on the end side of this application.
[0022] Figure 3 This is a schematic diagram illustrating the construction of the multimodal interactive data block in this application.
[0023] Figure 4 This is a schematic diagram illustrating the determination of abnormal locations in sound and images and the division of data regions in this application.
[0024] Figure 5 This is a schematic diagram of dynamic computing power allocation and mixed precision feature extraction in this application.
[0025] Figure 6 This is a schematic diagram illustrating the determination of abnormal event types and the calculation of spatial coordinates for abnormal events in this application. Detailed Implementation
[0026] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only used to explain this application and are not intended to limit the scope of protection of this application.
[0027] The industrial mobile inspection equipment in this embodiment can be a wheeled inspection equipment, a tracked inspection equipment, or a rail-mounted inspection equipment. The industrial mobile inspection equipment is equipped with an acoustic sensor array, a vision sensor, a depth measurement component, a synchronous positioning and mapping component, and an edge computing component.
[0028] The acoustic sensor array includes at least two acoustic sensors spaced at a known interval for acquiring the accompanying audio stream and determining the azimuth angle of the sound source of abnormal sounds. A vision sensor is used to acquire a continuous video stream. A depth measurement component is used to acquire depth measurements of target points within the field of view of the vision sensors. A simultaneous localization and mapping component is used to output the position and orientation of the industrial mobile inspection equipment in a three-dimensional world coordinate system. An edge computing component is used to run an edge-side multimodal large model and perform the following six steps.
[0029] Step S101: Obtain the continuous video stream and accompanying audio stream of the current inspection environment, and synchronously cut the continuous video stream and the accompanying audio stream into multiple multimodal interactive data blocks according to a preset time length.
[0030] Industrial mobile inspection equipment collects the accompanying audio stream of the current inspection environment through an acoustic sensor array, and simultaneously collects the continuous video stream of the current inspection environment through a visual sensor.
[0031] In this embodiment, the accompanying audio stream refers to the continuous audio sampling data synchronously acquired by the acoustic sensor array during continuous video stream acquisition. Synchronous acquisition means that the accompanying audio stream and the continuous video stream use the same device clock or the same time reference after time correction, and does not require the acoustic sensor array and the vision sensor to start at exactly the same physical moment.
[0032] The edge computing component sets timestamps for both audio sampling data and video frames, and establishes a common timeline based on the same device clock. This common timeline means that both the audio sampling data accompanying the audio stream and the video frames in the continuous video stream can be mapped to a unified acquisition time via timestamps.
[0033] The edge computing component establishes a sliding time window with a preset time length on the same time axis and controls the sliding time window to slide in steps according to a preset sliding step size.
[0034] In one implementation, the preset time length is 1 second, and the preset sliding step size is 0.2 seconds. When the visual sensor's acquisition frame rate is 25 frames per second and the accompanying audio stream's sampling rate is 16 kHz, one sliding time window includes 25 video frames and 16,000 audio sampling points. The above values are merely examples and can be adjusted according to the duration of the abnormal event, the moving speed of the industrial mobile inspection equipment, and the processing power of the edge computing components.
[0035] The edge computing component extracts video frame sequences from the continuous video stream and time-aligned audio segments from the accompanying audio stream based on the start and end times of the sliding time window. The edge computing component then combines the audio segments, video frame sequences, and their timestamps to construct a multimodal interactive data block.
[0036] In this embodiment, a multimodal interactive data block refers to a data processing unit formed within the same sliding time window, comprising at least one audio segment and one video frame sequence. "Interaction" indicates that the audio segment and video frame sequence are associated in subsequent processing through temporal and spatial relationships, but does not imply communication between the two data streams.
[0037] Multimodal interaction data blocks may include the following data fields: By constructing multimodal interactive data blocks along the same timeline, the edge computing component can determine in subsequent steps whether sound changes and image changes occur within the same time range, thus avoiding incorrect association between sound changes and image changes occurring at different times.
[0038] Step S102: Detect the degree of abrupt change in sound energy and the degree of change in image contour in the multimodal interactive data block.
[0039] The edge computing component extracts audio segments from the multimodal interaction data block and divides the audio segments into multiple audio frames. In one implementation, each audio frame has a length of 25 milliseconds, and the frame shift between adjacent audio frames is 10 milliseconds.
[0040] The edge computing component calculates the short-time energy of each audio frame. The short-time energy of each audio frame is: in, Indicates the first The short-time energy of an audio frame. Indicates the first The first audio frame Each audio sample value, This indicates the number of audio sample points in an audio frame.
[0041] In this embodiment, the short-time energy variation rate refers to the ratio of the short-time energy change between adjacent audio frames to a reference short-time energy, used to describe the relative degree of change in sound energy over a short period of time. The short-time energy variation rate is: in, Indicates the short-term energy change rate. This represents a stable term with the same dimensions as the short-time energy, determined by the product of a reference short-time energy and a preset dimensionless positive number. Therefore, the short-time energy and the stable term in the denominator can be added together, and the rate of change of the short-time energy is a dimensionless parameter.
[0042] The edge computing component performs a fast Fourier transform on the audio frames to obtain the amplitude spectrum of each audio frame, and then normalizes the amplitude spectrum.
[0043] In this embodiment, audio spectral flux refers to the positive change between the normalized amplitude spectra of adjacent audio frames, used to reflect sudden changes in the sound spectral structure. Mechanical collisions, abnormal friction, and gas leaks typically manifest not only as energy increases but also as changes in spectral structure. Therefore, using both short-time energy variation rate and audio spectral flux simultaneously can reduce misjudgments caused by continuous stable environmental noise.
[0044] The edge computing component normalizes the short-time energy variation rate and audio spectral flux to a value range of 0 to 1, and then performs weighted fusion according to preset audio weights to obtain the degree of sound energy abrupt change: in, Indicates the degree of abrupt change in sound energy. This represents the normalized short-time energy change rate. This represents the normalized audio spectral flux. and This indicates the preset audio weights, and the sum of the two is 1.
[0045] Since all parameters involved in the weighting are dimensionless, the degree of change in sound energy is also dimensionless. In this embodiment, the degree of change in sound energy does not simply represent the volume level, but rather the combined change in short-term energy and spectral structure between adjacent audio frames.
[0046] In one implementation, It is 0.6. The value is 0.4. If the normalized short-time energy variation rate is 0.8 and the normalized audio spectral flux is 0.6, then the degree of abrupt change in sound energy is 0.72.
[0047] For video frame sequences, the edge computing component first uses a feature point matching algorithm to determine the global transformation relationship between adjacent video frames, and then performs motion compensation according to the global transformation relationship. Motion compensation is used to eliminate the displacement of the entire image caused by the movement of the industrial mobile inspection equipment itself or the jitter of the vision sensor.
[0048] After motion compensation is completed, the edge computing component uses an optical flow algorithm to calculate the motion vector of each pixel in adjacent video frames. The optical flow algorithm can be either the Lucas-Kanade pyramid optical flow algorithm or the Farneback dense optical flow algorithm.
[0049] In this embodiment, the optical flow anomaly index refers to the residual motion vector obtained by subtracting the background optical flow vector of the corresponding image region from the local optical flow vector. The optical flow anomaly index is used to represent the direction and magnitude of motion of a local object relative to the background.
[0050] Since optical flow anomaly indices cannot be directly weighted with scalars, the edge computing component first calculates the magnitude of the optical flow anomaly index and then divides the magnitude by the diagonal pixel length of the video frame to obtain a dimensionless optical flow anomaly index.
[0051] The edge computing component uses the Canny edge detection algorithm to extract edge pixels of adjacent video frames and matches the corresponding edges in adjacent video frames after motion compensation.
[0052] In this embodiment, the edge matching residual refers to the proportion of edge pixels that failed to complete a match in adjacent video frames out of the total number of edge pixels involved in the matching. The edge matching residual is used to describe the addition, disappearance, or shape change of an object's outline, and it is a dimensionless parameter.
[0053] The edge computing component normalizes the optical flow anomaly index and edge matching residual to a value range of 0 to 1, and performs weighted fusion according to preset image weights to obtain the degree of image contour change: in, Indicates the degree of change in image contours. This represents the normalized optical flow anomaly index. This represents the normalized edge matching residual. and This represents the preset image weights, and the sum of the two is 1.
[0054] In this embodiment, the degree of image contour change refers to the combined degree of motion change and edge shape change in a local image area after excluding the influence of the movement of the industrial mobile inspection equipment itself.
[0055] Step S103: Determine the abnormal locations of sound and image based on the degree of change in sound energy and the degree of change in image contour, and divide the multimodal interaction data block into target region data and non-target region data.
[0056] The edge computing component compares the degree of change in sound energy with a preset sound threshold. When the degree of change in sound energy exceeds the preset threshold, the edge computing component extracts the audio timestamp of the corresponding audio frame. The time range corresponding to multiple audio frames that consecutively exceed the preset sound threshold is determined as an abnormal sound time period.
[0057] The preset sound threshold can be set based on normal inspection audio samples. The edge computing component can calculate the mean and standard deviation of the degree of sound energy mutation in the normal inspection audio samples, and set the sum of the mean and the preset multiple standard deviation as the preset sound threshold.
[0058] The end-side computing component uses a multi-channel phase difference algorithm to calculate the azimuth angle of the abnormal sound source during the abnormal sound period. The multi-channel phase difference algorithm can employ a generalized cross-correlation phase transform algorithm.
[0059] The generalized cross-correlation phase transform algorithm determines the time difference between the arrival of abnormal sounds at different acoustic sensors by calculating the phase difference between the audio signals received by different acoustic sensors in the acoustic sensor array. The end-side computing component then calculates the azimuth angle of the abnormal sound relative to the industrial mobile inspection equipment based on the geometry of the acoustic sensor array, the spacing between the acoustic sensors, and the speed of sound propagation.
[0060] For a linear array consisting of two acoustic sensors, the azimuth angle of the sound source can be calculated using the following formula: in, Indicates the azimuth angle of the sound source. Indicates the speed of sound propagation. This indicates the time difference between the arrival of the abnormal sound at the two acoustic sensors. This indicates the distance between the two acoustic sensors. Because... and The units for both are meters, and the ratio of the two is a dimensionless parameter, which meets the input requirements of the arcsine function.
[0061] The edge computing component compares the degree of image contour change with a preset image threshold. When the degree of image contour change exceeds the preset threshold, the edge computing component extracts the video timestamp of the corresponding video frame and performs connected component analysis on the pixel regions that exceed the preset threshold.
[0062] The edge computing component calculates the minimum bounding rectangle of each connected component and determines the minimum bounding rectangle that meets the preset area condition as the image anomaly bounding box.
[0063] In this embodiment, the image anomaly bounding box refers to the rectangular area that surrounds the abnormal image region in the pixel coordinate system of the video frame, including at least the upper left pixel coordinates, the lower right pixel coordinates, the center pixel coordinates, and the corresponding video timestamp.
[0064] The edge computing component aligns the audio timestamps with the video timestamps. When the time range of the abnormal sound corresponding to the audio timestamp overlaps with the time range of the video frame corresponding to the video timestamp, the edge computing component further performs spatial consistency matching.
[0065] In this embodiment, spatial consistency matching refers to determining whether the spatial direction indicated by the azimuth angle of the sound source and the visual direction corresponding to the abnormal bounding box of the image point to the same inspection area.
[0066] Specifically, the edge computing component converts the center pixel coordinates of the image anomaly bounding box into a visual ray based on the camera intrinsic parameter matrix of the visual sensor. Then, according to a pre-determined installation relationship between the visual sensor and the acoustic sensor array, it converts the visual ray to the device coordinate system where the acoustic sensor array is located. The edge computing component calculates the angular difference between the azimuth angle of the visual ray and the azimuth angle of the sound source. When the angular difference is not greater than a preset direction tolerance, it is determined that the azimuth angle of the sound source and the image anomaly bounding box satisfy the spatial consistency condition.
[0067] When the time period of the sound anomaly meets the time alignment condition with the time range of the video frame, and the azimuth of the sound source meets the spatial consistency condition with the bounding box of the image anomaly, the edge computing component generates the sound and image anomaly locations.
[0068] In this embodiment, the abnormal locations of sound and images are not single coordinate points, but rather spatiotemporal data of sound and images used to describe the same abnormal event, which includes at least: The edge computing component merges video data located within the bounding box of image anomalies and audio data within the corresponding time period of sound anomalies into target region data.
[0069] In this embodiment, the target region data refers to video region data and audio time interval data directly related to candidate abnormal events, determined based on the location of sound and image anomalies. The region includes both spatial regions within video frames and temporal intervals within the accompanying audio stream.
[0070] The edge computing component combines the remaining background data after stripping the target region data from the multimodal interactive data block into non-target region data.
[0071] In this embodiment, non-target area data refers to data in the multimodal interaction data block other than the target area data. Non-target area data is still used to describe equipment type, inspection location, environmental structure, and normal operating status; it is not invalid data.
[0072] Step S104: Generate a dynamic computing power allocation strategy based on the target area data and the non-target area data, and call the preset edge multimodal large model to execute the dynamic computing power allocation strategy.
[0073] The edge computing component acquires the current edge status parameters of the industrial mobile inspection equipment.
[0074] In this embodiment, the edge status parameters refer to a set of parameters that reflect the current resources and operating status of the edge computing component, including at least available memory bandwidth, chip temperature, and remaining power.
[0075] Available memory bandwidth refers to the data transfer capacity of the edge computing component for model inference within the current time period. Chip temperature refers to the processor temperature in the edge computing component that performs model inference. Remaining power refers to the percentage of the industrial mobile inspection equipment's current remaining power relative to its rated power.
[0076] The edge computing component also acquires the target end-to-end latency requirement. In this embodiment, the target end-to-end latency requirement refers to the longest processing time allowed from the completion of the multimodal interaction data block construction to the output of the abnormal event identification result.
[0077] Before device deployment, the edge computing components are tested for computation time, memory bandwidth usage, and power consumption under different thread counts, computational precisions, and input data volumes, and a computing power overhead mapping table is established. An example of a computing power overhead mapping table is shown below: The relative computing overhead in the table is dimensionless, using the first-level benchmark as a reference, to compare resource consumption across different computing configurations. In actual deployment, the relative computing overhead can be replaced with the equivalent tensor operations per second or the processing time required to complete the corresponding operation.
[0078] The edge computing component selects computing configurations from the computing power overhead mapping table that can be completed within the target end-to-end latency and do not exceed the current resource limits, based on available memory bandwidth, chip temperature, remaining power, and target end-to-end latency requirements.
[0079] In this embodiment, the current available computing power limit refers to the maximum equivalent computing power that the edge computing component can use for the current multimodal interactive data block, provided that the target end-to-end latency requirement is met and the current available memory bandwidth, temperature limit, and power limit are not exceeded.
[0080] The edge computing component prioritizes allocating the first computing power quota that meets the preset floating-point precision operation requirements to the target area data.
[0081] In this embodiment, the first computing power quota refers to the high-precision floating-point computing resources allocated to the target region data from the current available computing power limit, including at least floating-point calculation precision, number of threads, number of model calculation layers, input resolution, and maximum execution time.
[0082] The edge computing component deducts the first computing power quota from the current available computing power limit and determines the remaining computing power budget as the second computing power quota.
[0083] In this embodiment, the second computing power quota refers to the low-precision integer computing resources allocated to non-target region data from the current available computing power limit, including at least integer computing bit width, number of threads, number of model computing layers, input resolution, and maximum execution time.
[0084] When the chip temperature exceeds a preset temperature threshold or the remaining power is lower than a preset power threshold, the edge computing component can reduce the second computing power quota and prioritize maintaining the first computing power quota to avoid excessive compression of critical abnormal features due to insufficient computing resources.
[0085] The edge computing component combines the first computing power quota and the second computing power quota to generate a dynamic computing power allocation strategy.
[0086] The dynamic computing power allocation strategy in this embodiment refers to the computing resource configuration data formed for the current multimodal interaction data block. This includes at least the computing precision and number of threads for the target region data, the computing precision and number of threads for the non-target region data, the processing order of the two types of data, and their respective execution time limits. "Dynamic" means that this strategy is regenerated based on the current multimodal interaction data block and the current edge-side state parameters, rather than remaining fixed indefinitely.
[0087] The edge computing component calls the preset edge multimodal large model to execute the dynamic computing power allocation strategy.
[0088] In this embodiment, the edge-side multimodal large model refers to a neural network model deployed in the edge-side computing component of the industrial mobile inspection equipment, which can jointly process video data, audio data, and spatial location data.
[0089] The edge-side multimodal large model includes a visual feature encoding network, an audio feature encoding network, a feature projection network, a cross-modal fusion network, and an event recognition network.
[0090] The end-side multimodal large model is constructed according to the following process: First, collect historical industrial inspection data. This data includes normal operation data, equipment noise data, mechanical collision data, liquid leakage data, gas leakage data, smoke data, spark data, abnormal vibration data, and personnel intrusion data.
[0091] Secondly, historical industrial inspection data should be labeled. The labeling information should include at least the event category, the time of the event, the bounding box of the image anomaly, the azimuth of the sound source, and the spatial coordinates of the anomaly event.
[0092] Furthermore, the visual feature coding network and audio feature coding network are trained using labeled historical industrial inspection data. The visual feature coding network is able to extract image texture, contour and motion features, while the audio feature coding network is able to extract audio energy, spectrum and sound source direction features.
[0093] Then, a feature projection network and a cross-modal fusion network are trained to map video and audio features to the same feature dimension and reduce the distance between video and audio features belonging to the same anomalous event in the feature space.
[0094] Finally, the event recognition network is trained to determine the type of abnormal event based on the global scene features output by the cross-modal fusion network.
[0095] After training, the edge-side multimodal large model is validated using validation data that was not used in the training, and the network layers capable of performing low-precision integer operations are quantized and calibrated. Through the above training, the edge-side multimodal large model establishes the correspondence between target region features, background context, and industrial anomaly event types, thus enabling it to perform subsequent feature extraction and anomaly event recognition.
[0096] Step S105: Based on the dynamic computing power allocation strategy, the high-precision floating-point calculation module is called to extract features from the target area data to obtain target feature representation, and the low-precision integer calculation module is called to extract features from the non-target area data to obtain background feature representation.
[0097] Based on the first computing power quota in the dynamic computing power allocation strategy, the edge-side multimodal large model activates the corresponding number of high-precision floating-point computing modules in the edge-side computing component.
[0098] In this embodiment, the high-precision floating-point calculation module refers to the computation path in the edge-side multimodal large model that uses 16-bit or 32-bit floating-point numbers to perform tensor operations. This module can be executed by a processor core, graphics processing unit core, or neural network processor core that supports floating-point operations, and is not limited to a standalone physical chip.
[0099] The edge-side multimodal large model performs floating-point level tensor operations on the target region data through a high-precision floating-point computing module.
[0100] In this embodiment, floating-point tensor operations refer to convolution, matrix multiplication, attention operations, normalization operations, and feature projection operations performed in floating-point format. Using floating-point tensor operations to process target region data can preserve the numerical differences in features such as small-scale sparks, minute cracks, weak leakage sounds, and localized abnormal vibrations.
[0101] The high-precision floating-point calculation module outputs a representation of the target features.
[0102] In this embodiment, the target feature representation refers to the feature tensor extracted from the target region data by the edge multimodal large model. The target feature representation includes at least the target image texture features, target contour features, target motion features, abnormal sound spectrum features, and sound source direction features.
[0103] According to the second computing power quota in the dynamic computing power allocation strategy, the edge multimodal large model activates the corresponding number of low-precision integer computing modules in the edge computing component.
[0104] In this embodiment, the low-precision integer calculation module refers to the computation path in the edge-side multimodal large model that uses 8-bit or 16-bit integers to perform quantization tensor operations. This module and the high-precision floating-point calculation module can share the model network structure, but use different data formats and computation kernels.
[0105] The low-precision integer calculation module converts floating-point input into integer input using a pre-determined quantization scaling factor and quantization zero point. The quantization scaling factor and quantization zero point are determined using quantization calibration data, which includes both normal and abnormal inspection data.
[0106] The edge-side multimodal large model performs quantization-level tensor operations on non-target region data through a low-precision integer computation module.
[0107] In this embodiment, quantized tensor operations refer to convolution, matrix multiplication, and attention operations performed in quantized integer format. The low-precision integer calculation module outputs a background feature representation.
[0108] In this embodiment, the background feature representation refers to the feature tensor extracted from non-target region data by the end-side multimodal large model. The background feature representation includes at least the inspection environment category, equipment background structure, normal operation sound, and non-abnormal movement information.
[0109] Before performing context association analysis, the edge multimodal large model dequantizes the background feature representation according to the quantization scaling factor, and then uses a feature projection network to convert the target feature representation and the background feature representation into the same data format and feature dimension. This process avoids the inability to directly associate the two types of features due to differences in computational precision and feature dimension.
[0110] Step S106: Perform context association analysis on the target feature representation and the background feature representation, and combine the abnormal location of the sound and image to determine the abnormal event type and spatial coordinates of the abnormal event in the inspection environment.
[0111] The edge-side multimodal large model inputs the target feature representation and background feature representation into the cross-modal fusion network.
[0112] The cross-modal fusion network first performs cross-modal feature concatenation between the target feature representation and the background feature representation to generate global scene features.
[0113] In this embodiment, cross-modal feature stitching refers to combining image features, audio features, target region features, and background features that have undergone time alignment and feature space mapping into a unified feature sequence according to a preset feature dimension.
[0114] In this embodiment, the global scene feature refers to a fused feature that simultaneously includes local information about the abnormal target and background information about the inspection environment. The global scene feature is used to prevent the end-side multimodal large model from determining the event type solely based on local changes. For example, the same foggy image area might be considered normal steam discharge near normal exhaust equipment, but it might be considered a gas leak near a closed gas pipeline accompanied by abnormal leakage sounds.
[0115] In this embodiment, context association analysis refers to the process of calculating the association weights between the target feature representation and the background feature representation through a cross-modal fusion network on the edge multimodal large model, and forming global scene features based on the association weights.
[0116] The edge-side multimodal large model calculates the similarity score between global scene features and various event reference features in the industrial abnormal event knowledge base.
[0117] In this embodiment, the industrial anomaly event knowledge base refers to a data set that stores the correspondence between industrial anomaly event categories, event reference features, event names, and handling identifiers. The industrial anomaly event knowledge base is not simply a text rule base; it includes at least event reference features that can be used for similarity calculations.
[0118] In this embodiment, the event reference feature refers to the category representative feature obtained by encoding and clustering multiple historical abnormal event samples of the same category using an edge-side multimodal large model.
[0119] The industrial anomaly event knowledge base is constructed according to the following process: First, the historical industrial inspection data is classified according to the type of abnormal event; Secondly, global scene features of each historical industrial inspection data are extracted through the trained edge multimodal large model; Next, cluster the global scene features of the same abnormal event type; Finally, the global scene feature or cluster center feature closest to the cluster center is determined as the event reference feature for the corresponding abnormal event type.
[0120] The edge-side multimodal large model can use cosine similarity to calculate the similarity score between global scene features and event reference features: in, Represents global scene features. Indicates the first The event reference features for anomalous events. Both the numerator and denominator are calculated from vectors in the same feature space, and the resulting similarity score is a dimensionless parameter.
[0121] When the highest similarity score is not less than the preset recognition threshold, the edge multimodal big model will determine the event category corresponding to the highest similarity score as the abnormal event type.
[0122] In this embodiment, the abnormal event type refers to the industrial abnormality category determined by the end-side multimodal large model, which may include abnormal equipment noise, mechanical collision, liquid leakage, gas leakage, smoke, sparks, abnormal vibration, and personnel intrusion.
[0123] When the highest similarity score is less than the preset recognition threshold, the edge computing component triggers an unknown anomaly warning operation.
[0124] In this embodiment, the unknown anomaly warning operation refers to the processing operation performed when the current global scene features cannot reliably match the known abnormal event type. This includes saving the multimodal interaction data block, recording the abnormal location of sound and image, recording the pose data of the industrial mobile inspection equipment, and sending warning information to the inspection management terminal.
[0125] When determining the spatial coordinates of anomalies, the edge computing component extracts the bounding box of the image anomaly from the location of the sound and image anomalies, and determines the pixel coordinates of the center point of the bounding box as two-dimensional coordinates in the pixel coordinate system. The azimuth angle of the sound source is used to verify the consistency of the anomaly direction and is not directly used as the two-dimensional pixel coordinates.
[0126] The edge computing component acquires the camera intrinsic parameter matrix, synchronous localization and mapping pose data, and depth measurement values corresponding to the two-dimensional coordinates from the vision sensor.
[0127] In this embodiment, the camera intrinsic parameter matrix refers to the matrix describing the mapping relationship between the focal length, principal point position, and pixel coordinates of the visual sensor. The camera intrinsic parameter matrix is obtained by determining the visual sensor.
[0128] In this embodiment, the synchronous positioning and mapping pose data refers to the position and orientation of the industrial mobile inspection equipment in the three-dimensional world coordinate system output by the synchronous positioning and mapping component. The end-side computing component converts the pose of the industrial mobile inspection equipment into the pose of the visual sensor based on a pre-determined fixed installation relationship between the vision sensor and the industrial mobile inspection equipment.
[0129] In this embodiment, the depth measurement value refers to the spatial distance from the center point of the image anomaly bounding box to the visual sensor or the depth along the optical axis of the visual sensor, and its unit is meters.
[0130] The edge computing component back-projects the pixel's two-dimensional coordinates onto the vision sensor coordinate system based on the pixel's two-dimensional coordinates, the camera's intrinsic parameter matrix, and the depth measurement value, to obtain the three-dimensional coordinates in the vision sensor coordinate system: in, Represents the three-dimensional coordinates in the vision sensor coordinate system. Indicates the depth measurement value. Represents the camera intrinsic parameter matrix. and Represents the two-dimensional coordinates of a pixel.
[0131] The inverse of the camera intrinsic matrix converts the pixel's two-dimensional coordinates into a dimensionless line-of-sight direction. The line-of-sight direction is then multiplied by a depth measurement in meters to obtain the resulting three-dimensional coordinates, which are in meters.
[0132] The edge computing component transforms the 3D coordinates in the visual sensor coordinate system to the 3D world coordinate system based on the synchronous positioning and mapping pose data: in, Represents coordinates in a three-dimensional world coordinate system. Represents a dimensionless rotation matrix. This represents a translation vector in meters. and Having the same dimensions, they can be added together.
[0133] In this embodiment, the spatial coordinates of the abnormal event refer to the position coordinates of the abnormal event in a pre-established three-dimensional world coordinate system. These coordinates can be used to display the abnormal location on the inspection map, guide industrial mobile inspection equipment for re-inspection, or guide maintenance personnel to the abnormal location.
[0134] like Figure 2 As shown, the edge-side multimodal large model real-time processing system includes an acquisition module 201, a detection module 202, a partitioning determination module 203, a generation and execution module 204, an extraction module 205, and an analysis and determination module 206.
[0135] The acquisition module 201 executes step S101; the detection module 202 executes step S102; the determination and division module 203 executes step S103; the generation and execution module 204 executes step S104; the extraction module 205 executes step S105; and the analysis and determination module 206 executes step S106.
[0136] The modules mentioned above can be software modules divided according to function, or processing units formed by combining software and hardware. Module division does not mean that each module must be set up in a different physical device.
[0137] When the processing component in the computing device calls the computer instructions in the storage component, steps S101 to S106 can be executed sequentially. When a computer program stored in the computer storage medium is executed by the computer, steps S101 to S106 can also be implemented.
[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A real-time processing method for end-side multimodal large models, characterized in that, include: Acquire the continuous video stream and accompanying audio stream of the current inspection environment, and synchronously cut the continuous video stream and the accompanying audio stream into multiple multimodal interactive data blocks according to a preset time length; Detect the degree of abrupt change in sound energy and the degree of change in image contours in the multimodal interactive data block; Based on the degree of change in sound energy and the degree of change in image contour, the location of sound and image anomalies is determined, and the multimodal interaction data block is divided into target region data and non-target region data. A dynamic computing power allocation strategy is generated based on the target region data and the non-target region data, and the dynamic computing power allocation strategy is executed by calling the preset edge multimodal large model. Based on the dynamic computing power allocation strategy, the high-precision floating-point calculation module is called to extract features from the target area data to obtain target feature representation, and the low-precision integer calculation module is called to extract features from the non-target area data to obtain background feature representation; The target feature representation and the background feature representation are subjected to contextual correlation analysis, and combined with the abnormal location of the sound and image, the abnormal event type and spatial coordinates of the abnormal event in the inspection environment are determined.
2. The method according to claim 1, characterized in that, Acquire a continuous video stream and accompanying audio stream of the current inspection environment, and synchronously segment the continuous video stream and accompanying audio stream into multiple multimodal interactive data blocks according to a preset time length, including: The acoustic sensor array and vision sensor mounted on the industrial mobile inspection equipment synchronously collect the accompanying audio stream and the continuous video stream of the current inspection environment; On the same timeline of the accompanying audio stream and the continuous video stream, a sliding time window with the preset time length is established; The sliding time window is controlled to slide step by step on the same time axis; The time-aligned audio segments and video frame sequences are extracted using the sliding time window. The audio clips and the video frame sequence are combined to construct the multimodal interactive data block.
3. The method according to claim 2, characterized in that, Detecting the degree of abrupt changes in sound energy and the degree of changes in image contours in the multimodal interactive data block includes: Calculate the short-time energy variation rate and audio spectral flux of the audio segment in the multimodal interactive data block; The weighted fusion value of the short-time energy variation rate and the audio spectral flux is determined as the degree of sound energy mutation. Extract the optical flow anomaly index and edge matching residual of the video frame sequence from the multimodal interactive data block; The weighted fusion value of the optical flow anomaly index and the edge matching residual is determined as the degree of image contour change.
4. The method according to claim 3, characterized in that, Based on the degree of abrupt change in sound energy and the degree of change in image contour, the location of sound and image anomalies is determined, and the multimodal interaction data block is divided into target region data and non-target region data, including: Extract the audio timestamps of sound energy mutations that exceed a preset threshold, and calculate the azimuth angle of the corresponding abnormal sound source using a multi-channel phase difference algorithm; Extract the video timestamps and corresponding image anomaly bounding boxes whose image contour changes are greater than a preset threshold. The audio timestamp and the video timestamp are time-aligned, and the spatial consistency of the sound source azimuth angle and the image anomaly bounding box is matched to generate the sound and image anomaly locations. The video data located within the image anomaly bounding box and the audio data within the time period corresponding to the audio timestamp are merged into the target region data; The remaining background data after stripping the target region data from the multimodal interactive data block is combined into the non-target region data.
5. The method according to claim 4, characterized in that, A dynamic computing power allocation strategy is generated based on the target region data and the non-target region data, and a preset edge-side multimodal large model is invoked to execute the dynamic computing power allocation strategy, including: The current edge status parameters of the industrial mobile inspection device are obtained, including available memory bandwidth, chip temperature, and remaining power. Based on the terminal state parameters and the preset target end-to-end delay requirements, calculate the current available computing power limit allocated to the multimodal interactive data block; Prioritize allocating a first computing power quota that meets the preset floating-point precision operation requirements to the data in the target area; The remaining computing power budget after deducting the first computing power quota from the current available computing power limit is determined as the second computing power quota allocated to the non-target area data; The dynamic computing power allocation strategy is generated by combining the first computing power quota and the second computing power quota.
6. The method according to claim 5, characterized in that, Based on the dynamic computing power allocation strategy, the high-precision floating-point calculation module is invoked to extract features from the target region data to obtain a target feature representation, and the low-precision integer calculation module is invoked to extract features from the non-target region data to obtain a background feature representation, including: According to the first computing power quota in the dynamic computing power allocation strategy, the high-precision floating-point computing module with the corresponding number of threads is activated in the multimodal large model on the edge side; The high-precision floating-point calculation module performs floating-point tensor operations on the target region data and outputs the target feature representation. According to the second computing power quota in the dynamic computing power allocation strategy, the corresponding number of low-precision integer calculation modules are activated in the multimodal large model on the edge side; The low-precision integer calculation module performs quantization-level tensor operations on the non-target region data to output the background feature representation.
7. The method according to claim 6, characterized in that, Contextual association analysis is performed between the target feature representation and the background feature representation, and combined with the abnormal location of the sound and image, to determine the type of abnormal event and the spatial coordinates of the abnormal event in the inspection environment, including: The target feature representation and the background feature representation are concatenated across modalities to generate global scene features; Calculate the similarity score between the global scene features and various event reference features in the pre-set industrial abnormal event knowledge base; If the highest similarity score is not less than the preset recognition threshold, the event category corresponding to the highest similarity score is determined as the abnormal event type; otherwise, an unknown anomaly warning operation is triggered. The camera intrinsic parameter matrix, synchronous positioning and mapping pose data of the equipment, and depth measurement values corresponding to the abnormal locations of sound and image are acquired by the industrial mobile inspection equipment. Based on the camera intrinsic parameter matrix, the device synchronous positioning and mapping pose data, and the depth measurement value, the two-dimensional coordinates of the abnormal sound and image location in the pixel coordinate system are projected and transformed to the three-dimensional world coordinate system to obtain the spatial coordinates of the abnormal event.
8. A real-time processing system for edge-side multimodal large models, characterized in that, The system is used to implement the real-time processing method for end-side multimodal large models according to any one of claims 1-7; the system includes: The acquisition module is used to acquire the continuous video stream and the accompanying audio stream of the current inspection environment, and synchronously cut the continuous video stream and the accompanying audio stream into multiple multimodal interactive data blocks according to a preset time length. The detection module is used to detect the degree of abrupt change in sound energy and the degree of change in image contour in the multimodal interactive data block; The segmentation module is used to determine the abnormal locations of sound and image based on the degree of change in sound energy and the degree of change in image contour, and to divide the multimodal interactive data block into target region data and non-target region data. The generation execution module is used to generate a dynamic computing power allocation strategy based on the target area data and the non-target area data, and to call a preset edge multimodal large model to execute the dynamic computing power allocation strategy. The extraction module is used to, based on the dynamic computing power allocation strategy, call the high-precision floating-point calculation module to extract features from the target area data to obtain target feature representation, and call the low-precision integer calculation module to extract features from the non-target area data to obtain background feature representation; The analysis and determination module is used to perform contextual correlation analysis between the target feature representation and the background feature representation, and combine the abnormal location of the sound and image to determine the type of abnormal event and the spatial coordinates of the abnormal event in the inspection environment.
9. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are invoked and executed by the processing component to implement the real-time processing method for edge-side multimodal large models as described in any one of claims 1-7.
10. A computer storage medium, characterized in that, The device contains a computer program that, when executed by a computer, implements a real-time processing method for edge-side multimodal large models as described in any one of claims 1-7.
