An audio and video multi-modal recognition method based on artificial intelligence

Abstract

The application discloses an audio and video multi-modal recognition method based on artificial intelligence, and relates to the technical field of artificial intelligence and biological feature recognition.The application dynamically generates a collection sequence through a reinforcement learning decision engine, and responds to environmental changes in real time.When a fingerprint sensor detects oil stains, the current mode is automatically skipped, and iris or voiceprint collection is preferentially started, thereby avoiding the blockage of a traditional fixed process, reducing the need for manual intervention of an operator, and greatly shortening the single-person verification time.A mode state matrix performs real-time quality scoring on a video, an audio and a biological feature sensor, and in combination with a preset threshold, forces re-collection, so that each biological feature meets the standard before entering the next link.A cross-modal auxiliary mechanism further enhances robustness, and when a video is blocked in a rainstorm, a voiceprint feature is used to drive face repair, so that the accuracy of living body detection is ensured, and the generation of low-quality samples is prevented from the source.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and biometric identification technology, and in particular to an audio-video multi-modal recognition method based on artificial intelligence. BACKGROUND

[0002] In mobile police service, site access and other field verification scenarios, multi-modal biometric feature collection faces strong dynamic environment challenges; for example, in temporary site verification, workers often wear dirty gloves to operate equipment, causing the fingerprint collection area to be covered with oil stains; at the same time, the strong light environment causes iris images to be overexposed, and traditional fixed collection processes, such as certificate -> portrait -> fingerprint -> iris -> voiceprint, are prone to interruption due to single-modal failure; personnel need to repeatedly adjust the device position or manually skip steps, and the process blocking rate is higher, with an average verification time of more than 3-5 minutes.

[0003] Some existing solutions use dynamic weight multi-modal fusion, such as the NVIDIA RAG system, which adjusts the weight of each modality through confidence score, but only solves the weight distribution problem in the feature fusion stage; at the collection execution level, it still relies on the preset order process and cannot dynamically adjust the collection order according to real-time quality; for example, when the fingerprint detector detects oil stains, the system only prompts to retry after cleaning, without starting the iris or voiceprint backup modality collection, causing the process to stall; similarly, the cross-modal attention mechanism proposed by some in 2023 improves the feature alignment accuracy, but does not involve real-time decision-making of the collection strategy.

[0004] Some solutions also introduce parallel collection modes, such as simultaneous iris and voiceprint recording, but multiple data streams exacerbate hardware resource contention, which can easily cause memory overflow on edge devices such as JetsonNano; the quality threshold jump mechanism proposed by some cloud in 2024 allows single-item skipping, but lacks automatic backfilling after jumping, requiring manual triggering of missing modality collection, which increases operational complexity; none of these solutions establish a closed-loop decision chain between modality state and collection scheduling, making it difficult to break through the bottleneck of field verification efficiency. SUMMARY

[0005] In view of the above existing problems, the present application is proposed.

[0006] The present application provides an audio-video multi-modal recognition method based on artificial intelligence to solve the problem of high blocking rate, frequent manual intervention and unstable data quality caused by environmental interference and fixed process in multi-modal collection in the field.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] In a first aspect, the present application provides an audio-video multi-modal recognition method based on artificial intelligence, which includes,

[0009] Acquire multi-modal input including video stream, audio stream and biometric sensor data;

[0010] Construct a modal state matrix and generate a quality score vector for each modal;

[0011] Output modal acquisition sequence instructions from the reinforcement learning decision engine based on the quality score vector and device state;

[0012] Trigger the acquisition module of the corresponding modal in the acquisition sequence;

[0013] When the quality of any modal does not meet the preset threshold, mark the modal as to-be-supplemented and add it to the supplement queue;

[0014] Under the supplement conditions of device idle and stable environment, automatically trigger the reacquisition of the to-be-supplemented modal until the threshold is met.

[0015] As a preferred scheme of the audio and video multi-modal recognition method based on artificial intelligence, the quality score vector is obtained by normalizing and weighting the measurable quality indicators of each modal, wherein the quality indicators of the video stream at least include the key area occlusion ratio and the motion blur degree, the quality indicators of the audio stream at least include the signal-to-noise ratio and the effective speech segment proportion, and the quality indicators of the biometric sensor data at least include the contact surface pollution coverage or the environmental interference intensity;

[0016] The scale and weight of the quality score can be obtained by configuration or offline calibration;

[0017] The normalization and weight aggregation steps of the quality score include: reading the original quality indicators of the video, audio and biometric sensor in the same acquisition cycle, making the direction consistent and interval normalized for each indicator, aggregating the weights in the modal to obtain a single value score, assembling the scores of the three types of modal into a quality score vector, writing into the cache for subsequent state machine and scheduling, processing the missing items with zero weight and re-normalizing the weight in the modal;

[0018] In any modal, the normalized score is linearly aggregated to obtain the modal score:

[0019]

[0020] wherein s m represents the quality score of the modal m, with a value of 0-1, dimensionless, m represents the modal index, taking {video, audio, biometric feature}, j represents the index of the indicator, with a unit of item sequence, J m represents the number of indicators of the modal m, with a unit of item, w m,j represents the weight of the jth indicator in the modal m, with a value of 0-1, dimensionless, satisfying denotes the normalized mapping of the jth original indicator, outputting a dimensionless score between 0 and 1, x m,j denotes the observed value of the jth original quality indicator, with the same unit as the indicator;

[0021] wherein:

[0022]

[0023] wherein, g m,j (x) denotes the mapping of observation x to a dimensionless score, x denotes the observation of the indicator in the current period, with the same unit as x m,j , l m,j , u m,j denote the engineering lower and upper limits of the indicator, respectively, with the same unit as x, clip(·) denotes the operator that truncates the input to the interval [0, 1], d m,j denotes the direction marker, with a value set of two types of size relations;

[0024] Quality score vector assembly: the video modality score, the audio modality score, and the biological feature modality score are assembled into a three-dimensional column vector in a fixed order.

[0025] As a preferred scheme of the audio-video multi-modal recognition method based on artificial intelligence, each modality corresponding to the modality state matrix is managed according to a preset state machine, and the states at least include to-be-collected, collecting, meeting the standard, not meeting the standard, to-be-supplemented, and prohibited; state transition is set according to a quality score threshold, a collection time upper limit, and a failure number upper limit, and a hysteresis strategy is used to suppress frequent switching.

[0026] As a preferred scheme of the audio-video multi-modal recognition method based on artificial intelligence, the input of the reinforcement learning decision engine includes a quality score vector, device state, and backfill queue information, and the output is the modality collection priority in the current period; the device state includes at least one of processor load, available memory, bus occupancy, and sensor occupancy; the decision goal considers both the minimization of total time consumption and the minimization of blocking rate, and a penalty is set for frequent switching to ensure time sequence stability;

[0027] The time sequence difference error and reward of the reinforcement learning decision engine are designed as follows:

[0028] In each collection period t, a state s t is first constructed, which is obtained by splicing a quality score vector, a device state, and a backfill queue representation, and an immediate reward r t is calculated according to the action of the previous period and the current state, r tThe evaluation network is updated based on the timing difference error, and the collection priority distribution of the policy network is adjusted accordingly. Finally, the updated weights and thresholds are written to the local cache, and the switching events are recorded for the next period of punishment item calculation;

[0029] In value-based policy optimization, the evaluation error is defined in terms of timing difference:

[0030] δ t =r t +γV φ (s t+1 )-V φ (s t ),

[0031] Where δ t represents the timing difference error at period t, dimensionless, r t represents the immediate reward at period t, scaled to dimensionless normalized value, γ represents the discount factor, taking values 0-1, dimensionless, V φ (·) represents the state value function characterized by the parameter vector φ, outputting an estimate of the long-term return for the state, dimensionless, s t+1 represents the next state after action execution at period t, s t represents the current state at period t; the state s t is defined as a three-part splice: quality score vector q t , equipment state vector e t and backfill queue vector b t ;

[0032] In the formula, the benefits and costs are linearly composed of normalized items:

[0033]

[0034] Where λ q , λ T , λ B , λ S represent the weight coefficients of quality benefits, time cost, blocking cost and switching penalty, respectively, taking non-negative values, dimensionless, derived from configuration issuance or offline calibration, represents the modal score selected by action a t at period t, corresponding to the modal aggregation result of the previous step, taking values 0-1, dimensionless, a t represents the action at period t, which is the engineering meaning of the modal index or priority decision for the current period, represents the normalized amount of collection and scheduling time, taking values 0-1, obtained by comparing the measured seconds with the engineering upper limit, A normalized quantity representing the blocking rate or the queue depth of the current cycle, taking values between 0 and 1, A normalized quantity representing the switching penalty, taking values between 0 and 1.

[0035] As a preferred scheme of the audio-video multi-modal recognition method based on artificial intelligence, the re-sampling trigger recovery condition comprises that the dependent modal of the target modal has reached the standard, the device is within the idle threshold, and the environmental parameters remain stable within the continuous time window; the confirmation signal is automatic confirmation or manual confirmation, and when the timeout is not confirmed, the process is automatically continued according to the degradation strategy.

[0036] As a preferred scheme of the audio-video multi-modal recognition method based on artificial intelligence, the method further comprises a resource arbitration step: according to the resource budget limit and the concurrent degree and buffer depth of the device, the multiple modalities available at the same time are queued and quota allocated, so that the acquisition process runs within the memory and bandwidth threshold, and when the resources are tight, the modalities with higher quality scores or key to the overall process are preferentially guaranteed.

[0037] As a preferred scheme of the audio-video multi-modal recognition method based on artificial intelligence, the method further comprises cross-modal acquisition assistance: when the video key area is blocked or overexposed and the audio quality meets the standard, a face key point heat map is generated based on the pre-trained cross-modal mapping, which is used to guide the framing frame adjustment or region relocation, and after passing the live detection and consistency check, it can participate in the recognition result fusion.

[0038] As a preferred scheme of the audio-video multi-modal recognition method based on artificial intelligence, the training of the strategy network comprises constructing a modal state transition model covering multiple environments and human-computer interaction situations, and using an offline data training and online fine-tuning combined manner to optimize the reward function of the acquisition path, so as to improve the scheduling robustness in outdoor strong light, contact surface pollution and noise background.

[0039] In a second aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the multi-modal biometric feature acquisition and recognition method based on artificial intelligence when executing the computer program.

[0040] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the multi-modal biometric feature acquisition and recognition method based on artificial intelligence.

[0041] The application has the advantages that: the application optimizes the biological feature collection process by the multi-modal biological feature collection and recognition method driven by artificial intelligence; the collection sequence is dynamically generated by the reinforcement learning decision engine, and the environment changes are responded in real time, for example, when the fingerprint sensor detects that the oil stain is covered, the current mode is automatically skipped and the iris or voiceprint collection is preferentially started, the blockage of the traditional fixed process is avoided, the manual intervention demand of the operator is reduced, the single person verification time is greatly shortened, and the application is especially suitable for high dynamic scenes such as construction sites and border inspection.

[0042] The mode state matrix of the application performs real-time quality scoring on video, audio and biological feature sensors, and combines the preset threshold to force re-collection, so that each biological feature (such as face sharpness and voiceprint effectiveness) meets the standard before entering the next link; the cross-modal auxiliary mechanism further enhances the robustness, for example, when the video is blocked in a rainstorm, the face repair is driven by the voiceprint feature to ensure the accuracy of the live detection and prevent the generation of low-quality samples from the source.

[0043] The application significantly improves the adaptability and safety of the system in harsh environments; the reinforcement learning strategy combines device states (such as processor load and memory occupation) and environmental parameters to realize resource-aware scheduling and avoid edge device overload; the automatic backfill mechanism combines dependency checking to ensure that the missing mode is efficiently collected when the device is idle; at the same time, end-to-end encryption and live multi-feature coupling checking (such as voiceprint micro-vibration and iris texture linkage) strengthen the security protection of personal biological information and meet the compliance requirements.

[0044] In summary, the application realizes the intelligentization, continuity and standardization of the collection process by artificial intelligence technology, and provides efficient, reliable and safe identity verification support for mobile police, construction site access and other scenes. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope of the present application.

[0046] Figure 1 The flowchart of the audio-video multi-modal recognition method of the embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical scheme and advantages of the present application more clear, the following will further illustrate the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0048] All terms used herein, including technical and scientific terms, have the meanings as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of the specification, and should not be interpreted in an idealized or overly formal way.

[0049] For example, the terms "first", "second", and the like as used herein merely mean to distinguish between two or more similar objects, to distinguish a first object from a second object, and not to describe a particular sequential or chronological order, nor to indicate or imply a relative importance.

[0050] The present application proposes an audio and video multi-modal recognition method based on artificial intelligence, which combines Figure 1 As shown in the figure, the method comprises the following steps:

[0051] Step 101, obtaining multi-modal input including video stream, audio stream and biological feature sensor data; in this embodiment and specifically, the multi-modal input refers to video, audio and biological feature sensor raw data stream with unified timestamp and synchronization strategy, the time reference is aligned by the device internal clock and compared with the last complete cycle; the collection period is 50ms by default, the adjustable range is 20-100ms, the video frame rate is 20fps by default (adjustable 15-60fps), the audio sampling rate is 16kHz by default (adjustable 8-48kHz), the parameters are determined in the trade-off of time delay and computing power occupation according to external field verification; the environmental parameters involved are used for subsequent stability judgment, and are derived from the sliding estimation of basic quantities such as illuminance, electroacoustic noise and temperature that can be read by the device end. Optionally, in order to reduce the influence of the first frame cold start, a preheating buffer of not more than 500ms is reserved after power-on and unstable samples are discarded; similarly, when the timestamp of any mode exceeds the threshold (the exceeding threshold is ±20ms by default), the current cycle of this mode is marked as missing and participates in the subsequent missing processing.

[0052] Step 102, constructing a modal state matrix and generating a quality score vector for each mode;

[0053] In this embodiment, the modal state matrix is defined as a structured cache with modes as rows and quality-related measures and state markers as columns, and the state markers include to-be-sampled, sampling, up-to-standard, not up-to-standard, to-be-supplemented and prohibited; the quality score vector is composed of (video score, audio score, biological feature score) in fixed order, the update period is 200ms by default, the adjustable range is 100-500ms, the window length is 1s by default, the adjustable range is 0.5-3s, and the threshold adopts an up-down hysteresis band (5% full scale by default) to suppress jitter. Optionally, the state matrix and the score vector adopt double buffering submission to avoid read-write competition; when cross-cycle frame loss occurs, only the time baseline is advanced, and no history is backfilled, so as to maintain the consistency of time sequence.

[0054] Step 103, output the modality collection sequence instruction by the reinforcement learning decision engine based on the quality score vector and the device state;

[0055] Step 104, trigger the collection module of the corresponding modality according to the collection sequence;

[0056] In this embodiment, the collection trigger contains three steps: module readiness verification, buffer reservation, and timeout control. The trigger timeout is 1.5 seconds by default (adjustable 0.5-5 seconds), and the buffer depth is 3 frames by default (adjustable 1-10 frames) or equivalent to 1s audio cache. For example, if the previous modality is still in the data landing stage and occupies more than the bus threshold, the current trigger is delayed to the next decision cycle and records a resource waiting event. Optionally, weak parallelism between audio and video can be enabled (such as only pre-sampling without landing) to shorten the first packet delay, but resource arbitration constraints must be met.

[0057] Step 105, when the quality of any modality does not meet the preset threshold, mark the modality as to be resampled and add it to the resampling queue;

[0058] In this embodiment, the quality threshold is determined by the modality score and the key indicator double threshold parallel mechanism, with a default threshold of modality score below 0.6 and at least one key indicator out of range. The threshold can be adjusted in the range of 0.4-0.8, set according to offline ROC analysis and on-site false rejection rate target. Further, the to-be-resampled item records the latest failure reason and retry cooling time, with a default cooling time of 5-10 seconds to avoid repeated failures in a short period of time. Optionally, for failure items caused by pollution or overexposure, a one-time guided prompt is added and automatically retried when the next environment stable condition is met. If necessary, when the resampling queue is congested, the modality related to identity verification is prioritized.

[0059] Step 106, automatically trigger the resampling of the to-be-resampled modality under the resampling conditions of device idle and environment stability until the threshold is met;

[0060] Further, device idle is defined as processor load below 70%, available memory above 20%, and bus occupancy below 60%, and environment stability is defined as illumination change, noise level, or contact pressure change within 1-2 seconds being below their respective thresholds (default 10% full scale). The above thresholds are derived from field stability backtesting and can be configured according to the scene. Optionally, if there are still multiple to-be-resampled items in the resampling window, they are prioritized according to the score potential and dependency relationship; if necessary, items that exceed the maximum retry number (default 2 times) are automatically downgraded and given a manual intervention prompt.

[0061] In one embodiment, the quality score vector is obtained by normalizing and weighting measurable quality indicators of each modality, wherein the quality indicators of the video stream at least include the key region occlusion ratio and the motion blurring degree, the quality indicators of the audio stream at least include the signal-to-noise ratio and the effective speech segment proportion, and the quality indicators of the biometric sensor data at least include the contact surface pollution coverage or the environmental interference intensity;

[0062] In this embodiment, the key region occlusion ratio and the motion blurring degree of the video side are respectively characterized by the dimensionless results of the face key point visibility and the clarity operator response, the signal-to-noise ratio and the effective speech segment proportion of the audio side are respectively characterized by the decibel domain estimation and the speech activity detection proportion, and the contact surface pollution coverage or the environmental interference intensity of the biometric side is quantified by the sensor quality output or the ridge contrast; the upper and lower limits and the direction marks of each indicator are derived from offline calibration and on-site playback, and by default, equal weight aggregation is adopted, the adjustable weight range is 0-1, and the weight is normalized within the modality. For example, the occlusion ratio and the pollution coverage are processed in the [0, 1] mapping interval with small as optimal direction; when any indicator is missing, the weight is set to zero and the remaining weight is normalized. Alternatively, if a single-value quality index is provided by the device manufacturer and verified to be equivalent, it can be directly used as the input of the modality score. If necessary, when the original observation is out of range or has a spike, the amplitude limiting strategy is preferred and the out-of-range count is recorded for subsequent abnormality reporting.

[0063] The scale and weight of the quality score can be obtained by configuration delivery or offline calibration;

[0064] The normalization and weight aggregation steps of the quality score include: reading the original quality indicators of the video, audio and biometric sensor in the same collection period, making the direction consistent and the interval normalized for each indicator, aggregating the single-value score within the modality by weight, assembling the scores of the three types of modalities into a quality score vector, and writing into the cache for subsequent use by the state machine and scheduling, missing items are processed with zero weight and the weight is re-normalized within the modality;

[0065] In any modality, the modality score is obtained by linear aggregation of the normalized scores:

[0066]

[0067] wherein s m represents the quality score of the modality m, with a value of 0-1, dimensionless, m represents the modality index, taking {video, audio, biometric}, j represents the index of the item, with a unit of item sequence, J m represents the number of indicators of the modality m, with a unit of item, w m,j represents the weight of the jth indicator in the modality m, taking 0-1, dimensionless, satisfying g m,j (·) represents the normalization mapping of the jth original indicator, outputting a dimensionless score of 0-1, x m,jdenotes the observation value of the jth original quality index, with the same unit as the index;

[0068] In the formula:

[0069]

[0070] wherein g m,j (x) denotes mapping the observation x to a dimensionless score, x denotes the observation of the index in the current period, with the same unit as x m,j , l m,j , and u m,j respectively denote the engineering lower limit and upper limit of the index, with the same unit as x, clip(·) denotes an operator for truncating the input to the interval [0, 1], d m,j denotes the direction mark, with a value set of two types of size relations, for processing differences such as a large signal-to-noise ratio being better and a small occlusion ratio being better;

[0071] Quality score vector assembly: the video modality score, the audio modality score, and the biological feature modality score are assembled into a three-dimensional column vector in a fixed order, which is used as the direct input for subsequent state determination and scheduling. For visualization needs, it can be linearly scaled to a percentage display without changing the internal determination scale.

[0072] When the observation is out of bounds, it is limited by clip. When a certain index is temporarily missing, its weight is set to zero, and the remaining weights are normalized proportionally. The weight and the engineering interval are written into the cache by configuration or offline calibration. During the stable stage of the environment, it is reused to reduce jitter.

[0073] Specifically, this step maps heterogeneous indexes to a unified scale, and then aggregates them within the modality with weights to form a compact and quantitative expression for scheduling and determination. Directional consistency is used to resolve differences in preferences for different indexes, making threshold and threshold design more intuitive. The engineering interval is provided by configuration and calibration, and combined with clipping to suppress occasional spikes and reduce beat disturbances caused by abnormal sampling. Linear aggregation has good readability, making it easy to adjust weights quickly in combination with historical playback or scenario backtesting. When missing items occur, zero weight and re-normalization are used to maintain continuous availability of scores. After the three modality scores are assembled into a vector, they can be directly input into the state machine and strategy network, and also facilitate distribution drift monitoring on the statistical side to capture slow changes in devices and environments.

[0074] In one embodiment, each modality corresponding to the modality state matrix is managed according to a preset state machine, and the states include at least standby, sampling, meeting the standard, not meeting the standard, standby for supplementary sampling, and prohibition. State transition is set according to quality score threshold, sampling time limit, and failure number limit, and a hysteresis strategy is used to suppress frequent switching.

[0075] Further, the minimum holding duration of a single acquisition is 1s by default (adjustable from 0.5s to 3s), the upper limit of consecutive failures of a single modality is 2 by default (adjustable from 1 to 5), and the upper limit of a single acquisition duration is 3s by default (adjustable from 1s to 10s); the recovery condition for entering the prohibited state is that the scores of two consecutive evaluation windows meet the standard and the resource occupation is lower than the idle threshold. Optionally, when the length of the backfill queue exceeds the upper limit (3 by default, adjustable from 1 to 5), the newly entered substandard modality replaces the tail item of the queue according to the first-in-first-out principle; similarly, when the items in the queue exceed the maximum waiting time (30s by default, adjustable from 10s to 120s) and still do not meet the backfill condition, they are automatically downgraded to prohibited and a policy alarm is recorded once.

[0076] In one embodiment, the input of the reinforcement learning decision engine includes a quality score vector, device state, and backfill queue information, and the output is the modality acquisition priority of the current period; the device state includes at least one of processor load, available memory, bus occupation, and sensor occupation; the decision goal considers both the minimization of total time consumption and the minimization of blocking rate, and sets a penalty for frequent switching to ensure timing stability;

[0077] In this embodiment, the decision period is 200ms by default (adjustable from 100ms to 500ms), the discount factor is typically taken as 0.9-0.99, the switching penalty coefficient, time consumption, and blocking item weight are obtained from offline playback fitting and can be fine-tuned online; the immediate reward is composed of quality benefit, time cost, queue pressure, and switching penalty, the evaluation network is updated in a timing difference approximation, and the policy network adjusts the priority distribution accordingly. Optionally, when the policy network does not complete sufficient convergence or the device is in a safe mode, it degenerates to a rule-based scheduling based on the score threshold and the minimum holding duration; if necessary, to prevent high-frequency jitter, set the minimum repetition interval of the same modality to 1s by default (adjustable from 0.5s to 3s).

[0078] The timing difference error and reward design of the reinforcement learning decision engine are as follows:

[0079] In each acquisition period t, first construct the state s t which is obtained by concatenating the quality score vector, device state, and backfill queue representation, calculate the immediate reward r t of the current period according to the action of the last period and the current state, r t contains time consumption, blocking, and switching penalty as well as quality benefit, then update the evaluation network based on the timing difference error, and adjust the acquisition priority distribution of the policy network accordingly, finally write the updated weights and thresholds to the local cache, and record the switching events for the calculation of the penalty term in the next period;

[0080] In value-based policy optimization, the evaluation error is defined in a timing difference:

[0081] δ t = rt + γV φ (s t+1 )- V φ (s t ),

[0082] where δ t represents the timing difference error of period t, dimensionless, r t represents the immediate reward of period t, scaled to dimensionless normalized score, γ represents the discount factor, taking value 0-1, dimensionless, V φ (·) represents the state value function characterized by parameter vector φ, outputting the estimation of long-term return for state s t+1 , represents the next state after action execution at period t, s t represents the current state at period t; state s t is defined as a three-part concatenation: quality score vector q t (three modality scores from last step), equipment state vector e t (normalized quantities of processor load, available memory, bus and sensor occupancy) and backfill queue vector b t (normalized quantities of pending sampling items, waiting time distribution, etc.);

[0083] In the formula, the revenue and cost are linearly composed as normalized items:

[0084]

[0085] where λ q , λ T , λ B , λ s represent the weight coefficients of quality revenue, time cost, blocking cost and switching penalty respectively, taking non-negative value, dimensionless, derived from configuration issuing or offline calibration, represents the modality score selected by action a t at period t, corresponding to the modality aggregation result of last step, taking value 0-1, dimensionless, a t represents the action at period t, engineering meaning as the modality index or priority decision of current period preferential sampling, represents the normalized quantity of sampling and scheduling time consumption at this period, taking value 0-1, obtained by comparing the measured seconds with the engineering defined calibration, represents the normalized quantity of blocking rate or queue depth at this period, taking value 0-1, represents the normalized quantity of switching penalty, taking value 0-1, which can be obtained by indicating quantity or sliding count according to whether the action of this period is the same as that of last period; in order to suppress frequent switching, the time window accumulation can be used when the continuous different modalities are switched;

[0086] ​State construction adopts the same cycle alignment and timestamp verification, rewards each item using and device-related upper limit to normalize the scale, so as to cross the environment reuse, when the backfill queue is not empty and the dependent mode has reached the standard, the strategy gives a higher probability to the corresponding action, if δ t The long-term is positive and the switching penalty ratio is high, and λ is preferentially increased s Or extend the switching window to stabilize the order; the evaluation network uses δ t As the regression target, the strategy network uses δ t As the advantage approximation to perform gradient update;

[0087] Specifically, the present step is carried out around the scheduling target of the strategy network, and the quality benefit and the resource cost are unified to a single immediate reward expression, and then the value evaluation and strategy update are driven by the time difference error; the state vector takes into account the sensing quality, device occupancy and backfill pressure, and can express the goal of "completing less congestion in less time while maintaining high quality" in the same framework; the reward term adopts normalized combination to avoid the dominant effect of different dimensions in gradient propagation, and the switching penalty reflects the demand for time stability by accumulating in the time window, so that the strategy makes a trade-off between high quality and low switching;

[0088] The time difference error as a single-step approximation long-term improvement signal on the one hand promotes the evaluation network to form a more accurate estimate of future returns, and on the other hand provides a directional advantage for the strategy, so as to gradually converge to a more suitable acquisition order for the current device and environmental conditions; since the quality score and device state are updated at the cycle level, the sensitivity of the reward can be quickly tuned by the weight coefficient and the normalized upper limit to adapt to scene changes such as outdoor strong light, contact surface pollution and noise background;

[0089] In one embodiment, the backfill condition triggered by resampling includes that the dependent mode of the target mode has reached the standard, the device is within the idle threshold, and the environmental parameters remain stable within the continuous time window; the confirmation signal is automatic confirmation or manual confirmation, and when the timeout is not confirmed, the process is automatically continued according to the degradation strategy;

[0090] In one embodiment, the method further includes a resource arbitration step: according to the resource budget limit and the concurrency and buffer depth of the device, a plurality of modes available at the same time are queued and quota allocated, so that the acquisition process runs within the memory and bandwidth threshold, and when the resources are tight, the modes with higher quality scores or key to the overall process are preferentially guaranteed;

[0091] Specifically, the concurrency degree defaults to no more than one heavy load mode and one light load mode running simultaneously, the heavy load refers to continuous acquisition of video / iris, and the light load refers to audio monitoring or short instruction sampling; the buffer quota is allocated with priority weighting, the video ring buffer defaults to 3-5 frames, and the audio ring buffer defaults to 1s equivalent capacity; for example, when the predicted peak memory is about to exceed the budget, the pre-sampling of the low-priority mode is suspended first, and the video buffer is reduced to the minimum value. Optionally, a soft limit + hard limit double threshold is enabled, triggering the soft limit only adjusts the quota, and triggering the hard limit immediately suspends new start and cleans up the cache. If necessary, when a resource shortage exception occurs, record the resource snapshot for offline parameter tuning.

[0092] In one embodiment, the method further comprises cross-modal acquisition assistance: when the video key area is blocked or overexposed while the audio quality meets the standard, generate a face key point heat map based on the pre-trained cross-modal mapping, which is used to guide the framing frame adjustment or area relocation, and can participate in the identification result fusion after passing the liveness detection and consistency check;

[0093] Further, the cross-modal mapping takes the heat map output by the pre-trained model as the guide signal, the heat map spatial resolution defaults to 64x64 (adjustable 32x128), and the update period defaults to 200ms, which is aligned with the video decision period; for example, the area where the heat peak value is located is used to drive the fine-tuning of the acquisition frame until the key area score rebounds above the compensation threshold; optionally, when the liveness or consistency check fails, it is only used for position guidance and does not participate in fusion, thereby ensuring the safety boundary of the identification link. If necessary, when there is no improvement in two consecutive guidances, automatically turn off the cross-modal assistance until the environment is stable.

[0094] In one embodiment, the training of the strategy network includes building a modal state transition model covering multiple environments and human-computer interaction situations, and using a combination of offline data training and online fine-tuning to optimize the reward function of the acquisition path, so as to improve the scheduling robustness in outdoor strong light, contact surface pollution and noise background;

[0095] Specifically, the offline training samples default to cover at least 100 subjects and no less than 50 hours of field segments, and the scenes include typical conditions such as strong light, backlight, low light, wind noise and oil pollution; online fine-tuning is performed with small learning steps and safety constraints, the minimum interval of strategy update defaults to 30s (adjustable 10-120s), and a trust domain is set to limit the single threshold adjustment amplitude to no more than 5% of the full range. Optionally, the offline strategy is frozen when the minimum sample trigger quantity is not reached in the online stage; if necessary, after the alarm of continuous strategy degradation occurs, roll back to the latest stable version and mark the current environment as needing manual review.

[0096] The embodiment also provides a computer device suitable for the case of the audio and video multi-modal recognition method based on artificial intelligence, and the computer device comprises a memory and a processor.

[0097] The computer device can be a terminal, and the computer device comprises a processor, a memory, a communication interface, a display screen and an input device which are connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, an operator network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, a trackball or a touchpad arranged on the shell of the computer device, or an external keyboard, a touchpad or a mouse.

[0098] The embodiment also provides a storage medium having a computer program stored thereon, and the program is executed by a processor to implement the audio and video multi-modal recognition method based on artificial intelligence. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0099] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, rather than limit the present application; even though the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some or all of the technical features thereof can be replaced equivalently; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0100] Furthermore, those skilled in the art could understand that, although some embodiments herein include certain features rather than other features included in other embodiments, the combination of features of different embodiments means to be within the scope of the present application and form different embodiments. For example, all the above embodiments can be used in any combination. The information disclosed in this part of background is merely intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known to those skilled in the art.

Claims

1. An artificial intelligence-based audio-video multi-modal recognition method, characterized in that, The method comprises the following steps: acquiring multi-modal input including video stream, audio stream and biological feature sensor data; constructing a modal state matrix and generating a quality score vector for each modal; outputting a modal acquisition sequence instruction by a reinforcement learning decision engine based on the quality score vector and device state; triggering the acquisition module of the corresponding modal according to the acquisition sequence; when the quality of any modal does not reach a preset threshold, marking the modal as to-be-reacquired and adding it to a reacquisition queue; automatically triggering the reacquisition of the to-be-reacquired modal under the reacquisition conditions of device idleness and environmental stability until the threshold is reached; the input of the reinforcement learning decision engine includes the quality score vector, device state and reacquisition queue information, and the output is the modal acquisition priority of the current period; the device state includes at least one of processor load, available memory, bus occupancy and sensor occupancy; the decision-making goal considers both the minimization of total time consumption and the minimization of blocking rate, and sets a penalty for frequent switching to ensure time sequence stability; the time difference error and reward of the reinforcement learning decision engine are designed as follows: At each collection cycle , a state is first constructed by concatenating the quality score vector, the device state and the backfill queue representation, the immediate reward of the current cycle is then calculated based on the action of the last cycle and the current state , which contains time-consuming, blocking and switching penalties as well as quality benefits, then the evaluation network is updated based on the time difference error, and the collection priority distribution of the policy network is adjusted accordingly, finally the updated weights and thresholds are written into the local cache, and the switching events are recorded for the calculation of the penalty term in the next cycle; in value-based policy optimization, the evaluation error is defined according to the time difference: ， wherein, denotes the period of the timing difference error, dimensionless, denotes the period of the immediate reward, scaled to dimensionless normalized scores, denotes the discount factor, taking values , dimensionless, denotes the state value function characterized by the parameter vector , outputting an estimate of the long-term return for a state, dimensionless, denotes the next state after action execution for the period , denotes the current state for the period ; the state is engineered as a three-part concatenation: a quality score vector , a device state vector and a backfill queue vector ; in the formula, the income and cost are linearly composed as a normalized item: ， in, These represent the weighting coefficients for quality gains, time costs, blocking costs, and handover penalties, respectively. They are non-negative, dimensionless, and derived from configuration distribution or offline calibration. Indicates period From action The selected modal score corresponds to the intramodal aggregation result from the previous step, and its value is... Dimensionless Indicates period The action, in engineering terms, refers to the modality index or priority decision to be collected first in the current cycle. This represents the normalized value indicating the time consumed by data collection and scheduling in this cycle, with values ​​ranging from [value missing]. It is obtained by comparing the measured number of seconds with the upper limit of the engineering standard. This represents the normalized value of the current cycle's congestion rate or queue depth, and its range is [value missing]. , This represents the normalized value of the switching penalty, and its range is... .

2. The artificial intelligence-based audio-video multi-modal recognition method of claim 1, wherein, the quality score vector is obtained by normalizing and weighting the measurable quality indicators of each modal, wherein the quality indicators of the video stream at least include the proportion of key area occlusion and the degree of motion blur, the quality indicators of the audio stream at least include the signal-to-noise ratio and the proportion of effective speech segment, and the quality indicators of the biological feature sensor data at least include the contact surface pollution coverage or environmental interference intensity; the scale and weight of the quality score can be obtained by configuration or offline calibration; the normalization and weight aggregation steps of the quality score include: reading the original quality indicators of the video, audio and biological feature sensor in the same acquisition period, making the direction consistent and interval normalized for each indicator, aggregating the weights in the modal to obtain a single value score, assembling the scores of the three types of modal into a quality score vector, writing into the cache for subsequent state machine and scheduling, processing the missing items with zero weight and re-normalizing the weight in the modal; in any modal, the normalized score is linearly aggregated to obtain the modal score: ， in, Representing modes Quality rating, with values Dimensionless Represents the modal index, take Video, audio, biometrics This represents the index of indicators, in terms of item order. Representing modes The number of indicators, in units of items. Representing modes The Middle The weight of each indicator is taken as follows: Dimensionless, satisfies Indicates the first Normalized mapping of the original indicators, output The dimensionless score, Indicates the first The observed values ​​of each original quality indicator, with the same units as the indicator; in the formula: ， wherein, denotes the mapping of the observation to a dimensionless score, denotes the observation of the indicator in the current period, which is in the same unit as , denotes the engineering lower and upper limits of the indicator, respectively, which are in the same unit as , denotes the operator that truncates the input to the interval , denotes the direction marker, which takes values from the set . quality score vector assembly: the video modal score, audio modal score and biological feature modal score are assembled into a three-dimensional column vector in a fixed order. 3.The audio-video multi-modal recognition method based on artificial intelligence of claim 1, wherein, each modal corresponding to the modal state matrix is managed according to a preset state machine, and the state at least includes to-be-acquired, being acquired, up to standard, not up to standard, to-be-reacquired and prohibited; state transition is set according to the quality score threshold, the upper limit of acquisition time and the upper limit of failure number, and a hysteresis strategy is used to suppress frequent switching. 4.The audio-video multi-modal recognition method based on artificial intelligence of claim 1, wherein, the reacquisition conditions of the reacquisition trigger include: the dependent modal of the target modal has reached the standard, the device is within the idle threshold, and the environmental parameters remain stable in the continuous time window; the confirmation signal is automatic confirmation or manual confirmation, and when the timeout is not confirmed, the process is automatically continued according to the degradation strategy. 5.The audio-video multi-modal recognition method based on artificial intelligence of claim 1, wherein, The method also includes a resource arbitration step: based on the device's resource budget limits on concurrency and buffer depth, multiple simultaneously available modalities are queued and quotas are allocated to ensure that the acquisition process runs within memory and bandwidth thresholds, and when resources are scarce, priority is given to modalities with higher quality scores or those critical to the overall process. 6.The audio-video multi-modal recognition method based on artificial intelligence of claim 1, wherein, The method also includes cross-modal acquisition assistance: when key areas of the video are occluded or overexposed but the audio quality meets the standards, a facial key point heatmap is generated based on pre-trained cross-modal mapping to guide viewfinder adjustment or region relocation, and can only participate in the fusion of recognition results after passing liveness detection and consistency verification.

7. The artificial intelligence-based audio-video multi-modal recognition method of claim 1, wherein, The training of the policy network includes constructing a modal state transition model covering various environments and human-computer interaction scenarios, and optimizing the reward function of the acquisition path by combining offline data training with online fine-tuning, so as to improve the scheduling robustness under outdoor strong light, surface contamination and noise backgrounds.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the multimodal biometric acquisition and recognition method based on artificial intelligence as described in any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that: When the computer program is executed by the processor, it implements the steps of the multimodal biometric acquisition and recognition method based on artificial intelligence as described in any one of claims 1 to 7.

Images