Machine vision-based large language model conversation management system

Abstract

This invention relates to the field of structured data technology and discloses a machine vision-based large language model conversation management system, including a perception module, a feature processing module, an intent evaluation module, a concurrent routing module, a state maintenance module, a conversation orchestration module, and a personnel database. The perception module collects audiovisual data; the feature processing module extracts image features to determine identity; the intent evaluation module calculates a comprehensive intent score to identify genuine interaction intentions; the concurrent routing module performs audiovisual consistency calculations, accurately allocating speech and text to corresponding data channels; the state maintenance module performs cross-modal comparison based on acoustic anchors when a target enters a visual blind spot; and the conversation orchestration module dynamically manages data partitions based on personnel perception events. This invention can quantify and mask invalid interactions to reduce computational power consumption, isolate concurrent audio to eliminate feature interference, and ensure conversation continuity under occlusion conditions through a cross-modal anti-shake mechanism.

Description

Technical Field

[0001] This invention relates to the field of structured data technology, specifically to a large language model conversation management system based on machine vision. Background Technology

[0002] The large language model conversation management system is the core control hub for natural language interaction between embodied intelligent hardware devices and users. The system is equipped with sensors such as microphone arrays and cameras to acquire sound and image data of the external physical environment. After front-end processing, the user's voice is converted into text and sent to the large language model for inference. The response content generated by the model is then fed back and output to complete the human-computer question-and-answer and multi-turn dialogue process.

[0003] During operation, the conventional system first collects multimodal sensor data, extracts features, and then identifies people in the surrounding area. The system mainly relies on preset voice wake words or fixed spatial detection distance to start the conversation process. When someone speaks in the surrounding area, the system converts the voice into text and assigns it to the corresponding processing channel to perform semantic understanding operations. During the dialogue, the system continuously tracks the facial features of people based on the vision module to maintain the online status of their identity, while also recording the dialogue context to support continuous multi-round language communication tasks.

[0004] In open environments with high foot traffic, the current system lacks the ability to quantitatively identify the genuine interaction intentions of those around them. The system responds whenever someone enters a certain distance or makes a sound, leading to frequent false triggers from unintentional actions by passersby or those with their backs turned, consuming significant computing resources. Furthermore, in situations with multiple users speaking concurrently or in noisy environments, the system struggles to accurately distinguish the origins of mixed audio using only sound or simple visual localization. The lack of a deep fusion and matching mechanism for visual and auditory spatial features results in speech and text often being incorrectly assigned to other users' data channels, causing severe multi-user feature interference. Moreover, the system relies too heavily on continuous visual facial features to maintain user online status. If a user turns around, looks down, or is briefly obscured by someone and enters a blind spot, the system immediately terminates the current conversation. The lack of a cross-modal transition de-jitter mechanism from visual to auditory perspective leads to breaks in the context of the conversation, disrupting the coherent conversation experience.

[0005] Therefore, the purpose of this invention is to provide a large language model conversation management system based on machine vision to address the shortcomings of the prior art. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a large language model conversation management system based on machine vision, which solves the problems of frequent false triggers caused by the lack of interaction intention identification, chaotic multi-user voice allocation in concurrent speech environments, and contextual conversation interruption caused by the temporary loss of visual features.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A machine vision-based large language model conversation management system includes: The sensing module is used to output continuous video frame sequences, multi-channel audio streams, and sound source arrival direction data; The feature processing module is used to extract features from a sequence of consecutive video frames to determine the identity of the target user. The intent assessment module is used to calculate the overall score of interaction intent and generate a personnel perception update event containing identity identifiers when the overall score of interaction intent reaches the trigger threshold. The concurrent routing module is used to perform audiovisual consistency calculation by combining the sound source arrival direction data with the lip movement state parameters extracted from the continuous video frame sequence, and to allocate the speech text content obtained by parsing the multi-channel audio stream to the data channel of the identity identifier. The state maintenance module is used to perform cross-modal comparison and output maintenance instructions when the visual feature quality of the target user is lower than the feature maintenance standard. The conversation orchestration module is used to receive personnel perception update events, maintenance instructions and voice and text content, and to establish data partitions with the identity identifier as the primary key and store the voice and text content. The personnel database is used to store the identity identifiers of target users, as well as their corresponding historical characteristics and basic records.

[0008] This system integrates visual and auditory multimodal features, quantifies multi-target physical parameters in spatial dimensions, establishes audiovisual consistency identification logic in concurrent environments, and realizes continuous tracking of target identity, concurrent audio isolation, and dynamic allocation and management of session context state in mobile environments.

[0009] Preferably, the feature processing module includes an image evaluation unit and an identity matching unit; the image evaluation unit performs face detection on a continuous video frame sequence to calculate sharpness parameters and occlusion parameters, removes video frames that do not meet quality standards and retains qualified image frames; the identity matching unit uses a lightweight convolutional neural network to extract the face feature vectors of qualified image frames, performs similarity retrieval with the face feature vectors and the historical feature records stored in the personnel database to obtain the maximum similarity score, and associates the target user with the identity identifier of the historical feature record when the maximum similarity score is greater than the identity confirmation threshold.

[0010] Preferably, the intent assessment module extracts depth values ​​from the 3D spatial depth information contained in the continuous video frame sequence to calculate the actual spatial distance between the target user and the front-end sensor array. It uses a gaze estimation model to extract the pupil feature distribution of the eye region in the continuous video frame sequence to calculate the confidence level of the gaze towards the robot. Based on facial key points, it calculates the 3D Euler angles of the head to calculate the probability score of a positive head pose. The intent assessment module combines the confidence level of the gaze towards the robot, the actual spatial distance, and the probability score of a positive head pose with system weight coefficients and a system-preset maximum effective interaction distance threshold to perform calculations to obtain a comprehensive interaction intent score. When calculating the comprehensive interaction intent score, the system extracts the ratio of the actual spatial distance to the maximum effective interaction distance threshold to measure spatial attenuation, and introduces a preset system weight coefficient to comprehensively calculate the weighted result of the gaze confidence level, the distance attenuation ratio, and the head pose probability component, quantifying and outputting the comprehensive interaction intent score.

[0011] Preferably, the concurrent routing module extracts the rate of change of the distance between the upper and lower lips of the mouth contour from a continuous video frame sequence as the confidence value of lip movement activity as a lip movement state parameter. It then uses internal and external parameter matrices to convert the pixel coordinates of the target user in the two-dimensional color image contained in the continuous video frame sequence into spatial relative orientation angles. The concurrent routing module combines the audiovisual feature fusion weight coefficients, the lip movement activity confidence value, the horizontal deviation angle of the sound source arrival direction angle, the horizontal deviation angle of the spatial relative orientation angle, and the maximum spatial orientation deviation angle to perform calculations to obtain an audiovisual consistency score. When calculating the audiovisual consistency score, the system allocates the dominant weight of visual and auditory perception through the audiovisual feature fusion weight coefficients, calculates the weighted component of the lip movement activity confidence value, extracts the absolute value of the difference between the horizontal deviation angle of the sound source arrival direction angle and the horizontal deviation angle of the spatial relative orientation angle, calculates the proportion of this absolute value to the maximum spatial orientation deviation angle to obtain the spatial orientation deviation degradation amount, and fuses the lip movement weighted component with the supplementary weighted component of the deviation degradation amount to obtain the audiovisual consistency score. The concurrent routing module compares the audiovisual consistency scores of all target users and selects the target user with the highest audiovisual consistency score. When the audiovisual consistency score of the target user with the highest audiovisual consistency score is greater than or equal to the fault-tolerant matching threshold, the speech recognition component is called to convert the multi-channel audio stream into speech-text content and allocate it to the data channel of the corresponding identity identifier.

[0012] Preferably, after obtaining the multi-channel audio stream corresponding to the identity identifier of the active state, the state maintenance module extracts the voiceprint feature vector from the multi-channel audio stream and combines the voiceprint feature vector with the sound source arrival direction data to generate an acoustic anchor point. The state maintenance module caches the acoustic anchor point along with the identity identifier in the system memory. When the number of consecutive lost frames of the target user's face in a continuous video frame sequence is greater than the number of frames to be lost or the facial posture deflection angle is greater than the limit deflection angle, the state maintenance module determines that the target user has entered the visual blind spot state.

[0013] Preferably, when the state maintenance module determines that the target user has entered the visual blind zone, it uses the acoustic anchor points cached in the system memory as reference objects, and combines the acoustic feature fusion weight coefficient, the current voiceprint feature vector extracted from the currently acquired multi-channel audio stream, the cached reference voiceprint feature vector, the horizontal deflection angle of the sound source arrival direction angle of the currently acquired multi-channel audio stream, and the horizontal deflection angle of the cached reference sound source arrival direction angle with the blind zone tolerance direction deviation angle set by the system to perform calculations to obtain the cross-modal matching score. When calculating the cross-modal matching score, the system calculates the normalized inner product of the current voiceprint feature vector and the reference voiceprint feature vector as the voiceprint similarity. It introduces acoustic feature fusion weight coefficients to obtain the voiceprint similarity weighted component. It extracts the absolute value of the difference between the horizontal deviation angle of the current sound source arrival direction angle and the horizontal deviation angle of the reference sound source arrival direction angle, calculates the proportion of this absolute value to the blind zone tolerance direction deviation angle to obtain the sound source spatial drift degradation amount, and fuses the voiceprint similarity weighted component with the supplementary weighted component of the spatial drift degradation amount to obtain the cross-modal matching score. When the cross-modal matching score is greater than or equal to the maintenance confirmation threshold, it sends a maintenance command to the session orchestration module.

[0014] Preferably, after receiving a user's perception update event, the conversation orchestration module establishes a mapping index using the identity identifier as a unique primary key. If there is no data partition corresponding to the unique primary key in the system's logical memory, a new data partition is dynamically allocated for the unique primary key, and system prompts and the initial task state tree are loaded into the new data partition. If there is a data partition corresponding to the unique primary key in the system's logical memory that is in an active state, the historical dialogue records, task execution progress, and user preference information contained in the active data partition are extracted to restore the target user's context.

[0015] Preferably, the conversation orchestration module appends the voice text content to the data partition corresponding to the unique primary key. When the ratio of the accumulated text length in the data partition to the maximum capacity of the context window reaches the capacity warning threshold, the text summarization algorithm is invoked to compress the historical dialogue records. The conversation orchestration module concatenates the system prompt words, historical dialogue records, and voice text content in a temporal sequence into prompt word text, and submits it to the large language model inference engine through read-write locks and thread isolation mechanisms.

[0016] Preferably, the personnel database receives and stores historical feature records and basic records. The personnel database uses a moving average algorithm to fuse and update the facial feature vectors extracted by the received feature processing module with the historical facial feature vectors contained in the historical feature records. The personnel database subtracts the current system timestamp from the most recent timestamp to obtain a time difference. This time difference, combined with the cumulative interaction rounds and a time decay weight coefficient and an activity weight coefficient, is used to calculate a data elimination priority score. When calculating the data elimination priority score, the system combines the time decay weight coefficient and the activity weight coefficient to calculate the difference between a positive weighted index of the time difference and a negative weighted offset index of the cumulative interaction rounds to obtain the score. When the total number of basic records stored in the personnel database reaches the maximum record capacity threshold, the personnel database sorts the basic records in descending order according to the data elimination priority score and deletes the high-priority basic records at the top to release physical storage space.

[0017] This invention provides a large language model conversation management system based on machine vision. It has the following beneficial effects: 1. This invention, by setting up an intent evaluation module, extracts the target user's actual spatial distance, confidence in the direction of their gaze toward the robot, and probability scores of a positive head posture. It then uses system weight coefficients to calculate a comprehensive score of interaction intent and generates a human perception update event when the comprehensive score reaches a trigger threshold. This allows for the quantitative identification of the true interaction intentions of people in the physical environment, blocking passersby or people with their backs turned without interaction intent, preventing false system triggers caused by invalid interactions, and reducing the ineffective use of computing resources.

[0018] 2. This invention sets up a concurrent routing module, combining the sound source arrival direction data output by the auditory acquisition unit with the lip movement activity confidence value and spatial relative direction angle extracted by the visual acquisition unit, to perform audiovisual consistency calculation to obtain an audiovisual consistency score. In environments with multiple users moving or speaking concurrently, the spatial consistency features of vision and hearing are used to isolate and match mixed audio, ensuring that the parsed speech text content can be accurately routed and allocated to the data channel of the corresponding identity identifier, thus eliminating feature interference during concurrent interaction.

[0019] 3. This invention establishes a cross-modal anti-shake mechanism by setting up a state maintenance module to dynamically cache acoustic anchor points generated by combining voiceprint feature vectors and sound source arrival direction data during normal interaction. When the target user's visual features are lost or the facial deflection angle is too large and enters the visual blind spot, cross-modal comparison is performed with the acoustic anchor points as reference objects. This enables the system to maintain the target user's online identity based on the continuity of acoustic features when the target user encounters brief physical occlusion or turns their back, thus ensuring the continuity of the context conversation. Attached Figure Description

[0020] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention; Figure 3 This is a bar chart showing the comprehensive feature score of the target user in a multi-concurrency environment according to the present invention. Figure 4 This is a line graph showing the correlation between cross-modal matching scores and online session status during visual occlusion, as presented in this invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 This invention provides a machine vision-based large language model conversation management system, which includes: a perception module, a feature processing module, an intent evaluation module, a concurrent routing module, a state maintenance module, a conversation orchestration module, and a personnel database.

[0023] The system is installed in an embodied smart hardware device, which is equipped with a computing platform that provides computing power support. The computing platform adopts an edge computing platform or a terminal computing platform. The perception module is connected to the front-end sensor array of the embodied smart hardware device through a physical interface. The front-end sensor array includes a camera, a depth camera, and a microphone array.

[0024] The perception module is used to collect sensor data of the physical environment and transmit the sensor data to the corresponding data processing link. The perception module includes a visual acquisition unit and an auditory acquisition unit. The visual acquisition unit acquires continuous video frames from the camera, and the auditory acquisition unit acquires audio streams from the microphone array and performs sound source localization calculations to output sound source arrival direction data.

[0025] The feature processing module receives video frame data from the visual acquisition unit. The feature processing module includes an image evaluation unit and an identity matching unit. The image evaluation unit performs face detection and image quality calculation on the video frames and removes image frames that do not meet the quality standards according to the set sharpness parameters and occlusion parameters. The identity matching unit extracts the face feature vectors from the qualified image frames and performs a similarity search between the face feature vectors and the historical feature records stored in the personnel database.

[0026] Based on the search results, the identity matching unit determines the identity of the target user in the current field of view. For target users who are successfully matched, the system associates the existing identity with the target user; for target users who are not matched, the system generates a new identity and writes the corresponding facial feature vector into the personnel database.

[0027] The intent assessment module calculates a comprehensive intent score based on the user's relative distance, gaze direction, and head posture information provided by the visual acquisition unit. When the comprehensive intent score reaches the trigger threshold, the intent assessment module generates a personnel perception update event containing the current identity identifier and outputs the personnel perception update event to the downstream processing link.

[0028] The concurrent routing module receives the sound source arrival direction data output by the auditory acquisition unit and, in conjunction with the lip movement state parameters of the target user provided by the visual acquisition unit, performs audiovisual consistency calculation. Based on the result of the audiovisual consistency calculation, the concurrent routing module allocates the parsed speech text content to the data channel corresponding to the identity identifier.

[0029] The state maintenance module caches the sound source arrival direction data and voiceprint features associated with active identity identifiers. When the image evaluation unit determines that the visual feature quality of the target user is lower than the feature maintenance standard, the state maintenance module compares the input audio with the cached acoustic features as a reference object, and maintains the online status of the identity identifier within a set time limit based on the comparison results.

[0030] The session orchestration module receives text data allocated by the concurrent routing module and online status information of the identity output by the status maintenance module. The session orchestration module uses the identity identifier as the primary key to dynamically create or restore the corresponding data partition in the system memory. The session orchestration module writes the text data corresponding to different identity identifiers into the corresponding data partitions. The personnel database records the lifecycle attributes associated with each identity identifier. The session orchestration module reads the lifecycle attributes in the personnel database and performs memory cleanup or persistent storage operations on data partitions that have exceeded the lifecycle attribute limit.

[0031] See attached document Figure 2 This invention includes a machine vision-based large language model conversation management method and a machine vision-based large language model conversation management system, comprising the following steps: S10: Acquire sensor data of the physical environment, including video frame data and audio stream data; S20, perform image quality assessment on video frame data and extract facial feature vectors from qualified image frames; S30, perform a similarity search between the facial feature vector and historical feature records in the personnel database to determine the identity of the target user; S40 calculates a comprehensive intent score based on the target user's relative distance, gaze direction, and head posture information, and generates a personnel perception update event containing identity information when the comprehensive intent score reaches the trigger threshold. S50 combines the sound source arrival direction data of the audio stream data with the lip movement state parameters of the target user to perform audio-visual consistency calculation, and allocates the parsed speech text content to the data channel of the corresponding identity identifier. S60, when the visual feature quality of the target user is lower than the feature maintenance standard, the input audio is compared with the acoustic features in the audio stream data to maintain the online status of the target user; S70 uses the identity identifier as the primary key to dynamically create or restore the corresponding data partition in the system memory, and writes the voice and text content into the corresponding data partition for extraction by the large language model.

[0032] The sensing module is deployed in the embodied smart hardware device and is physically connected to the front-end sensor array to obtain raw data of the external physical environment. The data acquisition and processing process of the sensing module includes the following steps: S101, the visual acquisition unit acquires two-dimensional color images and three-dimensional spatial depth information of the physical environment through a front-end sensor array. The camera in the front-end sensor array uses an RGB sensor to output two-dimensional color images, and the depth camera in the front-end sensor array uses a time-of-flight sensor or a structured light sensor to output three-dimensional spatial depth information. The visual acquisition unit timestamps the two-dimensional color images and three-dimensional spatial depth information according to the hardware system clock, aligns and integrates image data and depth data with the same timestamp, and generates a continuous video frame sequence. The continuous video frame sequence is output at a fixed frame rate, providing underlying data support for face detection and relative distance calculation in the downstream link.

[0033] S102, the auditory acquisition unit acquires multi-channel audio streams through the microphone array in the front-end sensor array and performs audio signal preprocessing operations on the multi-channel audio streams. The microphone array is composed of multiple microphone units arranged in a linear array or a ring, which simultaneously capture sound signals from different spatial directions. The embodied intelligent hardware device is in an open space during operation. In order to reduce the interference of non-interactive ambient sound on the system, the auditory acquisition unit performs preprocessing before calculating the direction.

[0034] The audio signal preprocessing operation is specifically implemented as follows: calling the acoustic echo cancellation algorithm to filter the sound frequency band emitted by the speaker of the intelligent hardware device itself, using the steady-state background white noise in the environment to filter out the steady-state background white noise in the environment, and outputting the preprocessed multi-channel audio stream.

[0035] S103, the auditory acquisition unit calculates and outputs sound source arrival direction data based on the preprocessed multi-channel audio stream. This data includes the horizontal and vertical angles of the sound source relative to the physical geometric center of the microphone array, indicating the spatial orientation of the sound-emitting entity in the physical environment. For specific implementation methods of calculating the multi-channel audio stream sound source arrival direction data, those skilled in the art can use a generalized cross-correlation phase transform algorithm combined with a time difference localization model, or a multiple signal classification algorithm to obtain spatial spectrum estimates. The auditory acquisition unit then transmits the generated sound source arrival direction data along a time sequence to the concurrent routing module and the state maintenance module for processing.

[0036] The feature processing module receives a continuous sequence of video frames from the visual acquisition unit. The data processing process of the feature processing module includes the following steps: S201, the image evaluation unit performs face detection on video frames in a continuous video frame sequence. The image evaluation unit obtains the face bounding box and face key points in the image. The image evaluation unit calculates the sharpness parameter based on the pixel information within the face bounding box and calculates the occlusion parameter based on the spatial distribution and visibility of the face key points. The image evaluation unit compares the sharpness parameter with a set sharpness threshold and the occlusion parameter with a set occlusion threshold. The image evaluation unit discards video frames with sharpness parameters lower than the sharpness threshold and video frames with occlusion parameters higher than the occlusion threshold, retaining qualified image frames that meet the quality standards.

[0037] The sharpness threshold and occlusion threshold are set by technicians during the system initialization phase through a configuration file based on the resolution of the front-end sensor array and the lighting conditions of the application scenario. The sharpness threshold is set to a value range of [0.6, 0.8], and the occlusion threshold is set to a value range of [0.2, 0.4].

[0038] S202, the identity matching unit receives a qualified image frame, extracts pixel region data within the face bounding box in the qualified image frame, inputs the pixel region data into the feature extraction model, and outputs the corresponding face feature vector. The specific implementation of the feature extraction model can be achieved by those skilled in the art using a lightweight convolutional neural network or a visual transformer based on an attention mechanism. Extracting feature data from face image regions using a deep learning model is a well-known technology in the field and will not be described in detail here.

[0039] S203, the identity matching unit obtains historical feature records stored in the personnel database. The personnel database stores the identity identifiers and corresponding facial feature vectors of registered users. The identity matching unit performs similarity retrieval calculations on the extracted facial feature vectors and the historical feature records in the personnel database one by one. The similarity retrieval calculation formula is as follows: ; in, The similarity score is calculated between the current facial features and known features in the database. This is the facial feature vector extracted from the current image frame; The corresponding number in the personnel database is Facial feature vectors of personnel; Let be the norm of the facial feature vector extracted from the current image frame; The corresponding number in the personnel database is The norm of the facial feature vector of the personnel; This is the inner product operator between eigenvectors.

[0040] S204, the identity matching unit will compare the calculated maximum similarity score with the set identity confirmation threshold. The value range of the identity confirmation threshold is set to [0.75, 0.90], and the specific value is adjusted according to the system's tolerance for false recognition rate.

[0041] When the maximum similarity score is greater than the identity verification threshold, the identity matching unit determines that the target user is successfully matched. The identity matching unit associates the target user with the identity identifier of the historical feature record corresponding to the maximum similarity score. When the similarity scores calculated from all historical feature records are less than or equal to the identity verification threshold, or when the personnel database is in an initial empty state, the identity matching unit determines that the target user is not matched. The identity matching unit generates a new identity identifier and writes the new identity identifier and the extracted facial feature vector as a new record into the personnel database.

[0042] The intent assessment module is used to quantify the degree of willingness of target users to interact with the embodied smart hardware device, and to prevent false system triggering caused by invalid interactions. The data processing of the intent assessment module includes the following steps: S301, the intent assessment module acquires multi-dimensional physical characteristics of the target user based on data provided by the visual acquisition unit. These multi-dimensional physical characteristics include the actual spatial distance between the target user and the front-end sensor array, the confidence level of the gaze direction towards the robot output by the visual acquisition unit, and the probability score of the head pose being positive output by the visual acquisition unit. The intent assessment module extracts the depth value of the corresponding pixel region of the face target in the three-dimensional spatial depth information to calculate the actual spatial distance between the target user and the front-end sensor array.

[0043] The intent assessment module uses a gaze estimation model to extract the pupil feature distribution in the eye region to calculate the confidence level of the gaze direction of the visual acquisition unit outputting the robot, and normalizes the confidence level of the gaze direction of the visual acquisition unit outputting the robot to the [0,1] interval. The intent assessment module calculates the three-dimensional Euler angles of the head based on facial key points, calculates the probability score of the positive head pose output by the visual acquisition unit based on the three-dimensional Euler angle mapping of the head, and normalizes the probability score of the positive head pose output by the visual acquisition unit to the [0,1] interval. For specific implementation methods of calculating spatial distance, gaze direction and head pose using visual images, those skilled in the art can use a spatial coordinate system transformation algorithm based on a depth camera and a convolutional neural network pose estimation model to implement it.

[0044] S302, the intent assessment module fuses multi-dimensional physical features based on set weighting coefficients to calculate a quantified comprehensive intent score. The formula for calculating the comprehensive intent score is as follows: ; in, The overall score is based on the interaction intent; , , These are the preset system weight coefficients; The confidence level of the gaze direction of the visual acquisition unit towards the robot; The actual spatial distance between the target user and the front-end sensor array; The maximum effective interaction distance threshold preset by the system; The probability score for the head pose output by the visual acquisition unit to be positive.

[0045] Preset system weight coefficients , , The sum is 1. The specific value of the system weight coefficient is configured by technical personnel during the system deployment phase according to the focus of the application scenario. In near-field interaction scenarios that emphasize eye contact, The value range is set to [0.4, 0.6]. The value range is set to [0.2, 0.3]. The value range is set to [0.2, 0.3], and the maximum effective interaction distance threshold is set to [2.0m, 5.0m] based on the effective detection range of the front-end sensor array.

[0046] S303, the intent assessment module compares the calculated comprehensive intent score with the set trigger threshold. When the comprehensive intent score is less than the trigger threshold, the intent assessment module determines that the target user is a passerby or a person with their back turned who has no interactive intent. The intent assessment module discards the current calculation record and does not trigger the downstream session management process. When the comprehensive intent score is greater than or equal to the trigger threshold, the intent assessment module determines that the target user has a clear interactive intent. The range of the trigger threshold is set to [0.60, 0.80]. The specific value of the trigger threshold is adapted and adjusted according to the congestion level of the operating environment of the smart hardware device.

[0047] S304, when the comprehensive intent score reaches the trigger threshold, the intent assessment module generates a personnel perception update event. The personnel perception update event is a structured data packet used to drive the downstream processing link. The personnel perception update event contains the target user's identity identifier, timestamp, comprehensive intent score, and interaction readiness status flag determined by the feature processing module. The intent assessment module outputs the personnel perception update event to the session orchestration module. The personnel perception update event indicates that the target user is in a valid interaction readiness state and notifies the session orchestration module to perform session establishment or historical session recovery operations for the target user.

[0048] The concurrent routing module receives the sound source arrival direction data output by the auditory acquisition unit, and combines it with the lip movement state parameters of the target user provided by the visual acquisition unit to perform audiovisual consistency calculation. It then allocates the parsed speech-text content to the data channel corresponding to the user's identity. The data processing procedure of the concurrent routing module includes the following steps: S401, the concurrent routing module extracts the lip movement state parameters and spatial relative direction angles of each target user within a set time window in the field of view. The visual acquisition unit identifies facial contours based on facial key points and extracts the change rate of the distance between the upper and lower lips of the mouth contour as the confidence value of lip movement activity. The concurrent routing module obtains the sound source arrival direction angle of the current audio stream calculated by the auditory acquisition unit. The concurrent routing module uses the internal parameter matrix and external parameter matrix of the camera to convert the pixel coordinates of the target user in the two-dimensional color image into the corresponding spatial relative direction angle, so that the visual data and auditory data are unified in the same three-dimensional spatial coordinate system. For the specific implementation of obtaining the internal parameter matrix and external parameter matrix, those skilled in the art can use the Zhang Zhengyou calibration method to implement it.

[0049] S402, the concurrent routing module combines the sound source arrival direction data and lip movement state parameters to perform audiovisual consistency calculation, calculating the audiovisual consistency score between the current speech stream and each target user in the field of vision. The specific formula for calculating the audiovisual consistency score is as follows: ; in, The first in the current speech stream and field of view Audiovisual consistency score for each target user; For audiovisual feature fusion weighting coefficients; The first extracted by the visual acquisition unit Confidence value of lip movement activity for a target user; The horizontal deflection angle of the sound source arrival direction angle of the current audio stream calculated by the auditory acquisition unit; The first direct observation obtained by machine vision The horizontal deflection angle of the spatial relative orientation angle of the target user.

[0050] The weighting coefficients for audiovisual feature fusion are set to a range of [0.5, 0.7], with an emphasis on visual lip movement activity features. The maximum spatial orientation deviation angle is determined by the physical resolution of the microphone array and is set to a range of [10°, 30°]. If the absolute value difference... The calculated result is greater than the maximum spatial orientation deviation angle, so the concurrent routing module will... The item is set to 0.

[0051] S403, the concurrent routing module allocates the parsed voice text content to the data channel corresponding to the identity identifier based on the result of the audio-visual consistency calculation. The concurrent routing module compares the audio-visual consistency scores of all target users, selects the target user with the highest audio-visual consistency score, and compares the audio-visual consistency score of the target user with the set fault tolerance matching threshold. The value range of the fault tolerance matching threshold is set to [0.65, 0.85].

[0052] When the audio-visual consistency score is greater than or equal to the fault-tolerant matching threshold, the concurrent routing module determines that the current audio stream belongs to the target user with the highest audio-visual consistency score. The concurrent routing module calls the speech recognition component to convert the current audio stream into speech-text content and allocates the speech-text content to the data channel of the corresponding identity identifier. For target users whose audio-visual consistency score does not reach the fault-tolerant matching threshold, the concurrent routing module does not perform text writing operations to avoid data miswriting caused by the mixing of multiple user audio.

[0053] The above-described processing logic, which calculates the fusion score of visual lip movement state and auditory sound source location, strips out mixed audio, and precisely routes the parsed text to the data channel corresponding to the identity identifier, constitutes a specific subordinate implementation of the feature of eliminating concurrent interaction interference in a mobile environment using multimodal alignment as described in the claims. The specification establishes a clear supporting relationship with the claims through textual description.

[0054] The state maintenance module dynamically caches data associated with active identity identifiers during normal visual interactions, and provides a cross-modal stabilization mechanism based on the cached data when the target user experiences brief occlusion or turns away, resulting in loss of visual features. The data processing procedure of the state maintenance module includes the following steps: S501, during normal visual interaction, the state maintenance module acquires the audio stream data corresponding to the active identity identifier. The state maintenance module extracts the voiceprint feature vector from the audio stream data and combines it with the sound source arrival direction angle of the current audio stream calculated by the auditory acquisition unit to generate an acoustic anchor point. The state maintenance module caches the acoustic anchor point along with the identity identifier in system memory. The caching operation is executed on a rolling basis according to a set update cycle to ensure that the cached acoustic anchor point can accurately match the real-time changes in the target user's physical location and vocal characteristics. For the specific implementation of extracting the voiceprint feature vector from the audio stream data, those skilled in the art can use Mel-frequency cepstral coefficients or time-delay neural networks to extract deep voiceprint vectors.

[0055] S502, the state maintenance module continuously monitors the visual feature quality of the target user and determines whether the visual feature quality is lower than the feature maintenance standard set by the system. The feature maintenance standard is jointly defined by the tracking confidence output by the image evaluation unit and the number of consecutive lost frames. When the number of consecutive lost frames of the target user's face is greater than the set number of lost frames, or the facial pose deflection angle is greater than the set limit deflection angle, the state maintenance module determines that the visual feature quality of the target user is lower than the feature maintenance standard and identifies the target user as entering the visual blind zone state. The value range of the lost frame tolerance is set to [10 frames, 15 frames], and the value range of the limit deflection angle is set to [75°, 90°].

[0056] S503, when a visual feature is determined to be downgraded and the system enters a visual blind zone, the state maintenance module stops relying on visual features and instead uses the acoustic anchor points cached in the system memory as references to perform cross-modal comparisons on the audio stream continuously acquired by the microphone array. The state maintenance module calculates the cross-modal matching score. The specific formula for calculating the cross-modal matching score is as follows: ; in, For cross-modal matching scores; These are the preset acoustic feature fusion weighting coefficients; This is the current voiceprint feature vector extracted from the current audio stream; The baseline voiceprint feature vector cached by the state maintenance module; The norm of the current voiceprint feature vector; The norm of the baseline voiceprint feature vector; The horizontal deflection angle of the sound source arrival direction angle of the current audio stream calculated by the auditory acquisition unit; The horizontal deflection angle of the reference sound source arrival direction angle buffered for the state maintenance module; The blind zone tolerance directional deviation angle set for the system.

[0057] The acoustic feature fusion weighting coefficient is set to a range of [0.60, 0.80] to allocate the weight between spatial orientation stabilization and acoustic signature stabilization. The blind zone tolerance directional deviation angle is set to a range of [15°, 25°]. If the absolute value difference... The calculated result is greater than the blind zone tolerance direction deviation angle, and the state maintenance module will... The item is set to 0. When the calculated cross-modal matching score is greater than or equal to the set maintenance confirmation threshold, the state maintenance module determines that the current voice subject and the target user before visual degradation are the same physical entity. The state maintenance module generates a verification pass signal and sends a maintenance instruction to the session orchestration module to maintain the online active state of the corresponding identity in the global system state. The value range of the maintenance confirmation threshold is set to [0.70, 0.85].

[0058] S504, when the state maintenance module triggers cross-modal comparison, it synchronously starts the blind zone timer. The state maintenance module maintains the online status of the identity identifier within the set maximum blind zone tolerance time. If, within the maximum blind zone tolerance time, the visual acquisition unit recaptures the valid facial image of the target user and the identity matching unit verifies that it is consistent, the state maintenance module performs cross-modal re-verification, clears the blind zone timer, and restores the vision-based dominant interaction logic.

[0059] If the recording time of the blind zone timer exceeds the maximum blind zone tolerance time and the visual features are not recovered, the state maintenance module sends an offline signal to the session orchestration module to cut off the online status of the current identity and terminate the cross-modal anti-shake process to prevent the data channel from being abnormally occupied for a long time. The maximum blind zone tolerance time is set to [5s, 15s] according to the tolerance of the specific application scenario.

[0060] The conversation orchestration module is used to schedule the context resources of a large language model and perform memory isolation in multi-user flow interactions. The data processing of the conversation orchestration module includes the following steps: S601, the session orchestration module receives the personnel perception update event output by the intent evaluation module, the voice and text content allocated by the concurrent routing module, and the maintenance command or offline signal output by the state maintenance module. The session orchestration module extracts the identity identifier from the personnel perception update event and the voice and text content. The session orchestration module uses the identity identifier as a unique primary key in the system's logical memory and establishes a mapping index based on the unique primary key for subsequent context environment retrieval and task state mapping.

[0061] S602, the session orchestration module performs a search and comparison in the system logical memory based on the unique primary key. If no data partition corresponding to the unique primary key exists in the system logical memory, or if an existing data partition has timed out and is closed, the session orchestration module determines that the target user is either engaging for the first time or that previous interactions have expired. The session orchestration module then dynamically allocates a new data partition in the system logical memory for the unique primary key and loads the preset system prompts and initial task state tree into the new data partition.

[0062] When a data partition with a unique primary key exists in the system's logical memory and is in an active state, the session orchestration module extracts the target user's historical conversation records, task execution progress, and user preference information from the active data partition to restore the target user's context environment.

[0063] S603, the conversation orchestration module takes the received voice text content as the input of the latest interaction round and appends it to the data partition corresponding to the unique primary key. During the writing process, the conversation orchestration module monitors the ratio of the accumulated text length in the data partition to the maximum capacity of the context window set by the large language model. When the ratio of the text length to the maximum capacity of the context window reaches the set capacity warning threshold, the conversation orchestration module calls the text summarization algorithm to compress the historical dialogue records, retain user preference information and core task status, and release memory space to prevent data overflow. The value range of the capacity warning threshold is set to [0.80, 0.90]. For the specific implementation of the text summarization algorithm, those skilled in the art can use the summarization extraction technology based on the internal instructions of the large language model or the external extraction-based summarization model to implement it.

[0064] In S604, the conversation orchestration module sequentially concatenates the system prompts, historical dialogue records, and the latest voice text content within the data partition to construct a prompt text that conforms to the input specifications of the large language model. The prompt text is then submitted to the underlying large language model inference engine. Data partitions with different identities are allocated independent physical address segments in the system's logical memory, and there is no pointer intersection between them. The conversation orchestration module uses read-write locks and thread isolation mechanisms to ensure that, at the same timestamp, the large language model only reads and responds to the prompt text within the data partition of the current main interaction object.

[0065] The personnel database is used to store user identity information and performs data eviction and persistence management in conjunction with the session lifecycle. The data processing of the personnel database includes the following steps: S701, the personnel database receives the newly created identity identifier and corresponding facial feature vector output by the feature processing module, and writes the newly created identity identifier and facial feature vector as basic records into the database storage space. For identity identifiers already existing in the personnel database, the personnel database receives the facial feature vector of the latest qualified image frame transmitted by the feature processing module. The personnel database uses a moving average algorithm to merge and update the facial feature vector of the latest qualified image frame with the historical facial feature vectors stored in the personnel database according to the set feature update learning rate, ensuring that the facial feature vector adapts to the physical appearance changes of the target user's face over time. The value range of the feature update learning rate is set to [0.05, 0.20].

[0066] S702, the personnel database records and dynamically maintains the status attributes associated with each identity. These status attributes include the most recent timestamp, cumulative interaction rounds, and session lifecycle parameters. The session lifecycle parameter is set as the maximum duration from when an identity enters the interactive state to when it is allowed to remain in a dormant / suspended state. The value range of the session lifecycle parameter is set to [30min, 120min]. The personnel database subtracts the most recent timestamp from the current system timestamp to obtain the time difference, and then performs linear normalization on the time difference and the cumulative interaction rounds, mapping them to the [0,1] interval. The personnel database uses the normalized values ​​to calculate the data elimination priority score for each identity. The formula for calculating the data elimination priority score is: ; in, Score the priority of data elimination; This is the time decay weighting coefficient; This is the current system timestamp; The most recent timestamp corresponding to the identity identifier; It is the overall time difference obtained by subtracting the current system timestamp from the most recent timestamp and then performing linear normalization. This is the activity level weighting coefficient; This represents the cumulative number of interaction rounds within the current session period for the identity identifier after linear normalization.

[0067] The time decay weight coefficient is set to a value range of [0.5, 0.7], and the activity weight coefficient is set to a value range of [0.3, 0.5].

[0068] S703, the personnel database periodically sends the calculated data elimination priority score and session lifecycle parameters to the session orchestration module. The personnel database has a maximum record capacity threshold set internally. The maximum record capacity threshold is determined based on the physical memory space of the embedded smart hardware device, and the value range is set to an integer in the range of [100, 500]. When the total number of basic records stored in the personnel database reaches the maximum record capacity threshold, the personnel database sorts the basic records in descending order according to the data elimination priority score.

[0069] S704, the personnel database collaborative session orchestration module performs memory cleanup or persistent archiving operations. For data records where the difference between the current system timestamp and the most recently occurring timestamp exceeds the session lifecycle parameter, the personnel database determines that the target user's interaction has completely ended and notifies the session orchestration module to destroy the corresponding data partition in the system logical memory. The personnel database deletes the high-priority basic record at the head of the descending sort, releasing physical storage space. For target users who comply with privacy protection protocols and have long-term service needs, the personnel database packages the user preference information output by the session orchestration module with the basic record and transfers it to the cloud persistent storage server for archiving through an encrypted network channel.

[0070] Specific application examples: To verify the effectiveness of the machine vision-based large language model conversation management system proposed in this invention in solving problems such as concurrent interference from multiple sound sources, loss of context due to brief occlusion by people, and memory overflow in complex flowing environments, this embodiment is based on the application scenario of an embodied intelligent guide robot in a large science and technology exhibition hall, and combined with the attached... Figure 3 and attached Figure 4 The data shown will be explained in detail.

[0071] Appendix Figure 3 and attached Figure 4 The data in this document are all real-time capture data of various internal features of this system, compared with traditional systems that rely solely on single-modal triggering mechanisms.

[0072] In the application scenario of this embodiment, the embodied intelligent hardware device is deployed in an open exhibition area, where there are multiple mobile people and complex environmental noise. The system sets the maximum effective detection distance of the front-end sensor array to 5.0m. In response to the nonlinear characteristics of environmental interference and personnel displacement, the system implements a joint control scheme of intent assessment, anti-concurrent interference routing, and cross-modal anti-shake.

[0073] During the system's detection period, two people approached the device: target user A was directly in front of the device, and passerby B was to the side and talking to someone else.

[0074] The visual acquisition unit measures the actual spatial distance between target user A and the front-end sensor array. The line of sight is directed toward confidence level Positive probability score of the head Distance of Person B The line of sight is directed toward confidence level Positive probability score of the head .

[0075] From the appendix Figure 3 It can be seen that, Figure 3 The horizontal axis represents the IDs of different individuals within the field of view, and the vertical axis represents the scores for each quantitative calculation. The light gray bars represent the overall intent score, and the dark gray bars represent the audiovisual consistency score. The system allocates weights based on near-field interaction scenarios that emphasize eye contact. , , For target user A, the system invokes the comprehensive intent scoring formula. Calculation .

[0076] For passerby B, the calculation is as follows 0.

[0077] The target user A's score of 0.785 is greater than the set trigger threshold of 0.70, so a valid human perception update event is generated and a data partition is established in the session orchestration module. Since the score of user B is less than the trigger threshold, the system determines that he has no intention to interact and discards the calculation record.

[0078] During the concurrent routing phase, the auditory acquisition unit measures the horizontal deflection angle of the current audio stream sound source. The visual acquisition unit extracts the confidence value of lip movement activity of target user A. Spatial relative direction angle Confidence value of lip movement activity of person B Spatial relative direction angle .

[0079] System settings for audiovisual fusion weights Maximum spatial orientation deviation angle The system is based on the formula Calculate the audiovisual consistency score for target user A: Regarding person B, due to absolute deviation If the deviation angle is greater than the maximum spatial orientation deviation angle of 20°, the concurrent routing module sets the deviation angle correction term to 0, resulting in... .

[0080] The system detected If the accuracy reaches 0.82 and is greater than the fault-tolerant matching threshold of 0.75, the text content parsed from the audio segment is accurately assigned to the data channel of target user A, as shown in the attached figure. Figure 3 As shown, the height of the dark gray column of target user A is much greater than that of person B. This invention suppresses voice input errors caused by background conversations through audiovisual feature fusion. After target user A has a series of dialogues with the device, a venue staff member carrying materials passes between user A and the device, causing the interaction line of sight to be blocked.

[0081] When the facial area of ​​target user A is obscured, the number of consecutive lost frames recorded by the image evaluation unit for the target user climbs to 12 frames, exceeding the set frame loss tolerance (10 frames), and the system enters a state of visual degradation and visual blind spot.

[0082] From the appendix Figure 4 The details of the related line chart are visible. Figure 4 The horizontal axis represents the system runtime sequence (s), the main vertical axis represents the cross-modal matching score, and the secondary vertical axis represents the Boolean value of the session online status (0 indicates offline, 1 indicates online). In the traditional scheme that uses a single visual judgment, once the number of lost frames reaches the limit of 10 frames, the system's session online status (the dotted line in the figure) drops directly to 0, forcibly triggering the reset of the large language model context. In this embodiment, however, the cross-modal matching process is enabled.

[0083] The state maintenance module extracts the current input voiceprint feature vector. And compared with the baseline voiceprint feature vector cached in system memory Perform the calculation to obtain the cosine similarity. The system obtains the current horizontal deflection angle of the sound source. Horizontal deflection angle of the reference sound source Based on the preset feature fusion weights Blind spot tolerance directional deviation angle Substitute into the cross-modal matching score formula Calculation .

[0084] The system detected the calculation result. Greater than the maintenance confirmation threshold of 0.80, combined with the attached... Figure 4 As shown by the solid line, during the brief occlusion period of 3 seconds (from the 4th to the 7th second), the cross-modal matching score always fluctuated above 0.80. The system determined that the physical entities were consistent and sent a verification pass command to the session orchestration module, so that the session online status of the present invention (the solid line in the figure) remained stable at 1 during this period without interruption or offline. Subsequently, visual capture resumed and the blind spot timer was reset. This mechanism effectively ensured the continuity of the large language model memory resources and the task state tree.

[0085] After running for a long time, the system accumulates target user data from different time periods in the physical memory, posing a risk of overload.

[0086] The system simultaneously holds data records for target user A and historical users C from previous time periods, along with the current system timestamp. Converted to time series point 8000, the most recent occurrence timestamp of target user A At time point 7800, the linearly normalized time difference is 0.20. The cumulative number of interaction rounds within the current session period, after normalization, is... The most recent appearance timestamp of person C At time point 1000, the linearly normalized time difference is 0.95, indicating that only identity registration and simple command response are completed, and the cumulative interaction rounds are normalized to be... .

[0087] Retrieve the set time decay weight from the personnel database Activity weight According to the elimination priority scoring formula Perform the calculation.

[0088] Priority score of target user A Personnel C's priority score .

[0089] When the total number of basic records reaches the maximum record capacity threshold, the personnel database is sorted in descending order. This is based on the elimination priority score of personnel C. At its highest level, the system's collaborative session orchestration module prioritizes destroying the data partition of the corresponding user C in the system's logical memory and erasing the basic records to free up space. Since the target user A has a low data score, the system determines to keep him in an active state to avoid the long-term task of the large language model from crashing.

[0090] Summary of application examples: This embodiment demonstrates the feasibility of engineering implementation of a large language model conversation management system based on machine vision. On the one hand, it isolates non-interactive behaviors and concurrent speech noise in complex spaces by calculating the intent score and audiovisual consistency value of the joint data of two-dimensional images and depth sensing. On the other hand, it extracts cross-modal matching data based on pre-anchored acoustic orientation and voiceprint indicators, breaking through the stringent requirements of continuous facial recognition in a purely visual environment. Furthermore, it calculates the elimination priority before resources are fully loaded, accurately destroys idle data, and achieves efficient allocation of system hardware and software resources and the computing power of the large language model.

Claims

1. A large language model conversation management system based on machine vision, characterized in that, include: The sensing module is used to output continuous video frame sequences, multi-channel audio streams, and sound source arrival direction data; The feature processing module is used to extract features from the continuous video frame sequence to determine the identity of the target user; The intent assessment module is used to calculate the comprehensive score of the interaction intent and generate a personnel perception update event containing the identity identifier when the comprehensive score of the interaction intent reaches the trigger threshold. The concurrent routing module is used to perform audiovisual consistency calculation by combining the sound source arrival direction data with the lip movement state parameters extracted from the continuous video frame sequence, and to allocate the speech text content obtained by parsing the multi-channel audio stream to the data channel of the identity identifier. The state maintenance module is used to perform cross-modal comparison and output maintenance instructions when the visual feature quality of the target user is lower than the feature maintenance standard; The conversation orchestration module is used to receive the personnel perception update event, the maintenance instruction and the voice text content, establish a data partition with the identity identifier as the primary key and store the voice text content; The personnel database is used to store the identity identifiers of the target users, as well as their corresponding historical feature records and basic records.

2. The machine vision-based large language model conversation management system according to claim 1, characterized in that, The feature processing module includes an image evaluation unit and an identity matching unit; The image evaluation unit performs face detection on the continuous video frame sequence to calculate sharpness parameters and occlusion parameters, removes video frames that do not meet the quality standards and retains qualified image frames. The identity matching unit uses a lightweight convolutional neural network to extract the facial feature vector of the qualified image frame, performs a similarity search between the facial feature vector and the historical feature records stored in the personnel database to obtain the maximum similarity score, and associates the identity identifier of the historical feature record with the target user when the maximum similarity score is greater than the identity confirmation threshold.

3. The machine vision-based large language model conversation management system according to claim 1, characterized in that, The intent assessment module extracts the depth value of the three-dimensional spatial depth information contained in the continuous video frame sequence to calculate the actual spatial distance between the target user and the front-end sensor array. It uses the gaze estimation model to extract the pupil feature distribution of the eye region in the continuous video frame sequence to calculate the confidence of the gaze orientation towards the robot. Based on the facial key points, it calculates the three-dimensional Euler angles of the head to calculate the probability score that the head pose is positive. The intent evaluation module calculates the comprehensive score of the interaction intent by combining the confidence level of the gaze toward the robot, the actual spatial distance, and the probability score of the head posture being positive with the system weight coefficient and the system's preset maximum effective interaction distance threshold.

4. The machine vision-based large language model conversation management system according to claim 1, characterized in that, The concurrent routing module extracts the rate of change of the distance between the upper and lower lips of the mouth contour from the continuous video frame sequence as the confidence value of the lip movement activity of the lip movement state parameter, and uses the internal parameter matrix and the external parameter matrix to convert the pixel coordinates of the target user in the two-dimensional color image contained in the continuous video frame sequence into spatial relative orientation angles. The concurrent routing module combines the audiovisual feature fusion weight coefficient, the lip movement activity confidence value, the horizontal deflection angle of the sound source arrival direction angle and the horizontal deflection angle of the spatial relative direction angle, and the maximum spatial direction deviation angle to perform calculations to obtain an audiovisual consistency score.

5. The machine vision-based large language model conversation management system according to claim 4, characterized in that, The concurrent routing module compares the audiovisual consistency scores of all target users and selects the target user with the highest audiovisual consistency score. When the audiovisual consistency score of the target user with the highest audiovisual consistency score is greater than or equal to the fault-tolerant matching threshold, the speech recognition component is invoked to convert the multi-channel audio stream into the speech-text content and allocate it to the data channel of the corresponding identity identifier.

6. The machine vision-based large language model conversation management system according to claim 1, characterized in that, After obtaining the multi-channel audio stream corresponding to the identity identifier in the active state, the state maintenance module extracts the voiceprint feature vector from the multi-channel audio stream and combines the voiceprint feature vector with the sound source arrival direction data to generate an acoustic anchor point. The state maintenance module caches the acoustic anchor point along with the identity identifier in the system memory. When the number of consecutive lost frames of the target user's face in the continuous video frame sequence is greater than the number of frames to be lost or the facial posture deflection angle is greater than the limit deflection angle, the state maintenance module determines that the target user has entered a visual blind spot state.

7. The machine vision-based large language model conversation management system according to claim 6, characterized in that, When the state maintenance module determines that the target user has entered the visual blind zone, it uses the acoustic anchor point cached in the system memory as a reference object, and combines the acoustic feature fusion weight coefficient, the current voiceprint feature vector extracted from the currently acquired multi-channel audio stream, the cached reference voiceprint feature vector, the horizontal deflection angle of the sound source arrival direction angle of the currently acquired multi-channel audio stream, and the cached horizontal deflection angle of the reference sound source arrival direction angle with the blind zone tolerance direction deviation angle set by the system to perform calculation to obtain a cross-modal matching score. When the cross-modal matching score is greater than or equal to the maintenance confirmation threshold, it sends the maintenance instruction to the session orchestration module.

8. The machine vision-based large language model conversation management system according to claim 1, characterized in that, After receiving the personnel perception update event, the session orchestration module establishes a mapping index using the identity identifier as a unique primary key. When the data partition corresponding to the unique primary key does not exist in the system logical memory, a new data partition is dynamically allocated for the unique primary key, and system prompt words and the initial task status tree are loaded into the new data partition. When the data partition corresponding to the unique primary key exists in the system's logical memory and is in an active state, the historical dialogue records, task execution progress, and user preference information contained in the active data partition are extracted to restore the target user's context environment.

9. The machine vision-based large language model conversation management system according to claim 8, characterized in that, The conversation orchestration module appends the voice text content to the data partition corresponding to the unique primary key. When the ratio of the accumulated text length in the data partition to the maximum capacity of the context window reaches the capacity warning threshold, the text summarization algorithm is invoked to compress the historical dialogue records. The conversation orchestration module concatenates the system prompts, the historical dialogue records, and the voice text content in a temporal sequence into a prompt text, and submits it to the large language model inference engine through read-write locks and thread isolation mechanisms.

10. The machine vision-based large language model conversation management system according to claim 1, characterized in that, The personnel database receives and saves historical feature records and basic records. The personnel database uses a moving average algorithm to fuse and update the facial feature vector extracted by the feature processing module with the historical facial feature vector contained in the historical feature records. The personnel database subtracts the current system timestamp from the most recently appeared timestamp to obtain a time difference value. The time difference value is then combined with the cumulative number of interaction rounds, a time decay weight coefficient, and an activity weight coefficient to perform calculations to obtain a data elimination priority score. When the total number of basic records stored in the personnel database reaches the maximum record capacity threshold, the personnel database sorts the basic records in descending order according to the data elimination priority score and deletes the high-priority basic records at the head to release physical storage space.

Images