Work AR intelligent auxiliary management system based on voice AI interaction driving

Abstract

The invention discloses an operation AR intelligent auxiliary management system based on voice AI interaction driving, which is applied to the field of intelligent warehousing and comprises an environment perception type voice interaction module, an AR multi-mode interaction compensation module, a dynamic noise robustness data processing module, an operation management module and a database module. The environment perception type voice interaction module is used for collecting voice signals of operating personnel and inhibiting environment noise; the dynamic noise robustness data processing module is used for carrying out noise reduction processing, semantic understanding and closed-loop optimization on the voice signal and generating feedback information and a control instruction; the AR multi-modal interaction compensation module is used for receiving feedback information of the dynamic noise robustness data processing module and providing visual and tactile interaction compensation, and the job management module is used for scheduling management equipment according to a control instruction; according to the method, the operation efficiency can be effectively improved, the accuracy is high, the real-time performance is high, and interaction experience optimization can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent warehousing, and particularly to a job AR intelligent auxiliary management system based on voice AI interaction driving. BACKGROUND

[0002] In the field of job management today, traditional job management systems occupy a considerable market share. Among them, the job management mode relying on manual recording and paper document circulation has a long history and is widely applied. Under this mode, job personnel need to manually fill in various paper forms to record relevant information of job tasks, such as task content, completion progress, equipment usage, etc. For example, in the warehouse scenario, the record of goods in and out of the warehouse is usually recorded by the staff on the paper account book, including the information of goods name, quantity, in and out time, and the person in charge. In the workshop production scenario, workers record the processing steps, processing time and raw materials used on the paper work order. Then, these paper documents are circulated among different departments or personnel to realize the transmission of information and the promotion of job flow.

[0003] With the preliminary development of information technology, simple digital systems have emerged. Such systems transfer part of the job management process from offline to online, realizing a certain degree of informatization. Job personnel can access the system through computer terminals to perform data entry and query operations. For example, some enterprises use simple spreadsheets or database management systems to record job data. Job personnel input job-related information in the system, and the system can perform simple storage and statistical analysis of these data. In terms of job scheduling, the system can generate a preliminary job plan according to preset rules and display it to management personnel in the form of an electronic spreadsheet or a simple report. However, this simple digital system has relatively limited functions and is not capable of dealing with complex job scenarios and diverse job requirements. The existing management system has the following deficiencies:

[0004] (1) Low efficiency: Manual operation is prone to various errors in job management processes, such as data entry errors, information omissions, etc. When filling out paper documents, staff may mistakenly write the wrong quantity of goods or job time due to negligence, which not only requires additional time for checking and correcting, but also may cause deviations in subsequent job processes, affecting the overall job progress. Moreover, manual operation is time-consuming, especially when dealing with a large number of job tasks, the speed of manual recording and organizing data cannot meet the actual demand. The paper process is cumbersome, from filling out paper documents, passing to storage, each link needs to spend a lot of manpower and time cost. In terms of data aggregation and analysis, it is difficult to quickly and accurately aggregate and analyze the data of paper documents, and managers need to spend a lot of time manually organizing and calculating data to obtain key information such as job progress and equipment utilization, which seriously affects the timeliness and accuracy of decision-making, and thus affects job progress.

[0005] (2) Poor interactive experience: The interface interaction design of traditional systems is often not intuitive, and the operation process is complex. Job personnel need to spend a lot of time learning how to use the system and familiarize themselves with various operation instructions and interface layouts. For some older or less familiar with information technology staff, the learning cost is higher, which may lead to their resistance to using the system and reduce their work enthusiasm. When obtaining key information, job personnel need to switch between multiple interfaces and perform tedious operations to find the required information, making it difficult to quickly obtain key information, which affects work efficiency and job quality to some extent.

[0006] (3) Lack of real-time: In traditional job management systems, data updates often have delays. After the actual situation of the job site changes, the relevant information cannot be timely reflected in the system, because data entry needs manual operation, and the speed and frequency of manual operation are limited, which cannot achieve real-time updating. For example, in workshop production, if the equipment fails or the production progress is completed ahead of schedule, etc., the situation cannot be fed back to the system in time, and the management personnel are difficult to adjust the job plan in time, which may cause waste of production resources or production delay. This delay in data updating makes the system unable to reflect the real situation of the job site in real time, reducing the guiding role of the system in job management.

[0007] (4) Voice AI interaction application problems in job scene: Although voice AI interaction has great potential in job scene application, it still faces many problems. In industrial environment, noise interference is one of the main reasons for low voice recognition accuracy. The huge noise generated by workshop equipment running, such as the roar of machine tools, the running sound of conveyors, and the noise of warehouse machinery such as forklift engines, will superimpose with the voice signal of the job personnel, making the signal-to-noise ratio of the voice signal collected by the microphone array low, thereby increasing the difficulty of voice recognition. When multiple job personnel work at the same time, the voice signals will interfere with each other, and the traditional beamforming algorithm is difficult to accurately locate the target sound source, resulting in the voice recognition system unable to accurately recognize the instructions of each job personnel. In addition, inaccurate voice instruction understanding is also a common problem. Due to the complexity and diversity of natural language, after the voice recognition system converts the voice signal into text, it may not accurately understand the semantics of the text and the intention of the job personnel, resulting in incorrect instruction execution.

[0008] (5) Limitations of AR technology application in job management: There are still some limitations in the application of AR technology in job management. The combination of AR display content and actual job flow is not close enough, and it cannot provide real-time and accurate guidance information for job personnel. In the warehouse scene, the goods storage location information displayed by the AR glasses may deviate from the actual location, or cannot be dynamically updated according to the real-time operation of the job personnel, which makes the job personnel still need to spend time to judge and confirm when using AR technology, and cannot fully play the advantages of AR technology. The single interaction mode is also a problem, and most of the current AR applications only support simple gesture or touch interaction, lacking intelligent guidance function. Job personnel need to manually operate AR devices to obtain information or input instructions during the operation process, which to some extent, distracts the attention of job personnel and affects the job efficiency. Moreover, the lack of intelligent guidance function makes job personnel unable to get effective guidance from the system when facing complex job tasks, increasing the difficulty of job and the possibility of errors.

[0009] Therefore, there is an urgent need for a voice AI interaction driven job AR intelligent auxiliary management system that can effectively improve the efficiency of job, has high accuracy, strong real-time performance, and can optimize the interactive experience, SUMMARY

[0010] The purpose of the present application is to overcome the shortcomings of the prior art and provide a voice AI interaction driven job AR intelligent auxiliary management system that can effectively improve the efficiency of job, has high accuracy, strong real-time performance, and can optimize the interactive experience.

[0011] To achieve the above invention purpose, the technical solution adopted by the present application is as follows:

[0012] The application discloses a job AR intelligent auxiliary management system based on voice AI interaction driving, which is applied to the field of intelligent storage and comprises an environment perception type voice interaction module, an AR multi-modal interaction compensation module, a dynamic noise robustness data processing module, a job management module and a database module.

[0013] The environment perception type voice interaction module is used for collecting voice signals of job personnel and suppressing environmental noise and is connected with the dynamic noise robustness data processing module.

[0014] The AR multi-modal interaction compensation module is used for receiving feedback information of the dynamic noise robustness data processing module and providing visual and tactile interaction compensation and is bidirectionally connected with the dynamic noise robustness data processing module.

[0015] The dynamic noise robustness data processing module is connected with the environment perception type voice interaction module, the AR multi-modal interaction compensation module, the job management module and the database module respectively and is used for carrying out noise reduction processing, semantic understanding and closed-loop optimization on the voice signals and generating feedback information and control instructions.

[0016] The job management module is connected with the dynamic noise robustness data processing module and production equipment or storage equipment and is used for scheduling and managing equipment according to the control instructions.

[0017] The database module is connected with the dynamic noise robustness data processing module and is used for storing job processes, equipment information, product information and scene noise data.

[0018] The environment-aware voice interaction module synchronously collects voice signals, time-domain waveforms, frequency-domain features, and spatial distribution data of environmental noise through a composite sensor array, outputs anti-noise voice signals to the dynamic noise robustness data processing module after processing by the adaptive beamforming processing unit, the dynamic noise robustness data processing module performs real-time noise modeling, noise reduction processing, and recognition model optimization based on confidence feedback on the input anti-noise voice signals, and outputs job instructions and feedback information after semantic understanding, the AR multi-modal interaction compensation module receives job instructions and feedback information and visualizes them, simultaneously acquires correction instructions of gesture or tactile feedback confirmation of the job personnel and returns them to the dynamic noise robustness data processing module, the dynamic noise robustness data processing module generates control instructions according to the correction instructions and sends them to the job management module, meanwhile, the job personnel completes corresponding warehouse operations according to the prompts of the AR multi-modal interaction compensation module, the job management module controls the warehouse equipment to perform warehouse operations according to the control instructions sent by the dynamic noise robustness data processing module, the state monitoring unit monitors the running state of the warehouse equipment in real time, collects the running data of the warehouse equipment, and transmits the data to the dynamic noise robustness data processing module, the dynamic noise robustness data processing module monitors and adjusts the job process in real time according to the real-time collected data to ensure the smooth progress of the job, when the warehouse operation is completed, the AR multi-modal interaction compensation module confirms the completion of the warehouse operation by acquiring voice instructions or tactile feedback of the job personnel and sends the confirmation information to the dynamic noise robustness data processing module, the dynamic noise robustness data processing module updates the warehouse information and job process record in the database module after receiving the confirmation information.

[0019] Preferably, the environment-aware voice interaction module further includes a processor, the processor adopts a Rockchip RK3588, the composite sensor array unit includes a microphone array, an environmental noise sensor, and a direction sensor connected to the processor through I2C and SPI interfaces, the environmental noise sensor includes a MEMS accelerometer and an ultrasonic sensor, the adaptive beamforming processing unit is connected to the composite sensor array unit through a parallel data bus to transmit noise spectrum and spatial distribution data in real time, and anti-noise voice signals are output to the dynamic noise robustness data processing module through a USB 3.0 interface.

[0020] Preferably, the microphone array adopts a multi-channel distributed layout, realizes directional enhancement of voice signals based on a delay-and-sum beamforming principle, strengthens the voice of the job personnel and suppresses peripheral interference through spatial filtering, and the core model is as follows:

[0021]

[0022] where y(t) represents the output signal, x m (t) is the signal received by the mth microphone, τ m is the time delay of the channel, and M is the number of microphones; the MEMS accelerometer in the ambient noise sensor extracts the frequency spectrum characteristics of the vibration signal through Fourier transform, and the formula is:

[0023]

[0024] where X(f) represents the frequency domain signal, x(t) is the time domain vibration acceleration signal, f is the frequency, and j is the imaginary unit; the MEMS signal reflects the amplitude-frequency relationship of the mechanical structure noise source; the ultrasonic sensor obtains the sound source direction information through the ranging and sound intensity positioning method, and the basic positioning model is the sound wave propagation time difference method:

[0025]

[0026] where θ represents the angle between the sound source and the front of the array, c is the sound speed, Δt is the time difference between the left and right channels, and d is the distance between the two sensors.

[0027] Preferably, the adaptive beamforming processing unit cooperates closely with the composite sensor array unit, receives the noise frequency spectrum characteristics and spatial distribution information output by the ambient noise sensor and the azimuth sensor in real time, constructs a dynamic noise field modeling and target sound source positioning mechanism, and dynamically calculates the weight coefficient of the microphone array based on the minimum variance distortionless response algorithm, thereby forming a high-gain pickup beam pointing to the oral position of the operator; the core optimization target is:

[0028]

[0029] where w represents the microphone array weight vector, R is the noise covariance matrix, d is the target direction vector, and w H represents the conjugate transpose of w; the analytical solution of the optimization problem is:

[0030]

[0031] where w optrepresents the optimal weight vector, which determines the contribution weight of each microphone channel to the final output signal, the oral cavity space coordinates provided by the orientation sensor are used to construct the steering vector d in real time, and the frequency spectrum data sensed by the ambient noise sensor is used to construct the covariance matrix R, so that the beam forming process has the response ability to the current sound field state; when the ambient noise sensor detects a strong noise event from a specific direction, the adaptive beam forming processing unit forms a main lobe between 30° and 60° in front by adjusting the array weighting strategy, and forms a strong suppression side lobe in the rear, so as to realize the enhancement of the target voice and the suppression of the noise direction signal; finally, the processed high signal-to-noise ratio voice signal is input in real time to the dynamic noise robustness data processing module for subsequent semantic analysis and job instruction generation.

[0032] Preferably, the dynamic noise robustness data processing module includes a real-time noise modeling submodule, a recognition-noise reduction combined optimization submodule, an analysis decision submodule, an instruction generation submodule, a semantic understanding unit and a voice recognition unit; the real-time noise modeling submodule cooperates with the environment perception type voice interaction module to dynamically model and real-time suppress the complex background noise in the industrial field, and the core mechanism is to model the noise state transition and parameter estimation based on the improved hidden Markov model, and to fuse the deep neural network and the long short-term memory network structure; the hidden Markov model models the characteristics of the background noise changing over time through the state transition probability matrix and the observation probability function, and the model form is:

[0033]

[0034] Wherein, P(O|λ) represents the observation sequence probability under the model parameter λ, Q is the hidden state sequence, O is the observed noise feature sequence, P(Q|λ) is the state transition probability, P(O|Q,λ) is the observation probability under the given state sequence, and the transition and output probability of the noise state can be updated online by combining the Baum-Welch algorithm, so as to realize the dynamic estimation of the noise power spectrum density; the estimation result and the noise voice signal collected by the current microphone array are input into the DNN-LSTM hybrid model, the DNN-LSTM hybrid model first extracts the static spectrum feature by the deep neural network, and then models the context correlation in the time sequence by the long short-term memory network, and the output is the estimated pure voice spectrum The overall prediction process can be represented as:

[0035]

[0036] Wherein, Y(f,t) represents the noise voice spectrum, N(f,t) represents the real-time noise spectrum feature, is the predicted clean speech spectrum; the model is trained to minimize reconstruction error with mean square error loss function:

[0037]

[0038] where S(f, t) is the real speech spectrum, is the model output, T and F are the number of time frames and frequency points, respectively. Preferably, the recognition-noise reduction joint optimization sub-module is the core component of the dynamic noise robustness data processing module, and forms a closed-loop feedback path with the real-time noise modeling sub-module and the semantic understanding unit, and drives the adaptive optimization of the noise reduction model and the recognition model through the confidence evaluation mechanism of the speech recognition result; the recognition-noise reduction joint optimization sub-module generates a confidence score based on the connection time sequence classification loss function, which is used to quantify the matching degree between the text output by the speech recognition unit and the target label, and the core loss function is:

[0039]

[0040] where, represents the connection time sequence classification loss, x is the input speech feature sequence, y is the target text label sequence, and π is all possible paths, B -1 ( y ) represents the path set that can be mapped from the target sequence, and P(π|x) is the probability of the path given the input; the recognition-noise reduction joint optimization sub-module calculates the confidence score in real time and compares it with the set threshold, and when the score is lower than 80%, the parameter adjustment strategy of the noise reduction algorithm is triggered, and the spectral adjustment process is represented as:

[0041]

[0042] where, is the corrected speech spectrum, Y(f, t) is the original noisy spectrum, is the estimated noise spectrum, α is the attenuation coefficient, and ∈ is the minimum threshold to avoid negative values; after the error correction process, the recognition-noise reduction joint optimization sub-module forms a training sample by combining the corrected text and the original speech, and updates the parameters of the recognition model and the noise reduction network in real time through back propagation to minimize the prediction error and enhance the adaptability to similar noise scenes; the AR intelligent auxiliary management system forms an online learning closed loop of "collection-noise reduction-recognition-feedback-optimization", and the optimization process aims to minimize the joint loss function, which is expressed as:

[0043] L total = λ1L CTC + λ2L denoise

[0044] wherein, L total is a joint loss function, L total is a recognition loss, L denoise is a denoising loss, and λ1, λ2 are adjustable weight coefficients.

[0045] The semantic understanding unit is connected with the recognition-denoising joint optimization submodule, and the optimized text is subjected to semantic analysis to determine the specific content and intention of the operation instruction; the data integration submodule calls information related to the goods from the database module, and the information related to the goods includes the storage position of the goods, the warehousing or out-of-warehouse process; the analysis and decision submodule analyzes and processes the integrated data, determines the best storage path and operation steps according to the preset operation optimization algorithm; and the instruction generation submodule generates corresponding feedback information and control instructions according to the results of the analysis and decision submodule.

[0046] Preferably, the AR multi-modal interaction compensation module comprises an AR display auxiliary confirmation unit and a tactile feedback correction unit; the AR display auxiliary confirmation unit is embedded in the AR glasses worn by the operation personnel and is in real-time communication connection with the dynamic noise robustness data processing module; the processor of the AR display auxiliary confirmation unit is an ARM Cortex-A53 processor, which interacts with the dynamic noise robustness data processing module through a MIPI_DSI interface, and a gesture correction signal is returned through an SPI interface; after the voice recognition unit recognizes the user instruction text, a semi-transparent floating window is generated in the field of view of the AR glasses to display the current recognition result and multiple candidate words with high confidence, thereby avoiding misjudgment of the operation instruction due to single voice recognition error; the floating window is in the form of parallel arrangement or up-down sliding list, thereby quickly completing confirmation.

[0047] Preferably, the tactile feedback correction unit is integrated in the AR glasses or smart gloves worn by the operation personnel, and is in real-time connection with the dynamic noise robustness data processing module to build an instant perception and intervention mechanism for the industrial operation site; the tactile feedback correction unit adopts an STM32H7 series processor, receives device state conflict signals through a GPIO interface, realizes early warning through vibration at different frequencies, and completes control instruction interaction with the dynamic noise robustness data processing module through a UART interface; the tactile feedback correction unit is used to send a warning signal to the operation personnel in the form of physical vibration when a voice instruction is recognized to cause a risk or misoperation, thereby guiding the operation personnel to actively check whether the current operation intention is consistent with the actual state; after the data processing module receives the voice recognition result and performs semantic understanding, device state information in the database module is automatically called and compared with the instruction, and if a conflict is found, the tactile feedback mechanism is triggered immediately.

[0048] Preferably, the job management module further comprises a task scheduling unit, a device control unit and a processor; the task scheduling unit is used to control the forklift to assist the job personnel to carry goods according to the control instruction sent by the dynamic noise robustness data processing module; the device control unit is used to generate specific device control signals to control the forklift to travel according to the specified path and accurately carry the goods to the warehouse or warehouse location; the processor of the job management module adopts an industrial-grade ARM Cortex-A72, and interacts with the dynamic noise robustness data processing module through Ethernet to control the control instruction in Modbus TCP protocol; when the processor of the job management module is connected with the production equipment or the warehouse equipment, the industrial sensor is adapted through the RS-485 interface, and the warehouse equipment is controlled in CANopen protocol; the state monitoring data of the warehouse equipment is transmitted to the dynamic noise robustness data processing module through WiFi in MQTT protocol, and a job process closed loop management is formed.

[0049] Preferably, a scenario noise database is constructed in the database module, typical industrial scene noise is collected, a plurality of noise types are included, and a 75-115dB sound pressure level range is covered; the typical industrial scene noise at least includes lathe noise, warehouse shelf carrying noise and pneumatic tool noise.

[0050] Beneficial effects

[0051] Compared with the prior art, the beneficial effects obtained by the present application are:

[0052] (1) Efficiency improvement: the present application is driven by voice AI interaction, the job personnel only needs to speak the job instruction, and the corresponding operation can be quickly identified and executed, which greatly reduces the time and workload of manual input instruction. In the warehouse scene, the job personnel does not need to manually find the goods storage location information, and can quickly obtain the goods location and operate through the voice instruction, which shortens the goods warehouse time, realizes the automatic scheduling and management of the operation process, reduces the manual intervention, improves the continuity and fluency of the operation. In the workshop production scene, the system can automatically arrange the processing task of the production equipment according to the production task and the device state, avoid the mistakes and delays that may occur in manual scheduling, and improve the production efficiency. In addition, the present application can also collect and process the job site data in real time, find problems in time and make adjustments, and further improve the operation efficiency.

[0053] (2) Accuracy improvement: The application of voice recognition and AR technology in the invention effectively reduces the occurrence of human errors. The voice recognition unit uses advanced algorithms to accurately recognize the voice commands of the operator, avoiding errors that may occur during manual input. The AR multi-modal interaction compensation module visually displays the operation information in the operator's field of view, allowing the operator to clearly understand the operation requirements and steps, reducing operation errors caused by information understanding errors. The invention has real-time verification and feedback functions, which can monitor and verify the operator's operation in real time, and timely discover and correct errors. During the process of goods entering and leaving the warehouse, the quantity, model and other information of the goods are checked in real time. If errors are found, the operator will be prompted to check and correct them, ensuring the accuracy of the operation.

[0054] (3) Interactive experience optimization: The natural voice interaction and immersive AR display of the invention make the operation more convenient and intuitive, reducing the learning cost of the operator. The operator does not need to learn complex operation interfaces and instructions, but only needs to interact with the system in natural language to complete the operation task, improving the operator's work enthusiasm and satisfaction. When using AR glasses for operation, the operator can experience immersive interaction, as if virtual information and real environment are integrated into one, enhancing the fun and immersion of the operation. In addition, the invention also supports multi-modal interaction, and the operator can choose voice, gesture, touch and other interaction methods according to their needs, improving the flexibility and convenience of interaction.

[0055] (4) Real-time enhancement: The invention can collect various data of the operation site in real time, including equipment status, operation progress, etc., and process and analyze these data in real time. According to the real-time data, the system can provide decision basis for the manager in time, helping the manager to make decisions quickly and adjust the operation plan. In workshop production, when the equipment fails, the invention can detect the fault immediately and feed back the fault information to the manager, who can arrange maintenance personnel for maintenance in time according to the information provided by the system, avoiding the impact of the fault on production, and also optimizing the operation process according to the real-time data to improve the operation efficiency and quality. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is the system architecture diagram of the invention;

[0057] Figure 2 is the system signaling diagram of the invention;

[0058] Figure 3 is the connection diagram of the dynamic noise robustness data processing module, the AR multi-modal interaction compensation module and the operation management module of the invention;

[0059] Figure 4 is a connection diagram of the environment-aware voice interaction module and the dynamic noise robustness data processing module of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application clearer and more complete, the present application will be further described in detail below in combination with embodiments. Obviously, the embodiments described below are some embodiments of the present application, but the scope of protection claimed by the present application is not limited to the specific embodiments.

[0061] As shown in Figures 1 to 4 , the voice AI interaction driven job AR intelligent auxiliary management system is applied to the field of intelligent warehousing and includes an environment-aware voice interaction module, an AR multi-modal interaction compensation module, a dynamic noise robustness data processing module, a job management module and a database module.

[0062] As shown in Figure 1 and Figure 4 , the environment-aware voice interaction module is used to collect the voice signals of the job personnel and suppress environmental noise, and is connected with the dynamic noise robustness data processing module. The environment-aware voice interaction module can effectively solve the problem of decreased voice command recognition accuracy caused by industrial environmental noise interference. The environment-aware voice interaction module includes a composite sensor array unit, an adaptive beamforming processing unit and a processor. The processor is an ARM architecture core processor using a Swallow Micro RK3588, and an information processing unit is constructed by combining a RAM memory, a ROM memory and a power management IC. The composite sensor array unit includes a microphone array, an environmental noise sensor and a direction sensor which are accessed to the processor through I2C and SPI interfaces. The environmental noise sensor includes a MEMS accelerometer and an ultrasonic sensor. The adaptive beamforming processing unit is connected with the composite sensor array unit through a parallel data bus to transmit noise spectrum and spatial distribution data in real time, and outputs the anti-noise voice signals to the dynamic noise robustness data processing module through a USB 3.0 interface.

[0063] The microphone array adopts a multi-channel distributed layout, realizes directional enhancement of voice signals based on the delay-and-sum beamforming principle, strengthens the voice of the job personnel and suppresses the surrounding interference through spatial filtering, and the core model thereof is as follows:

[0064]

[0065] wherein y(t) represents an output signal, x m (t) is a signal received by the mth microphone, τ mThe time delay of the channel is M, and M is the number of microphones; the MEMS accelerometer in the ambient noise sensor extracts the frequency spectrum characteristics of the vibration signal through Fourier transform, and the formula is:

[0066]

[0067] wherein X(f) represents a frequency domain signal, x(t) is a time domain vibration acceleration signal, f is a frequency, and j is an imaginary unit; the MEMS signal reflects the amplitude-frequency relationship of the mechanical structure noise source; the ultrasonic sensor obtains the direction information of the sound source through the ranging and sound intensity positioning method, and the basic positioning model is the sound wave propagation time difference method:

[0068]

[0069] wherein θ represents the included angle between the sound source and the front of the array, c is the sound speed, Δt is the time difference between the left and right channels, and d is the distance between the two sensors. The orientation sensor cooperates with the head or face tracking system to obtain the relative position of the oral cavity of the worker and the spatial geometric relationship between the oral cavity and the noise source in real time, provides spatial constraints for speech signal extraction, and supports the spatial filtering input of the sound source separation and echo suppression algorithm, thereby improving the stability and accuracy of speech signal processing in a dynamic industrial environment.

[0070] The adaptive beamforming processing unit closely cooperates with the composite sensor array unit, constructs a dynamic noise field modeling and target sound source positioning mechanism by receiving the noise spectrum characteristics and spatial distribution information output by the ambient noise sensor and the orientation sensor in real time, and dynamically calculates the weight coefficient of the microphone array based on the minimum variance distortionless response algorithm, thereby forming a high-gain pickup beam pointing to the oral cavity position of the worker; this method minimizes the output power on the premise of maintaining the amplitude distortionlessness of the target speech signal, so as to achieve the effect of suppressing noise in non-target directions, and the core optimization target is:

[0071]

[0072] wherein w represents the microphone array weight vector, R is the noise covariance matrix, d is the steering vector of the target direction, and w H represents the conjugate transpose of w; the analytical solution of the optimization problem is:

[0073]

[0074] wherein w optThe optimal weight vector determines the contribution weight of each microphone channel to the final output signal. In actual operation, the oral cavity space coordinates provided by the orientation sensor are used to construct the steering vector d in real time, and the frequency spectrum data sensed by the ambient noise sensor is used to construct the covariance matrix R, so that the beam forming process has the response ability to the current sound field state. When the ambient noise sensor detects a strong noise event from a specific direction (such as a forklift approaching), the adaptive beam forming processing unit forms a main lobe between 30° and 60° in front by adjusting the array weighting strategy, while forming a strong suppression side lobe in the rear, thereby achieving the enhancement of the target voice and the suppression of the noise direction signal. Finally, the processed high signal-to-noise ratio voice signal is input in real time to the dynamic noise robustness data processing module for subsequent semantic analysis and job instruction generation.

[0075] The AR multi-modal interaction compensation module is used for receiving feedback information of the dynamic noise robustness data processing module and providing visual and tactile interaction compensation, and is bidirectionally connected with the dynamic noise robustness data processing module. The AR multi-modal interaction compensation module includes an AR display auxiliary confirmation unit and a tactile feedback error correction unit. The AR display auxiliary confirmation unit is embedded in the AR glasses worn by the operation personnel and is in real-time communication connection with the dynamic noise robustness data processing module. The processor of the AR display auxiliary confirmation unit is an ARM Cortex-A53 processor, which interacts with the dynamic noise robustness data processing module through a MIPI_DSI interface, and a gesture correction signal is returned through an SPI interface. After the voice recognition unit recognizes the user instruction text, a semi-transparent floating window is generated in the field of view of the AR glasses to display the current recognition result and multiple candidate words with high confidence, thereby avoiding misjudgment of the operation instruction caused by single voice recognition error. The floating window adopts a parallel arrangement or an up-down sliding list form, so as to quickly complete confirmation. If "take goods" is misrecognized as "put goods", the floating window will display "put goods", "take goods", "return goods" and other related operation options, and the operation personnel only need to click the target word with the index finger or slide left and right to complete selection, and will immediately receive the selected content and update the recognition result. The whole process constitutes an interactive closed loop of "voice input-AR confirmation-gesture correction", which not only significantly improves the recognition reliability, but also enhances the practicality of the system under noise interference. This mechanism allows the recognition result to be corrected by user intervention before being actually executed, ensures the accuracy of the task operation, effectively reduces the risk of instruction miscommunication caused by voice misrecognition, and is especially suitable for industrial scenarios such as warehouse handling and equipment control which require high operation precision.

[0076] The tactile feedback correction unit is integrated in the AR glasses or smart gloves worn by the operator, and is connected in real time with the dynamic noise robustness data processing module to build an instant perception and intervention mechanism for the industrial operation site; the tactile feedback correction unit adopts an STM32H7 series processor, receives device state conflict signals through a GPIO interface, realizes early warning through vibration at different frequencies, and completes control instruction interaction with the dynamic noise robustness data processing module through a UART interface; the tactile feedback correction unit is used to send a warning signal to the operator in the form of physical vibration when the system identifies that a voice instruction may cause risks or misoperations, so as to guide the operator to actively check whether the current operation intention is consistent with the actual state; after the data processing module receives the voice recognition result and performs semantic understanding, it will automatically call the device state information in the database module and compare it with the instruction, and if a conflict is found, the tactile feedback mechanism will be triggered immediately. For example, when the operator issues the order "start No. 3 machine tool", before the system identifies and issues the instruction, it finds that No. 3 machine tool is currently in a fault shutdown state, so the tactile feedback unit will prompt the operator that the current instruction has an execution risk through two consecutive high-frequency short vibrations. After receiving the vibration prompt, the operator can choose to re-input the instruction through voice or further check the device state information through the AR display auxiliary confirmation unit to make a judgment. The tactile feedback signal realizes the coding of multiple warning levels through the combination of different vibration frequencies, vibration durations and vibration intervals, for example, a single short vibration represents a slight conflict, continuous vibration represents a serious conflict, and intermittent vibration represents a fuzzy state that needs to be confirmed manually, so as to realize a more fine operation safety prompt mechanism and improve the intelligent fault tolerance capability and operation robustness of the system in the human-computer interaction process. This mechanism is especially suitable for workshops or warehouse environments with high noise and high complexity. In the case of limited audio-visual channels or increased voice recognition errors, a multi-modal interaction closed loop complementary to voice and vision is formed, effectively avoiding misoperations, equipment damage or safety hazards caused by misjudgment.

[0077] As Figure 3As shown, the dynamic noise robustness data processing module is connected with the environment-aware voice interaction module, the AR multi-modal interaction compensation module, the operation management module and the database module respectively, and is used for noise reduction processing, semantic understanding and closed-loop optimization of the voice signal, and generates feedback information and control instructions; the dynamic noise robustness data processing module includes a real-time noise modeling submodule, an identification-noise reduction joint optimization submodule, a data integration submodule, an analysis and decision submodule, an instruction generation submodule, a semantic understanding unit, a voice recognition unit and a voice preprocessing unit; the real-time noise modeling submodule cooperates with the environment-aware voice interaction module to dynamically model and real-time suppress the complex background noise in the industrial field, and the core mechanism is to model and estimate the noise state transition based on the improved hidden Markov model, and to fuse the deep neural network and the long short-term memory network structure; the hidden Markov model models the characteristics of the background noise changing over time through the state transition probability matrix and the observation probability function, and the model form is:

[0078]

[0079] Wherein, P(O|λ) represents the observation sequence probability under the model parameter λ, Q is the hidden state sequence, O is the observed noise feature sequence, P(Q|λ) is the state transition probability, P(O|Q,λ) is the observation probability under the given state sequence, and the transition and output probability of the noise state can be updated online by combining the Baum-Welch algorithm to realize dynamic estimation of the noise power spectrum density; the estimation result and the noise-containing voice signal collected by the current microphone array are jointly input into the DNN-LSTM hybrid model, the DNN-LSTM hybrid model first extracts static spectrum features by the deep neural network, and then models the context correlation in the time sequence by the long short-term memory network, and the output is the estimated pure voice spectrum The overall prediction process can be represented as:

[0080]

[0081] Wherein, Y(f,t) represents the noise-containing voice spectrum, N(f,t) represents the real-time noise spectrum feature, The predicted pure voice spectrum; in the training process, the model takes minimizing the reconstruction error as the goal, and adopts the mean square error loss function for optimization:

[0082]

[0083] Wherein, S(f,t) is the real voice spectrum, For model output, T and F are the number of time frames and frequency points, respectively. Through the above structure, the real-time noise modeling submodule can not only accurately adapt to the non-stationary noise characteristics in the workshop, but also effectively improve the restoration quality of the speech signal, providing high-confidence speech input for the dynamic noise robust data processing module.

[0084] The recognition-noise reduction joint optimization submodule, as the core component of the dynamic noise robust data processing module, forms a closed-loop feedback path with the real-time noise modeling submodule and the semantic understanding unit, and drives the adaptive optimization of the noise reduction model and the recognition model through the confidence evaluation mechanism of the speech recognition result. The recognition-noise reduction joint optimization submodule relies on the connection time sequence classification loss function to generate a confidence score for the recognition result, which is used to quantify the matching degree between the text output by the speech recognition unit and the target label. Its core loss function is:

[0085]

[0086] wherein, represents the connection time sequence classification loss, x is the input speech feature sequence, y is the target text label sequence, π is all possible paths, B -1 ( y ) represents the path set that can be mapped from the target sequence, P(π|x) is the probability of the path given the input; the recognition-noise reduction joint optimization submodule calculates the confidence score in real time and compares it with the set threshold. When the score is lower than 80%, the parameter adjustment strategy of the noise reduction algorithm is triggered, wherein the attenuation coefficient α of the spectral subtraction is adaptively improved to enhance the noise suppression ability. The spectral adjustment process is represented as:

[0087]

[0088] wherein, is the corrected speech spectrum, Y(f,t) is the original noisy spectrum, is the estimated noise spectrum, α is the attenuation coefficient, ∈ is the minimum threshold to avoid negative values; after the error correction process, the recognition-noise reduction joint optimization submodule forms a training sample by combining the corrected text with the original speech, and updates the parameters of the recognition model and the noise reduction network in real time through back propagation to minimize the prediction error and enhance the adaptability to similar noise scenes; the overall AR intelligent auxiliary management system forms an online learning closed loop of "collection-noise reduction-recognition-feedback-optimization", and the optimization process aims to minimize the joint loss function, which is expressed as:

[0089] L total = λ1L CTC + λ2L denoise

[0090] wherein, Ltotal L is a joint loss function total L is a recognition loss denoise L is a noise reduction loss, and λ1 and λ2 are adjustable weight coefficients; through the above mechanism, the sub-module realizes robust processing and continuous evolution of unstructured voice data, and significantly improves the recognition accuracy and interaction stability of the system in a complex industrial environment.

[0091] The semantic understanding unit is connected with the recognition-noise reduction joint optimization sub-module, performs semantic analysis on the optimized text, and determines the specific content and intention of the operation instruction; the data integration sub-module calls information related to the goods from the database module, the information related to the goods includes the storage position of the goods, the warehousing or delivery process; the analysis and decision sub-module analyzes and processes the integrated data, determines the best warehousing path and operation steps according to the preset operation optimization algorithm. The instruction generation sub-module generates corresponding feedback information and control instructions according to the result of the analysis and decision sub-module.

[0092] The system of the application is pre-trained for different noise scenes before leaving the factory, generates scene-specific noise reduction parameter configuration files (such as "workshop processing mode" and "warehouse handling mode"), and operation personnel can switch the adaptive mode through the AR interface. The dynamic noise robustness data processing module retrieves the corresponding noise reduction parameters for noise processing according to the selected mode.

[0093] The operation management module is connected with the dynamic noise robustness data processing module and production equipment or warehouse equipment, and is used for scheduling and managing equipment according to control instructions; the operation management module includes a state monitoring unit, a task scheduling unit, a device control unit and a processor; the state monitoring unit is used for monitoring the running state of the warehouse equipment and collecting the running data of the warehouse equipment; the task scheduling unit is used for arranging a forklift to assist an operation personnel to carry goods according to the control instruction sent by the dynamic noise robustness data processing module; the device control unit is used for generating a specific device control signal to control the forklift to travel according to a specified path and accurately carry the goods to a warehousing position; the processor of the operation management module adopts an industrial-grade ARM Cortex-A72, and interacts with the dynamic noise robustness data processing module through Ethernet to control the control instruction in the ModbusTCP protocol; when the processor of the operation management module is connected with the production equipment or the warehouse equipment, the industrial sensor is adapted through the RS-485 interface, and the warehouse equipment is controlled in the CANopen protocol; the state monitoring data of the warehouse equipment is transmitted to the dynamic noise robustness data processing module through WiFi in the MQTT protocol, and a work flow closed loop management is formed.

[0094] The database module is connected with the dynamic noise robustness data processing module, and is used for storing job flow, equipment information, product information and scenario noise data; a scenario noise database is constructed in the database module, typical industrial scene noise is collected, various noise types are contained, and a range of 75-115 dB sound pressure level is covered; the typical industrial scene noise at least includes lathe noise, warehouse shelf handling noise and pneumatic tool noise.

[0095] The environment perception type voice interaction module synchronously collects voice signal, environment noise time domain waveform, frequency domain feature and spatial distribution data through a composite sensor array, outputs anti-noise voice signal to the dynamic noise robustness data processing module after being processed by the adaptive beam forming processing unit; the dynamic noise robustness data processing module performs real-time noise modeling, noise reduction processing and recognition model optimization based on confidence feedback on the input anti-noise voice signal, and outputs job instruction and feedback information after semantic understanding; the AR multi-modal interaction compensation module receives the job instruction and feedback information and visually displays, simultaneously acquires the correction instruction of gesture or tactile feedback confirmation of the job personnel and returns to the dynamic noise robustness data processing module; the dynamic noise robustness data processing module generates control instruction according to the correction instruction and sends to the job management module; meanwhile, the job personnel completes corresponding warehouse operation according to the prompt of the AR multi-modal interaction compensation module; the job management module controls the warehouse equipment to perform warehouse operation according to the control instruction sent by the dynamic noise robustness data processing module; the state monitoring unit monitors the running state of the warehouse equipment in real time, collects the running data of the warehouse equipment, and transmits the data to the dynamic noise robustness data processing module; the dynamic noise robustness data processing module monitors and adjusts the job process in real time according to the real-time collected data, and ensures the smooth progress of the job; when the warehouse operation is completed, the AR multi-modal interaction compensation module confirms the completion of the warehouse operation by acquiring the voice instruction or tactile feedback of the job personnel, and sends the confirmation information to the dynamic noise robustness data processing module; after receiving the confirmation information, the dynamic noise robustness data processing module updates the warehouse information and job flow record in the database module.

[0096] Compared with the prior art, the present application has the following beneficial effects:

[0097] (1) Efficiency improvement: The application is driven by voice AI interaction. The operator only needs to speak the operation instruction, which can quickly identify and execute the corresponding operation, greatly reducing the time and workload of manual input instruction. In the warehouse scenario, the operator does not need to manually search for the storage location information of the goods. Through voice instruction, the operator can quickly obtain the location of the goods and operate, shorten the time of goods in and out of the warehouse, realize the automatic scheduling and management of the operation process, reduce manual intervention, and improve the continuity and smoothness of the operation. In the workshop production scenario, the system can automatically arrange the processing tasks of the production equipment according to the production tasks and equipment status, avoiding the mistakes and delays that may occur in manual scheduling, and improving the production efficiency. In addition, the application can also collect and process operation site data in real time, timely find problems and make adjustments, and further improve the operation efficiency.

[0098] (2) Accuracy improvement: The application of voice recognition and AR technology effectively reduces the occurrence of human errors. The voice recognition unit uses advanced algorithms to accurately recognize the voice instructions of the operator, avoiding errors that may occur in manual input. The AR multi-modal interaction compensation module displays the operation information directly in the operator's field of view, allowing the operator to clearly understand the operation requirements and operation steps, reducing operation errors caused by information understanding errors. The application has real-time verification and feedback functions, which can monitor and verify the operator's operation in real time, and timely find and correct errors. In the process of goods in and out of the warehouse, the quantity, model and other information of the goods are verified in real time. If an error is found, a prompt will be issued immediately, requiring the operator to check and correct it, ensuring the accuracy of the operation.

[0099] (3) Interaction experience optimization: The natural voice interaction and immersive AR display of the application make the operation more convenient and intuitive, reducing the learning cost of the operator. The operator does not need to learn complex operation interface and instructions, but only needs to interact with the system in natural language to complete the operation task, improving the operator's work enthusiasm and satisfaction. When using AR glasses for operation, the operator can experience immersive interaction experience, as if virtual information and real environment are integrated into one, enhancing the interest and immersion of operation. In addition, the application also supports multi-modal interaction, and the operator can choose voice, gesture, touch and other interaction modes according to their needs, improving the flexibility and convenience of interaction.

[0100] (4) Real-time enhancement: The present application can collect various data of the work site in real time, including equipment status, work progress, etc., and process and analyze these data in real time. According to the real-time data, the system can provide decision basis for the management personnel in time, help the management personnel make decisions quickly, and adjust the work plan. In the workshop production, when the equipment fails, the present application can detect it immediately and feed back the fault information to the management personnel, who can arrange maintenance personnel to repair in time according to the information provided by the system, avoiding the impact of the fault on production, and also optimizing the work flow according to the real-time data to improve the work efficiency and quality.

[0101] The working flow of the present application is described in detail below taking the warehouse work as an example:

[0102] For example, as shown in FIG. 1, the warehouse work flow includes the following steps: Figure 2As shown, the operator wears AR glasses and ensures that the microphone array is in normal working condition. When the goods need to be put into storage, the operator issues a voice command through the AR multi-modal interactive compensation module, such as "I want to put XX goods into storage". The microphone array collects the voice signal of the operator and transmits it to the adaptive beamforming processing unit for noise reduction processing, and inputs it to the voice preprocessing unit for noise reduction, gain adjustment and other preprocessing operations to improve the quality of the voice signal. Then, the voice recognition unit converts the preprocessed voice signal into a digital signal and uses a voice recognition algorithm to recognize the digital signal to obtain a text form of the operation instruction "I want to put XX goods into storage" and transmit it to the semantic understanding unit of the data processing module. After receiving the operation instruction, the semantic understanding unit uses a natural language processing algorithm to analyze the semantics of the text to determine the specific content and intent of the operation instruction as the goods storage operation. Then, the data integration submodule retrieves information related to XX goods from the database module, such as the storage location of the goods, the storage process, etc., while receiving real-time data collected by the operation site sensor, such as the temperature, humidity of the warehouse, the occupancy of the goods storage area, etc. The analysis and decision-making submodule analyzes and processes the integrated data to determine the optimal storage path and operation steps according to the pre-set operation optimization algorithm. The instruction generation submodule generates corresponding feedback information and control instructions according to the results of the analysis and decision-making submodule. The feedback information is informed to the operator through the AR multi-modal interactive compensation module, such as "please carry XX goods to XX shelf X layer", and the storage location and storage path of the goods are displayed in the form of AR in the field of view of the AR glasses. The operator carries the goods to the designated location according to the prompt of the AR multi-modal interactive compensation module. At the same time, the task scheduling unit of the operation management module controls the forklift and other equipment to assist the operator in carrying the goods according to the control instructions sent by the dynamic noise robustness data processing module. The device control unit generates specific device control signals to control the forklift to drive according to the specified path and accurately carry the goods to the storage location. The state monitoring unit monitors the running state of the forklift and other equipment in real time, collects the running data of the equipment such as the driving speed and the carrying weight of the goods, and transmits the data to the dynamic noise robustness data processing module; the dynamic noise robustness data processing module adjusts the operation process in real time according to these real-time data to ensure the smooth operation. When the goods are put into storage, the operator confirms the completion of the storage operation through a voice command or a tactile feedback of the AR multi-modal interactive compensation module. The dynamic noise robustness data processing module receives the confirmation information and updates the goods inventory information and operation process record in the database module to complete the goods storage operation.

[0103] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed by the present application should be covered in the protection scope of the present application. According to the disclosure and teaching of the above description, the person skilled in the art can also change and modify the above embodiments. Therefore, the present application is not limited to the specific embodiments disclosed and described above, and some modifications and changes of the present application should fall within the protection scope of the claims of the present application. In addition, although some specific terms are used in the specification, these terms are only for convenience of description and do not constitute any limitation on the present application.

Claims

1. A voice-AI interaction-driven AR-based intelligent auxiliary management system for operations, applied in intelligent warehousing or production fields, characterized in that: include: The system includes an environment-aware voice interaction module, an AR multimodal interaction compensation module, a dynamic noise robust data processing module, a job management module, and a database module. The environment-aware voice interaction module is used to collect the voice signals of the operators and suppress environmental noise. It is connected to the dynamic noise robust data processing module. The environment-aware voice interaction module includes a composite sensor array unit and an adaptive beamforming processing unit. The AR multimodal interaction compensation module is used to receive feedback information from the dynamic noise robustness data processing module and provide visualization and tactile interaction compensation. It is bidirectionally connected to the dynamic noise robustness data processing module. The dynamic noise robust data processing module is connected to the environment-aware voice interaction module, the AR multimodal interaction compensation module, the job management module, and the database module, respectively, and is used to perform noise reduction processing, semantic understanding, and closed-loop optimization on the voice signal, and generate feedback information and control commands. The operation management module is connected to the dynamic noise robust data processing module and the production equipment or storage equipment, and is used to schedule and manage the equipment according to control commands; the operation management module includes a status monitoring unit, which is used to monitor the operating status of the storage equipment and collect the operating data of the storage equipment; The database module is connected to the dynamic noise robust data processing module to store work processes, equipment information, product information, and contextualized noise data; The environment-aware voice interaction module synchronously collects voice signals, environmental noise time-domain waveforms, frequency-domain characteristics, and spatial distribution data through a composite sensor array. After processing by the adaptive beamforming processing unit, it outputs noise-resistant voice signals to the dynamic noise robust data processing module. The dynamic noise robustness data processing module performs real-time noise modeling, noise reduction processing, and confidence-based recognition model optimization on the input noise-resistant speech signal, and outputs semantically understood work instructions and feedback information; the AR multimodal interaction compensation module receives work instructions and feedback information and displays them visually, and at the same time obtains correction instructions confirmed by the operator's gestures or tactile feedback and sends them back to the dynamic noise robustness data processing module. The dynamic noise robustness data processing module generates control commands based on the correction instructions and sends them to the operation management module; simultaneously, operators perform corresponding storage operations based on prompts from the AR multimodal interaction compensation module; the operation management module controls the storage equipment to perform storage operations based on the control commands sent by the dynamic noise robustness data processing module; the status monitoring unit monitors the operating status of the storage equipment in real time, collects the operating data of the storage equipment, and transmits the data to the dynamic noise robustness data processing module; The dynamic noise robustness data processing module monitors and adjusts the operation process in real time based on the real-time collected data to ensure the smooth progress of the operation. When the warehousing operation is completed, the AR multimodal interaction compensation module confirms the completion of the warehousing operation by obtaining the voice command or tactile feedback of the operator and sends the confirmation information to the dynamic noise robustness data processing module. After receiving the confirmation information, the dynamic noise robustness data processing module updates the warehousing information and operation process records in the database module.

2. The AR-assisted management system for assignments based on voice AI interaction as described in claim 1, characterized in that: The environmentally aware voice interaction module includes a processor; the processor is a Rockchip RK3588; the composite sensor array unit includes a microphone array, an environmental noise sensor, and an orientation sensor connected to the processor via I2C and SPI interfaces; the environmental noise sensor includes a MEMS accelerometer and an ultrasonic sensor; the adaptive beamforming processing unit is connected to the composite sensor array unit via a parallel data bus to transmit noise spectrum and spatial distribution data in real time, and outputs the noise-resistant voice signal to the dynamic noise robust data processing module via a USB 3.0 interface.

3. The AR-assisted management system for assignments based on voice AI interaction as described in claim 2, characterized in that: The microphone array adopts a multi-channel distributed layout, which enhances the operator's voice and suppresses surrounding interference through spatial filtering. Its core model is as follows: Where y(t) represents the output signal, x m (t) represents the signal received by the m-th microphone, τ m The time delay of this channel is M, and the number of microphones is M; the MEMS accelerometer in the environmental noise sensor extracts the spectral characteristics of the vibration signal through Fourier transform, as shown in the formula: Where X(f) represents the frequency domain signal, x(t) is the time domain vibration acceleration signal, f is the frequency, and j is the imaginary unit; the MEMS signal reflects the amplitude-frequency relationship of the mechanical structure noise source; the ultrasonic sensor obtains the direction information of the sound source through ranging and sound intensity localization methods, and its basic localization model is the sound wave propagation time difference method: Where θ represents the angle between the sound source and the front of the array, c is the speed of sound, Δt is the time difference between the left and right channels, and d is the distance between the two sensors.

4. The AR-assisted management system for assignments based on voice AI interaction as described in claim 3, characterized in that: The adaptive beamforming processing unit collaborates with the composite sensor array unit to construct a dynamic noise field modeling and target sound source localization mechanism by receiving the noise spectrum characteristics and spatial distribution information output in real time from the environmental noise sensor and the orientation sensor. It also dynamically calculates the weighting coefficients of the microphone array based on the minimum variance distortionless response algorithm, thereby forming a high-gain pickup beam pointing towards the operator's mouth. The core optimization objective is: Where w represents the microphone array weighting vector, R is the noise covariance matrix, and d is the steering vector in the target direction. H Let w be the conjugate transpose; the analytical solution to this optimization problem is: Among them, w opt The optimal weight vector determines the contribution weight of each microphone channel to the final output signal. The oral cavity spatial coordinates provided by the orientation sensor are used to construct the steering vector d in real time, while the spectral data sensed by the environmental noise sensor participates in constructing the covariance matrix R, thereby enabling the beamforming process to respond to the current sound field state. When the environmental noise sensor detects a strong noise event from a specific direction, the adaptive beamforming processing unit adjusts the array weighting strategy to form a main lobe between 30° and 60° in front, while simultaneously forming a strongly suppressed side lobe in the rear, thereby enhancing the target speech and suppressing the noise direction signal. Finally, the processed high signal-to-noise ratio speech signal is input in real time to the dynamic noise robust data processing module for subsequent semantic analysis and task instruction generation.

5. The AR-assisted management system for assignments based on voice AI interaction as described in claim 1, characterized in that: The dynamic noise robustness data processing module includes a real-time noise modeling submodule, a recognition-noise reduction joint optimization submodule, a data integration submodule, an analysis and decision-making submodule, an instruction generation submodule, a semantic understanding unit, and a speech recognition unit; The real-time noise modeling submodule works in conjunction with the environment-aware voice interaction module to dynamically model and suppress complex background noise in industrial settings. Its core mechanism is based on an improved Hidden Markov Model (HMM) for noise state transition modeling and parameter estimation, integrating deep neural networks and long short-term memory (LSTM) network structures. The HMM models the characteristics of background noise changing over time using a state transition probability matrix and an observation probability function. The model form is as follows: Where P(O|λ) represents the observation sequence probability under model parameter λ, Q is the hidden state sequence, O is the observed noise feature sequence, P(Q|λ) is the state transition probability, and P(O|Q,λ) is the observation probability under a given state sequence. Combined with the Baum-Welch algorithm, the transition and output probabilities of the noise state can be updated online, achieving dynamic estimation of the noise power spectral density. This estimation result, along with the noisy speech signal acquired by the current microphone array, is input into a DNN-LSTM hybrid model. This DNN-LSTM hybrid model first extracts static spectral features using a deep neural network, then models the contextual relevance in the time series using a long short-term memory network. Its output is the estimated clean speech spectrum. The overall prediction process can be represented as: Where Y(f,t) represents the noisy speech spectrum, and N(f,t) represents the real-time noise spectrum characteristics. The model is the predicted clean speech spectrum; during training, the goal is to minimize the reconstruction error, and the mean squared error loss function is used for optimization. Where S(f,t) is the actual speech spectrum. For the model output, T and F are the number of time frames and the number of frequency points, respectively.

6. The AR-assisted management system for assignments based on voice AI interaction as described in claim 5, characterized in that: The recognition-denoising joint optimization submodule is a core component of the dynamic noise robust data processing module. Together with the real-time noise modeling submodule and the semantic understanding unit, it forms a closed-loop feedback path. It drives the adaptive optimization of the denoising and recognition models through a confidence evaluation mechanism of the speech recognition results. The recognition-denoising joint optimization submodule generates confidence scores for the recognition results based on a connection-time classification loss function. These scores quantify the degree of matching between the output text of the speech recognition unit and the target label. Its core loss function is: in, Let x represent the connection-time classification loss, y represent the input speech feature sequence, π represent the target text label sequence, and B represent all possible paths. -1 (y) represents the set of paths that can be mapped from the target sequence, and P(π|x) is the probability of a path given the input. The recognition-denoising joint optimization submodule calculates the confidence score in real time and compares it with a set threshold. When the score is lower than 80%, the parameter adjustment strategy of the denoising algorithm is triggered. The spectrum adjustment process is expressed as follows: in, Y(f,t) represents the corrected speech spectrum, and Y(f,t) represents the original noisy spectrum. To estimate the noise spectrum, α is the attenuation coefficient, and ∈ is the minimum threshold to avoid negative values. After completing error correction, the recognition-denoising joint optimization submodule combines the corrected text with the original speech to form training samples, and updates the parameters of the recognition model and denoising network in real time through backpropagation. The overall AR intelligent auxiliary management system forms an online learning closed loop of "collection-denoising-recognition-feedback-optimization," with the optimization process aiming to minimize the joint loss function, expressed as: L total =λ1L CTC +λ2L denoise Among them, L total For the joint loss function, L total To identify the loss, L denoise For noise reduction loss, λ1 and λ2 are adjustable weighting coefficients; The semantic understanding unit is connected to the recognition-denoising joint optimization submodule to perform semantic analysis on the optimized text and determine the specific content and intent of the work instructions. The data integration submodule retrieves cargo-related information from the database module, including the cargo's storage location and inbound or outbound process. The analysis and decision-making submodule analyzes and processes the integrated data and determines the optimal warehousing path and operation steps based on a preset work optimization algorithm. The instruction generation submodule generates corresponding feedback information and control instructions based on the results of the analysis and decision-making submodule.

7. The AR-assisted management system for assignments based on voice AI interaction as described in claim 6, characterized in that: The AR multimodal interaction compensation module includes an AR display auxiliary confirmation unit and a tactile feedback error correction unit; the AR display auxiliary confirmation unit is embedded in the AR glasses worn by the operator and maintains a real-time communication connection with the dynamic noise robust data processing module; the processor of the AR display auxiliary confirmation unit is an ARM Cortex-A53 processor, which interacts with the dynamic noise robust data processing module through the MIPI_DSI interface to display data, and the gesture correction signal is transmitted back through the SPI interface; Once the speech recognition unit recognizes the user's command text, it will generate a semi-transparent floating window in the AR glasses' field of view to display the current recognition result and multiple candidate words with high confidence, thus avoiding misjudgment of the operation command due to a single speech recognition error. The floating windows are arranged side-by-side or in a sliding list format, allowing for quick confirmation.

8. The AR-assisted management system for assignments based on voice AI interaction as described in claim 7, characterized in that: The tactile feedback error correction unit is integrated into the AR glasses or smart gloves worn by the operator. It connects in real-time with the dynamic noise robust data processing module to construct an instant perception and intervention mechanism for industrial work sites. The tactile feedback error correction unit uses an STM32H7 series processor, receives device status conflict signals via the GPIO interface, and provides early warnings through vibrations at different frequencies. Simultaneously, it interacts with the dynamic noise robust data processing module via the UART interface to complete control command interactions. When the tactile feedback error correction unit detects that a voice command may cause risk or misoperation, it sends an early warning signal to the operator through physical vibration, guiding them to actively check whether their current operational intention matches the actual state. When the data processing module receives the voice recognition result and performs semantic understanding, it automatically retrieves device status information from the database module and compares it with the command. If a conflict is found, the tactile feedback mechanism is immediately triggered.

9. The AR-assisted management system for assignments based on voice AI interaction as described in claim 1, characterized in that: The operation management module also includes a task scheduling unit, an equipment control unit, and a processor. The task scheduling unit controls the forklift to assist operators in handling goods according to control commands sent by the dynamic noise robustness data processing module. The equipment control unit generates specific equipment control signals to control the forklift to travel along a designated path and accurately transport goods to the inbound or outbound location. The processor of the operation management module uses an industrial-grade ARM Cortex-A72 and interacts with the dynamic noise robustness data processing module via Ethernet using the Modbus TCP protocol to exchange control commands. When the processor of the operation management module is connected to production equipment or warehousing equipment, it adapts to industrial sensors through an RS-485 interface and controls the warehousing equipment using the CANopen protocol. The status monitoring data of the warehousing equipment is transmitted to the dynamic noise robustness data processing module via WiFi through the MQTT protocol, forming a closed-loop management of the operation process.

10. The AR-assisted management system for assignments based on voice AI interaction as described in claim 1, characterized in that: The database module contains a scenario-based noise database that collects noise from typical industrial scenarios, including various noise types and covering a sound pressure level range of 75-115 dB. The typical industrial scenario noise includes at least lathe noise, warehouse rack handling noise, and pneumatic tool noise.

Images