Task action generation method, device, equipment and medium

Abstract

The present application relates to the technical field of artificial intelligence, which can be applied to business scenarios such as financial technology and medical health, and discloses a task action generation method, device, equipment and medium, comprising: acquiring and preprocessing multi-source perception data to generate preprocessed perception data; extracting a plurality of single-modal feature vectors and fusing to generate a fused perception feature vector; determining a perception state based on the fused perception feature vector and generating a training action using a perception strategy network; executing the training action and receiving environmental feedback, generating experience sample data and storing it in an experience replay buffer; sampling training sample data and updating perception strategy network parameters; and generating a task action through the updated perception strategy network. The present application fuses multi-source perception data and introduces a reinforcement learning mechanism, so that the perception strategy network is continuously optimized in interaction with the environment. The network stability is improved through experience replay sampling, and finally the adaptability of the system to the dynamic environment and the perception accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a task action generation method and device, equipment and a storage medium. BACKGROUND

[0002] In current environmental perception enhancement research, traditional methods usually rely on fixed sensor data processing flow and static feature extraction model, lacking adaptability to dynamic changes in the environment. Especially in complex and rapidly changing environments, the perception system often fails to accurately reflect the real environment state, leading to misjudgment or response delay in the decision-making process of intelligent systems. Such problems are particularly prominent in application scenarios involving high reliability and high precision requirements.

[0003] In the medical and health business field, perception systems are widely used in surgical robots, rehabilitation assistance devices and other applications. Such systems place extremely high requirements on the real-time and accuracy of environmental perception. However, existing solutions mostly use preset parameter-driven perception processes, lacking the ability to adjust strategies in real time for dynamic changes during surgery (such as tissue morphology changes, light interference, occlusion, etc.), and are prone to perception deviations in unexpected situations or complex scenarios, thereby affecting operation precision and safety.

[0004] In the financial technology business field, environmental perception capabilities are gradually introduced into counter intelligent service terminals, remote interaction risk control devices, identity behavior recognition systems and other systems. These systems usually need to judge customer intentions or operation states in real time based on video, audio, behavior action and other multi-source perception data. However, traditional techniques are difficult to extract high-quality feature representations from unstructured perception data, and lack a mechanism to adjust perception strategies according to real-time feedback, making it prone to misidentification, delayed response and other problems, thereby affecting risk control judgment or customer experience. SUMMARY

[0005] The main purpose of the present application is to provide a task action generation method, device, equipment and storage medium, aiming to solve the technical problems that the prior art lacks a mechanism to dynamically optimize perception strategies based on environmental interaction, and is difficult to achieve efficient fusion and adaptive enhancement of multi-source perception data.

[0006] To achieve the above-mentioned purpose, the present application provides a task action generation method, comprising:

[0007] Obtain multi-source perception data and preprocess the multi-source perception data to generate preprocessed perception data;

[0008] Extract multiple single-modal feature vectors from the preprocessed perception data, and fuse the multiple single-modal feature vectors to generate a fused perception feature vector;

[0009] determine a perception state based on the fused perception feature vector, and generate a training action according to the perception state by using a perception policy network;

[0010] execute the training action and receive environment feedback information, generate experience sample data based on the training action and the environment feedback information, and store the experience sample data into an experience replay buffer;

[0011] sample training sample data from the experience replay buffer, and iteratively update network parameters of the perception policy network according to the training sample data;

[0012] process a target environment perception task by using the updated perception policy network, and generate a target task action.

[0013] Further, to achieve the above object, the present application provides a task action generation device, comprising:

[0014] a perception data preprocessing module configured to acquire multi-source perception data, and preprocess the multi-source perception data to generate preprocessed perception data;

[0015] a feature fusion extraction module configured to extract a plurality of single-modal feature vectors from the preprocessed perception data, and fuse the plurality of single-modal feature vectors to generate a fused perception feature vector;

[0016] a perception policy generation module configured to determine a perception state based on the fused perception feature vector, and generate a training action according to the perception state by using a perception policy network;

[0017] an experience sample generation module configured to execute the training action and receive environment feedback information, generate experience sample data based on the training action and the environment feedback information, and store the experience sample data into an experience replay buffer;

[0018] a policy network updating module configured to sample training sample data from the experience replay buffer, and iteratively update network parameters of the perception policy network according to the training sample data;

[0019] a task action inference module configured to process a target environment perception task by using the updated perception policy network, and generate a target task action.

[0020] Further, to achieve the above object, the present application further provides a computer device, which comprises a memory, a processor, and a task action generation program stored in the memory and executable on the processor, and the task action generation program, when executed by the processor, implements the steps of the task action generation method as described above.

[0021] Further, to achieve the above object, the present application also provides a computer readable storage medium, wherein the storage medium stores a task action generation program, and the task action generation program, when executed by a processor, implements the steps of the task action generation method.

[0022] Beneficial effects: The present application relates to the field of artificial intelligence technology, and can be applied to business scenarios such as financial technology and medical health, and discloses a task action generation method, device, equipment and medium, which comprises the following steps: acquiring multi-source perception data and preprocessing to generate preprocessed perception data; extracting a plurality of single-modal feature vectors from the preprocessed perception data and fusing to generate a fused perception feature vector; determining a perception state based on the fused perception feature vector and generating a training action by using a perception strategy network; executing the training action and receiving environmental feedback information, generating experience sample data and storing it in an experience replay buffer; sampling training sample data from the experience replay buffer and iteratively updating network parameters of the perception strategy network; and processing a target environmental perception task by using the updated perception strategy network to generate a target task action. The present application constructs a feature expression mode of fused multi-source perception data and introduces a reinforcement learning mechanism, so that the perception strategy network can be continuously self-optimized in interaction with the environment. The experience replay mechanism is used to sample training data, which improves the generalization ability and stability of the network. Finally, the task action generation is realized by using the updated perception strategy network, which improves the adaptability of the system to the dynamic environment and the accuracy of the perception decision. BRIEF DESCRIPTION OF DRAWINGS

[0023] The present application will be further described below in conjunction with the accompanying drawings and embodiments. In the drawings:

[0024] Figure 1 An application environment diagram of the task action generation method in an embodiment of the present application;

[0025] Figure 2 A flow diagram of the task action generation method in an embodiment of the present application;

[0026] Figure 3 A functional module diagram of the task action generation device in a preferred embodiment of the present application;

[0027] Figure 4 A structure diagram of a computer device in an embodiment of the present application;

[0028] Figure 5 Another structure diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0029] It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0030] The task action generation method provided in this embodiment of the invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can acquire multi-source sensing data from the user terminal and preprocess it to generate preprocessed sensing data; extract and fuse multiple single-modal feature vectors from the preprocessed sensing data to generate fused sensing feature vectors; determine the sensing state based on the fused sensing feature vectors and generate training actions using a sensing policy network; execute the training actions and receive environmental feedback information, generating experience sample data and storing it in an experience replay buffer; sample training sample data from the experience replay buffer and iteratively update the network parameters of the sensing policy network; process the target environment sensing task through the updated sensing policy network and generate the target task action. This invention constructs a feature representation method that fuses multi-source sensing data and introduces a reinforcement learning mechanism, enabling the sensing policy network to continuously self-optimize in interaction with the environment. The experience replay mechanism samples training data, improving the network's generalization ability and stability. Finally, the updated sensing policy network generates task actions, improving the system's adaptability to dynamic environments and the accuracy of sensing decisions. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0031] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the task action generation method provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0032] like Figure 2 As shown, the task action generation method proposed in this invention includes the following steps:

[0033] S10, acquire multi-source sensing data, and preprocess the multi-source sensing data to generate preprocessed sensing data;

[0034] In this embodiment, the data input stage of the perception system collects different modal information from the external environment through multiple heterogeneous sensors, including image, audio, distance, and environmental parameters, etc. The acquisition of visual data is usually completed by cameras, covering infrared visible light, depth map, and other image types, which can reflect the scene structure, object contour, motion trajectory, and other information. Audio data is collected in real time by a microphone array, which can represent speech, mechanical noise or abnormal audio events in the environment. Distance information is collected by devices such as laser radar, ultrasonic sensor or millimeter wave radar, which represents the relative position or three-dimensional structure distribution of obstacles in space. Environmental parameters include temperature, humidity, air pressure, air quality and other variables, which can be measured by gas sensors, temperature sensors and other devices, and are used to evaluate the overall fluctuation trend of the environment.

[0035] During the data synchronization collection process, the clock offset problem of various sensors needs to be corrected, and the multi-source data is made time consistent through a unified timestamp system or hardware synchronization signal. In order to improve the quality of input data, the original data needs to be preprocessed such as denoising, calibration and normalization. Discrete noise points in visual images can be removed using the median filtering method, which preserves the edge structure while smoothing local pixel interference. Background noise or multi-source overlapping sound in audio data can be denoised based on spectral subtraction, wavelet threshold or adaptive filter. Distance data may have jump values or nonlinear deviations due to reflection interference, which can be corrected using a calibration model based on scene modeling. The model parameters come from calibration scenes or experimental data. The normalization processing can be used to map the data to a unified numerical space by using the maximum and minimum value mapping or Z-score standardization, so that different dimension data can be mapped to a unified numerical space.

[0036] After completing the quality improvement of multi-source data, spatial and temporal alignment operations are needed. Spatial alignment refers to mapping each modal data to a unified spatial reference system to ensure the consistency of sensor observation points, such as geometric transformation of the camera and radar view angle to make their observation data superimposable. Time alignment refers to ensuring the consistency of different data modalities at the same time, which is usually achieved through interpolation or delay scheduling mechanisms. After the spatial and temporal alignment is completed, the multi-modal information is encoded and combined according to a unified data structure to form a multi-dimensional joint tensor containing image matrices, audio vectors, distance point clouds and environmental variable sequences, which are used as input for subsequent calculation and processing. This joint structure not only preserves the independent features of each modal data, but also supports the subsequent fusion network to extract its cross-correlation representation, thereby achieving efficient information integration capability.

[0037] In one implementation, RGB-D camera can be used to collect images and depth information to construct visual data stream. Microphone array is installed around the robot body to collect audio input in 360-degree space, and spectral features are extracted through short-time Fourier transform. Millimeter wave radar is deployed on the top platform to obtain distance field distribution within 5 meters in front. Digital sensors for collecting temperature and humidity upload environmental parameters through I2C interface at a fixed time, sampling once per second. Data synchronization of all sensors is realized through the main control board sending unified time stamps, and the frame rates of vision and audio are set to 30fps and 16kHz respectively. The pre-processing part runs image filtering model, audio noise reduction algorithm and distance data filter based on embedded GPU platform, and environmental parameters are normalized to floating-point values between 0 and 1. The fusion process combines the four types of pre-processed data through tensor concatenation operation, and connects the multi-channel input port of the feature extraction network.

[0038] In another implementation, multi-thread architecture can be used to process each type of modal data. Vision data performs convolution filtering in GPU threads, audio data performs frequency domain noise reduction in DSP threads, distance information is calibrated in real time by FPGA module, and environmental parameters are pre-processed by low-power MCU. After all modules complete processing, they are written into shared memory area through DMA, and the main program is synchronized and aligned to generate multi-source joint data structure. In addition, cloud edge collaborative architecture can also be used in some scenarios to upload some raw data to a remote server for enhanced processing, such as using deep learning model to perform super-resolution reconstruction on images, separate and enhance audio, and return to local use after processing.

[0039] Example: In the field of medical health, there are complex background noise and equipment operation interference in hospital environment, for example, voice input in emergency ward needs to extract instruction signal from monitoring audio, at this time, through audio signal preprocessing and visual image alignment, synchronous recognition of doctor's gesture instruction and voice instruction can be realized, and the interaction ability and response accuracy of perception system are improved.

[0040] In the financial business scene, self-service terminal equipment faces various sensor input interference, such as outdoor light change, user language dialect interference, and equipment environment temperature fluctuation, etc. Through pre-processing and fusion mechanism of multi-modal perception data, stable input stream construction can be realized, and more robust environmental understanding ability is provided for subsequent recognition of user identity or judgment of abnormal behavior.

[0041] The embodiment can effectively reduce the structural differences and redundant interference brought by various sensors, and improve the availability and representation quality of perception data by fine-grained preprocessing and unified encoding integration of multi-modal data. The processing mechanisms such as median filtering, frequency domain noise reduction and scene calibration can ensure the stability and accuracy of the input information, and avoid the negative impact of false data on subsequent strategy generation. Through the spatial and temporal alignment mechanism, the perception system can real-time master the structural consistency and dynamic change relationship among various modalities, and provide a clear data basis for subsequent perception state expression and action strategy reasoning.

[0042] S20, extracting a plurality of single-modal feature vectors from the preprocessed perception data, and fusing the plurality of single-modal feature vectors to generate a fused perception feature vector;

[0043] In the embodiment, after the multi-source perception data is preprocessed, representative feature vectors need to be extracted from each modal information, and a unified fused perception representation is formed through a cross-modal fusion mechanism to support the subsequent state determination and action generation process. The extraction of single-modal feature vectors is differentiated according to the structural characteristics and semantic expression methods of each kind of perception data. Visual information contains a large amount of spatial structure and texture details, which is suitable for using deep convolutional neural networks to extract local spatial features and edge structure distribution. In actual implementation, the visual data is first extracted by an improved convolution module to extract bottom-level perception features. The module includes multiple scale convolution kernels and cross-channel attention structures for capturing spatial position and semantic correlation, and then extracts long-term dependency and global region response through a self-attention mechanism, and finally forms a visual feature vector with complete semantic expression.

[0044] Audio perception data has obvious feature change trend in time sequence dimension. First, a time-frequency graph structure is constructed to represent the evolution process in time and frequency domain. Through acoustic feature extraction methods based on Mel frequency cepstral coefficient (MFCC) or log power spectrum, the basic frequency domain features of audio are generated, and further input into time sequence modeling module such as gated recurrent unit (GRU) or one-dimensional convolution-Transformer hybrid network to capture short-term and long-term sound change patterns, and output audio feature vector expressing instruction or event information.

[0045] Distance perception data is usually represented as sparse point cloud or distance matrix. Through Fourier transform, discrete wavelet transform or polar coordinate projection operation on the original data, it is converted into a frequency domain feature map that is easier to represent the relative structure of the target. Combined with spatial feature extraction network such as PointNet, three-dimensional convolution network or graph convolution structure, the spatial shape pattern and obstacle distribution features are extracted, and the distance feature vector is formed through embedding compression.

[0046] The environmental parameter data is generally a time series composed of multiple continuous numerical streams. Different parameters need to be normalized and input into a shallow full connection network or a time series convolution module after being unified in scale, so that they can represent the trend of small perturbations or the sudden change of relevance, and finally obtain the environmental parameter feature vector.

[0047] In the fusion operation, a fusion network is introduced to cross-modal modeling of the four types of feature vectors. The network includes a multi-head self-attention mechanism for capturing interdependent relationships and joint response patterns between modalities. During the fusion process, each modal feature vector is input into the attention module as a query, key, and value to calculate the attention distribution and weight the aggregated feature content to generate a context-enhanced fusion representation. The final output fusion perception feature vector is a joint representation with consistent dimensions, which can be input into the subsequent perception state determination module.

[0048] In one implementation, an improved ResNet network is used to extract features from RGB images. An SE attention module is added at the front end of the network to enhance the channel perception ability, and a visual feature vector with a size of 256 is output. The audio input is processed through a CNN-BiGRU network with convolution stacking. The front-end CNN is used for local time-frequency structure extraction, and the back-end bidirectional GRU is used for modeling time-dependent relationships. The output dimension of the audio feature vector is 128. The distance data is input after three-dimensional voxelization and given to the 3D convolution network for processing to generate a 64-dimensional distance feature vector. The environmental parameters are modeled through a three-layer fully connected network to output a 32-dimensional environmental feature vector. The above four feature vectors are associated through a multi-head self-attention mechanism in the fusion network, and a residual connection is used to enhance the semantic alignment capability. Finally, a 384-dimensional fusion perception feature vector is output.

[0049] In another way, each type of feature can be modeled based on the Transformer encoder architecture. First, all feature vectors are concatenated into a multi-channel feature sequence, and position encoding is constructed and input into a multi-layer encoder stack. Through cross-modal attention mechanism, the information between each modality is updated and guided by each other. The fusion result is output by the final layer of the encoder as the fusion perception feature vector, which is used for subsequent state expression.

[0050] Example: In the medical health scene, the surgical robot needs to perceive the doctor's voice command, the image information of the operating field, and the environmental equipment state at the same time. By extracting the command vector from the audio, extracting the tissue structure features from the image, and obtaining the operating room temperature and equipment load data from the environmental parameters, and fusing these information, the current operation state can be accurately expressed, providing a complete basis for the next action generation.

[0051] In the financial service device, the ATM or VTM machine needs to integrate user voice, camera picture and environmental safety index, generate unified scene representation through fusion, judge user identity authenticity and transaction security level, and effectively reduce operation risk and fraud occurrence rate.

[0052] The embodiment can retain the characteristic information of various modal data to the greatest extent by designing feature extraction structures for the various modal data respectively, and avoid distortion of modal information in the preprocessing or mapping process. The fusion process realizes dynamic weighted calculation between multi-modal features by introducing an attention mechanism, which not only improves the global consistency of the perception expression, but also enhances the robustness of the system to noise, abnormal data and dynamic environment, so that the finally generated fusion perception feature vector has rich semantic support and strong generalization ability.

[0053] S30, determining a perception state based on the fusion perception feature vector, and generating a training action according to the perception state by using a perception strategy network;

[0054] In the embodiment, after the fusion perception feature vector is generated, it needs to be further mapped and structurally transformed to construct a perception state, which is used as the input of the perception strategy network to generate a training action. The determination process of the perception state needs to solve the problems of inconsistent perception representation dimensions, complex semantic distribution and mixed space-time features. Therefore, first, the fusion perception feature vector is subjected to state space definition processing to construct a perception state space, i.e., to determine a feature structure for representing the environmental state. The construction of the state space is usually based on high-dimensional feature compression and semantic reduction methods, for example, using a multi-layer perceptron network to project the fusion perception feature vector to a low-dimensional embedding space, and using batch normalization, nonlinear activation and other operations to improve the stability and expressiveness of the state vector.

[0055] After the state space is constructed, the information in the current environment needs to be mapped to the state representation in the state space. This process directly uses the fusion perception feature vector to determine the current perception state in the state space in combination with the feature input of the perception time point. The perception state is an abstract expression of the current time environmental perception data, which has stability, distinguishability and predictability, and can be used as the input basis for the subsequent action decision process.

[0056] To support the decision-making of behaviors in complex perception states, a perception action space is defined as the output set. The perception action space consists of three parts, including sensor parameter adjustment actions, working mode switching actions, and data acquisition frequency adjustment actions. The sensor parameter adjustment actions include adjusting the camera focal length, modifying the microphone gain, and adjusting the radar scanning range, aiming to dynamically adapt the collection accuracy and range of the perception device; the working mode switching actions include switching the infrared imaging mode and the laser ranging mode, which are used to cope with different environmental characteristics or task requirements; the data acquisition frequency adjustment actions include increasing the high-definition video acquisition frame rate and reducing the audio sampling frequency, which are used to balance the real-time performance and resource consumption.

[0057] The perception policy network takes the current perception state as input and outputs the probability distribution of the perception action. The structure of the perception policy network includes multiple deep neural network layers and attention mechanism layers. The former is used for nonlinear modeling of perception state features, and the latter is used for focusing on key factors in the state. After the policy network outputs the probability distribution of the action, a preset number of actions with high probability values are selected as the candidate action set. To ensure the action executability and physical rationality, it is necessary to further verify whether the candidate actions are within the perception action space and meet the physical parameter constraints of the sensor device and the environmental safety requirements, and to filter and generate an effective candidate action set.

[0058] For each effective candidate action, the expected reward value that can be obtained in the current state is estimated. The expected reward value is output by the policy network or value network evaluation submodule, which is used to measure the long-term value of the candidate action. Finally, the action with the highest reward expectation value is selected as the training action, which is input to the execution module to produce environmental feedback and accumulate experience data.

[0059] In one implementation, the fused perception feature vector is first input into a three-layer fully connected network, which compresses its dimension from 512 to 64 to form a perception state representation. The perception policy network consists of two layers of Transformer encoder, which can handle the context characteristics and semantic dependencies of the input state. After outputting the action probability distribution, the Top-5 actions are selected as the candidate set according to the probability values. Each candidate action is matched with the action space template to confirm whether it can be supported by the sensor hardware and to verify whether it meets the energy consumption and latency constraints of the perception module. Then, the value estimation network is called to calculate the expected reward value of each candidate action, and finally the action corresponding to the maximum value is selected as the training action.

[0060] In another implementation, the perception state space is modeled using a variational autoencoder, which maps the fused perception feature vector to a Gaussian distribution of hidden variables, and then samples to generate a state vector input to the policy network. The perception action space uses a configurable template definition method to flexibly expand the action types and execution constraints according to the task scenario, improving the adaptability of the training action selection strategy.

[0061] Example: In the medical health scene, after the surgical auxiliary robot receives the environmental perception data and generates the fusion perception feature vector, it can judge the current surgical field visibility or audio instruction recognition confidence through perception state modeling, and adjust the training actions such as camera focal length or audio gain according to the state selection to improve the perception quality and improve the safety of the operation.

[0062] In the automatic customer service terminal in the financial field, the system identifies the decrease of voice recognition accuracy in a high-noise environment through state analysis, and automatically generates training actions to switch to an instruction confirmation prompt mode, thereby reducing the misrecognition rate and optimizing the service experience.

[0063] The embodiment can significantly enhance the response ability and dynamic adjustment ability of the system to complex environmental changes by constructing a mapping mechanism from the fusion perception feature vector to the perception state, and combining with the action space design containing different types of perception operations. The strategy network generates training actions using perception state information, which strengthens the goal-orientedness and physical feasibility of action selection, and provides a high-quality exploration basis for subsequent reinforcement learning process.

[0064] S40, executing the training action and receiving environmental feedback information, generating experience sample data based on the training action and environmental feedback information, and storing the experience sample data into an experience replay buffer;

[0065] In the embodiment, after the training action is generated, it needs to be actually applied in the environment to complete the interactive learning process and collect feedback information. The training action is output by the perception strategy network according to the current perception state, representing the perception behavior expected to obtain a higher reward in the state. When the training action is executed, the system needs to dispatch the corresponding sensor control instructions, adjust the parameter configuration or switch the working mode, etc. to ensure that the training action has feasibility in the actual environment. The execution process may include focal length adjustment command issuing, audio channel gain adjustment signal transmission or radar scanning angle resetting, etc. specific action execution path.

[0066] After the execution of the training action, the system synchronously collects the environmental feedback information, which includes two types of core data: immediate reward signal and state transition information. The immediate reward signal is used to measure the performance of the training action in the current state, which is usually derived from the set external evaluation function, such as image clarity change, target recognition confidence improvement or task response time shortening, etc.; the state transition information describes the change of the environment state after the execution of the training action, which is usually represented as the change trend of the perception state or the fusion perception feature vector in the next time.

[0067] While collecting the environment feedback information, the current perception state before the execution of the training action needs to be recorded to ensure the completeness and traceability of the data. Subsequently, the system assembles the current perception state, the training action, the immediate reward signal, and the state transition information into an experience sample data. Each experience sample data should have completeness, standardization, and time sequence consistency. In order to ensure the stability of the training process, the generated experience sample data also needs to be checked for completeness, including format checking, value range checking, and timestamp consistency checking.

[0068] The experience sample data that passes the verification will be written into the experience replay buffer, which supports random reading and batch sampling operations for subsequent policy network training. The experience replay buffer is usually implemented in a ring structure or a priority queue structure, and can set a maximum capacity limit and update strategy, such as first-in-first-out, priority sampling, or reward gradient-based dynamic replacement strategy, to improve sample utilization efficiency and avoid sample bias.

[0069] In one implementation, the training action execution module sends the action command to the device control layer through the interface and listens to the data returned by the sensor to obtain the feedback information in real time. The immediate reward signal is evaluated by the online reward calculation module, such as the percentage of mAP improvement of image target detection. The state transition information is calculated by the forward network to obtain the next perception state vector and compared with the current perception state. The current perception state, training action, immediate reward signal, and state transition information are packaged into a four-tuple and added to the experience sample data pool. Each data is attached with a unique identifier and timestamp and enters the experience replay buffer. The experience replay buffer uses a priority-based replacement mechanism to replace and update the samples that have not been sampled for a long time or have a reward value below a threshold.

[0070] In another implementation, the feedback information is returned by the environment response system, including the visibility score based on image content, the environment change coefficient, and the sensor state code. The system defines rules to preprocess the state code and filter out illegal samples, and only keeps the stable feedback data pairs. The experience sample data is organized in a sequence structure to facilitate time series modeling or multi-step policy expansion in the training process.

[0071] Example: In the medical health scene, the surgical navigation system executes the training action, such as automatically adjusting the camera position, and the environment feedback information may include indicators such as reduced image occlusion or improved tissue recognition accuracy as the immediate reward signal, while recording the current visual perception state and the next image feature to form the state transition information, generating experience samples for subsequent learning.

[0072] In the intelligent customer service device in the financial field, after the system performs a training action, such as adjusting the voice recognition sensitivity, a clarity score and a semantic matching result of a user's answer can be received as a reward and a state transition basis to form structured experience data input to a learning module.

[0073] By performing a training action and receiving feedback information containing an immediate reward and a state transition, the embodiment can build a mapping relationship between perception behavior and reward in an actual environment, and then accumulate high-quality experience samples that can be used for policy training. Storing these experience samples in an experience replay buffer can support the reuse of historical experience by the policy network in different time windows, thereby improving training efficiency and policy stability.

[0074] S50, sampling training sample data from the experience replay buffer, and iteratively updating network parameters of the perception policy network according to the training sample data;

[0075] In the embodiment, during the training process, data samples for optimizing the perception policy network need to be obtained from the experience replay buffer. The sampling operation generally uses a random sampling mechanism to extract representative training sample data from the buffer to prevent training bias caused by continuous data in time and to enhance the generalization ability of the policy network to changes in the environment state. The composition structure of the training sample data is fixed, including four parts: the current perception state, the training action, the immediate reward signal, and the state transition information.

[0076] Each training sample data is first parsed to extract the perception state and the training action, which are used to input the current policy network and obtain the action prediction value in that state. Then, the state transition information is extracted from the same sample to calculate the target action value, i.e., the optimal policy feedback that the state should obtain, through a state transition mapping function or a target network. The difference between the action prediction value and the target action value is quantified as an error, which is used to evaluate the deviation of the current policy network parameters.

[0077] The error value is input into a loss function module for processing. The loss function can take different forms such as mean square error, time difference loss, and policy gradient loss, depending on the structure of the reinforcement learning algorithm. The loss result will be fed back to the perception policy network, and the trainable parameters in the network will be updated by gradient through the backpropagation algorithm. Gradient update can use a fixed learning rate or an adaptive learning rate strategy such as Adam, RMSprop, etc. optimizer, and dynamically adjust according to the training round, loss convergence trend.

[0078] The entire update process is performed iteratively, usually with a fixed training batch or a convergence threshold as the termination condition. Each iteration fine-tunes the network parameters, gradually reducing the difference between the action prediction and the target value, making the policy network more stable and improving the adaptability and effectiveness of the perception action.

[0079] In an embodiment, the training module randomly samples 64 training sample data from the experience replay buffer each round, extracts the perception state and training action in each data, and obtains the action prediction value through forward propagation. The target action value is calculated by the target network, combined with the reward signal and state transition information for time difference processing. The mean square error loss function is used to calculate the error between the prediction value and the target value, and the Adam optimizer is used for parameter update. After completing a batch of training, the current loss value and network weight state are recorded for convergence evaluation.

[0080] In another embodiment, the system uses a priority experience replay mechanism, assigns different sampling weights according to the reward intensity and usage frequency of each sample, thereby increasing the sampling probability of high-value samples. After batch processing of training samples, L2 regularization is used to limit the risk of parameter overfitting, and Dropout strategy is added in the gradient update process to enhance the generalization ability of the policy network. The iteration period is related to the task complexity, and is usually configured for hundreds of rounds.

[0081] Example: In medical health applications, the surgical assistance system generates training sample data by sampling previous operation, updates the policy network, and improves the recognition ability of target tissue characteristics and the accuracy of surgical path adjustment.

[0082] In the risk identification system in the financial field, sample sample data containing voice response accuracy and context understanding feedback, after iterative training of the policy network, it can improve the ability of keyword recognition and response optimization in voice interaction, enhance the intelligent level of service and customer experience satisfaction.

[0083] This embodiment can continuously optimize the policy behavior on the basis of preserving historical experience information by sampling training sample data from the experience replay buffer and performing parameter iterative update on the perception policy network, so that the network can more accurately map the relationship between the perception state and the optimal perception action, and improve the dynamic response ability and adaptability of environmental perception decision.

[0084] S60, the target environment perception task is processed by the updated perception policy network to generate the target task action.

[0085] In this embodiment, after the training is completed, in order to realize the actual deployment of the environment perception capability and the output of the action decision, the perception strategy network with the completed parameter update needs to be used to process the environment perception task and output the task action. First, the system loads the perception strategy network model optimized through iteration, which has the ability to intelligently respond to dynamic environment changes. Then, the system receives a task configuration of an environment perception task, which includes scene information, response requirements, device constraints, time range, and other parameter sets for guiding strategy execution. The task configuration provides boundary conditions and strategy constraints for subsequent action generation.

[0086] Next, the system collects multi-source perception data of the current environment according to the current environment state, which is consistent with the data source in the training stage, including but not limited to visual, audio, distance, and environmental parameter data of various sensor channels. In order to ensure the consistency and availability of the data, the collected data will be subjected to the same preprocessing process as in the training stage to generate current preprocessed perception data, ensuring that the data distribution input into the perception strategy network remains consistent, thereby improving the action prediction accuracy.

[0087] Subsequently, various single-modal feature vectors are extracted from the current preprocessed perception data and fused into a current fused perception feature vector. The extraction method is consistent with the training stage to ensure the consistency of the feature space. The current fused perception feature vector is mapped to the state space to convert it into a current task perception state. The task perception state is input into the perception strategy network, and after forward propagation, a probability distribution of a set of task actions is output.

[0088] Finally, the system selects the action with the highest probability value that meets the task constraint conditions as the final task action for execution according to the task configuration and the output task action probability distribution, combined with the target requirements and physical executable conditions of the task. The task action can directly control the sensor behavior, adjust the device parameters, trigger the working mode switching, or perform the path planning operation, thereby completing the dynamic response of the environment perception task.

[0089] In one embodiment, the system receives a task configuration of a hospital surgery auxiliary task, which contains target tissue identification, operation precision range, available device constraints, and other information. After loading the updated perception strategy network, the system collects images, sounds, and environmental parameters of the current surgery area, generates preprocessed perception data through image filtering, audio noise reduction, and parameter calibration, and extracts visual, audio, spatial, and parameter vectors. After fusing the features, the perception state is generated and input into the perception strategy network. The task action output by the network includes adjusting the camera angle, modifying the lighting intensity, and activating the edge recognition algorithm. According to the task configuration, the action that meets the precision requirements and device constraints is selected as the task action for execution.

[0090] In another implementation, in a financial customer-oriented voice service scenario, after the system loads the trained perception strategy network, it collects the voice stream and background noise parameters in the current call environment, performs preprocessing to generate standardized perception data, extracts multi-modal features, and then generates the current perception state. The perception strategy network output includes a set of task actions such as adjusting speech recognition sensitivity, switching noise suppression mode, and reconstructing text summary. Combined with the requirements for interaction time and accuracy in the task configuration, the task actions are filtered to improve the customer interaction experience.

[0091] Example: In the medical health field, for the real-time auxiliary surgery scenario, to cope with the challenges of perception information mutation and dynamic adjustment of task demand in the surgical environment, the system uses the updated perception strategy network to process the current fusion perception feature vector of the surgical area, and generates task actions that match the task configuration. For example, when the system identifies changes in lighting conditions and blurred edges of target tissues during surgery, the updated perception strategy network generates task actions to adjust the camera angle and activate the edge enhancement module based on the new perception state, thereby improving perception clarity and positioning accuracy and ensuring stable progress of the surgical task.

[0092] In the field of financial technology business, in the face of unstructured input changes in voice interaction, such as changes in customer speech rate, background interference, and other factors affecting recognition accuracy, the system uses the updated perception strategy network to analyze the voice environment state reflected by the current fusion perception feature vector, and generates optimal task actions to adjust speech recognition parameters or switch noise reduction strategies. For example, when high-frequency noise interference is identified in the voice, the perception strategy network generates task actions to enable high-order noise suppression and semantic rollback modules to ensure semantic recognition accuracy and task execution continuity during the interaction process. The generation of such task actions directly reflects the ability to use strategy optimization in the context of environmental changes, thereby achieving precise processing of voice tasks and dynamic matching of strategies.

[0093] This embodiment can achieve dynamic response to complex and variable environmental states by applying the updated perception strategy network to the actual processing flow of environmental perception tasks, and generate optimal task actions according to the current task demand, thereby improving the response efficiency and task execution accuracy of environmental perception tasks.

[0094] The application relates to the technical field of artificial intelligence, can be applied to business scenes such as financial technology and medical health, and discloses a task action generation method, device, equipment and medium, which comprises the following steps: acquiring multi-source perception data and preprocessing to generate preprocessed perception data; extracting a plurality of single-modal feature vectors from the preprocessed perception data and fusing to generate a fused perception feature vector; determining a perception state based on the fused perception feature vector, generating a training action by using a perception strategy network; executing the training action and receiving environmental feedback information, generating experience sample data and storing the experience sample data in an experience replay buffer; sampling training sample data from the experience replay buffer, and iteratively updating network parameters of the perception strategy network; and processing a target environmental perception task by using the updated perception strategy network, and generating a target task action. The application constructs a feature expression mode of fused multi-source perception data and introduces a reinforcement learning mechanism, so that the perception strategy network can be continuously self-optimized in interaction with the environment. The experience replay mechanism is used to sample training data, so that the generalization ability and stability of the network are improved. Finally, the task action generation is realized by using the updated perception strategy network, and the adaptability of the system to a dynamic environment and the accuracy of perception decision are improved.

[0095] In one embodiment, the above step S10 comprises:

[0096] S101, synchronously collecting multi-source perception data including visual perception data, audio perception data, distance perception data and environmental parameter perception data by using a plurality of sensors;

[0097] S102, applying median filtering processing to the visual perception data to generate denoised visual perception data;

[0098] S103, applying a calibration model to the distance perception data and the environmental parameter perception data to generate calibrated distance perception data and calibrated environmental parameter perception data;

[0099] S104, performing noise reduction processing on the audio perception data to generate denoised audio perception data;

[0100] S105, performing space-time alignment on the denoised visual perception data, the denoised audio perception data, the calibrated distance perception data and the calibrated environmental parameter perception data to generate aligned multi-source perception data;

[0101] S106, combining the aligned multi-source perception data to generate preprocessed perception data.

[0102] In this embodiment, the operation of acquiring multi-source perception data is the basis for the execution of the perception task, and a synchronous collection mechanism oriented to heterogeneous perception channels needs to be established. This operation is usually deployed in an edge computing unit or an embedded controller, and through a unified clock reference and scheduling mechanism, the synchronous sampling of multiple heterogeneous sensors is coordinated. Among them, visual perception data mainly acquires two-dimensional or three-dimensional visual information through RGB cameras, infrared cameras or macro imaging devices; audio perception data captures voice, background noise or other sound source features in the environment through a microphone array; distance perception data is generated based on laser radar, ultrasonic radar or ToF (Time-of-Flight) depth camera, reflecting the spatial structure and target position; environmental parameter perception data includes temperature and humidity, air pressure, illumination, particulate matter concentration, etc., which are collected through environmental sensor modules. These perception data have diversity in data type, sampling rate and timing structure, so synchronous collection not only depends on the coordination of the physical sampling clock, but also needs to implement a buffer and timestamp mechanism at the software layer to eliminate data frame differences.

[0103] Median filtering processing is aimed at the salt and pepper noise and local pixel anomaly in visual perception data, and by defining the median value of pixels in a sliding window (such as 3x3 or 5x5) to replace the current pixel value, the extreme value interference is eliminated and the edge features are preserved. This processing method has certain spatial adaptability while maintaining the clarity of structural edges, and is suitable for scenes such as intraoperative images and traffic monitoring that have high requirements for image continuity and structural fidelity. This process can be executed in real time by an embedded image processing chip at the image acquisition end, or can be completed through GPU acceleration.

[0104] The calibration model processing of distance perception data and environmental parameter perception data is mainly used to compensate for the perception deviation caused by sensor drift, system error or external interference. Such calibration models establish static mapping or dynamic prediction mechanisms based on historical observation data and standard data, and commonly use linear regression, polynomial fitting, Kalman filtering, neural network mapping, etc. Calibration parameters can be obtained through offline training or online adaptive updating, and are suitable for data consistency enhancement under conditions of sensor array tolerance and multi-device deployment.

[0105] The noise reduction processing of audio perception data needs to deal with reverberation, background noise, overlapping of human voice, etc. The processing methods can include spectral subtraction, Wiener filtering, adaptive filtering, deep learning voiceprint enhancement model, etc. For the improvement of speech recognition accuracy in speech interaction or medical scenarios, an end-to-end noise reduction network based on attention mechanism or real-time filtering operation on audio stream is usually introduced, and a semantic model is combined to improve the speech understanding ability after noise reduction.

[0106] The processed multi-source perception data needs to be spatio-temporally aligned to ensure consistency of various data in spatial coordinates and time frames. Temporal alignment is completed through timestamp correction or interpolation compensation methods, such as a sliding window buffer mechanism based on synchronous sampling markers. Spatial alignment is performed through geometric mapping conversion based on sensor external parameter calibration parameters, such as through a coordinate transformation matrix of a camera and a radar to realize spatial registration of image and distance data. The alignment operation also needs to consider the problem of inconsistent data density caused by sampling frequency differences, which is often handled through interpolation, downsampling, and inter-frame padding.

[0107] After the above noise reduction, calibration, and alignment of multi-source data, different modal data is constructed into a unified tensor expression or vector set through a structured combination operation to form preprocessed perception data. This expression needs to consider the compatibility of data types and the efficiency of subsequent processing, and is often organized in the form of multi-channel images, nested vectors, or fusion feature matrices. The structure combination method needs to be compatible with the input interface of the subsequent perception module, and has scalability and online processing capability to meet the requirements of dynamic adjustment and real-time processing of tasks.

[0108] The embodiment realizes synchronous acquisition of multi-source perception data through a unified time reference and a heterogeneous sensor coordination mechanism, effectively avoiding the problem of inconsistent state description caused by asynchronous sensors. The median filter, model calibration, and multi-strategy noise reduction algorithm are introduced to greatly reduce the influence of environmental noise, system error, and device deviation while preserving key perception features. Time stamp calibration and external parameter transformation are used to complete the spatio-temporal alignment of perception data, ensuring the consistency of the fusion basis of different modal information. Finally, structured preprocessed perception data is generated, providing time-efficient and accurate input basis for subsequent feature extraction and strategy generation, thereby improving the adaptability of the perception system to complex dynamic environments and the accuracy of task response.

[0109] In one embodiment, the above step S20 includes:

[0110] S201, extracting visual perception data from the preprocessed perception data, and processing the visual perception data through a convolution network to generate visual basic features;

[0111] S202, processing the visual basic features through a self-attention mechanism to generate a visual feature vector;

[0112] S203, extracting audio perception data from the preprocessed perception data, and performing acoustic feature extraction processing on the audio perception data to generate audio basic features;

[0113] S204, processing the audio basic features through a time series analysis mechanism to generate an audio feature vector;

[0114] S205, distance perception data is extracted from the preprocessed perception data, and time-frequency conversion processing is performed on the distance perception data to generate distance frequency domain features;

[0115] S206, the distance frequency domain features are processed by a spatial feature extraction mechanism to generate a distance feature vector;

[0116] S207, environmental parameter perception data is extracted from the preprocessed perception data, and the environmental parameter perception data is normalized to generate normalized environmental parameters;

[0117] S208, the normalized environmental parameters are processed by a parameter feature extraction mechanism to generate an environmental parameter feature vector;

[0118] S209, the visual feature vector, the audio feature vector, the distance feature vector, and the environmental parameter feature vector are input into a fusion network, and a feature correlation weight is calculated by a multi-head self-attention mechanism of the fusion network;

[0119] S210, the visual feature vector, the audio feature vector, the distance feature vector, and the environmental parameter feature vector are fused according to the feature correlation weight to generate a fused perception feature vector.

[0120] In this embodiment, the operation of extracting visual perception data from preprocessed perception data is based on the structured data format collected by the sensor. The visual data part is separately taken out through the index channel dimension or the identification field, and then processed by using an improved convolution network. The improved convolution network can introduce residual connection, dilated convolution, or attention gate mechanism, etc., to enhance the modeling ability of complex regions such as edges and textures while maintaining the receptive field, and output visual basic features with consistent dimensions and spatial structure characteristics.

[0121] To further improve the recognition ability of key regions in visual data, a self-attention mechanism is introduced to process the visual basic features. The self-attention mechanism calculates the correlation matrix between different positions to strengthen the response of high semantic information regions, thereby generating a visual feature vector with stronger representation ability. This vector can be used as the input of the visual channel in the multi-modal fusion process.

[0122] After the audio perception data is extracted from the preprocessed perception data, acoustic feature extraction processing is first performed. This processing can include Mel frequency cepstral coefficient (MFCC), log power spectrum, or spectral centroid calculation, to quantify the frequency structure of the audio signal and obtain audio basic features. Then, a time series analysis mechanism is used for processing, which can be a recurrent neural network (such as bidirectional LSTM) or a one-dimensional convolution network, to capture the dynamic patterns evolving over time in the audio, and output a compact structured audio feature vector.

[0123] The distance perception data is extracted after preprocessing, and a time-frequency conversion method is used to generate frequency domain features. The conversion method can be short-time Fourier transform or continuous wavelet transform, the purpose is to convert time series data into representation on time-frequency plane, so as to reveal its energy distribution at different frequencies. Then the spatial feature extraction mechanism is used to process these frequency domain features, and three-dimensional convolution or graph convolution network can be selected to construct spatial relationship, and finally the distance feature vector is obtained.

[0124] The environmental parameter perception data is standardized by normalization method, and the normalization can be based on maximum and minimum value normalization, Z-score standardization or robust scaling, etc. The original value is adjusted to a unified scale to improve the training stability. Then the parameter feature extraction mechanism is used for processing, and the periodicity, variability or trend features of this type of parameters are extracted, which can be generated by one-dimensional convolution, sliding window aggregation or frequency domain analysis, etc. to generate environmental parameter feature vector.

[0125] The above four single-modal feature vectors are input into the fusion network, and the fusion network constructs a multi-head self-attention mechanism, each attention head is responsible for capturing the interaction relationship between different types of features, and calculating the correlation weight matrix between features. Based on the matrix, weighted summation is performed to fuse the channel feature vectors and construct a unified spatial semantic expression fusion perception feature vector, which provides a high-level representation for subsequent perception state generation and action decision.

[0126] The embodiment extracts four types of features of vision, audio, distance and environmental parameters through separate channels and introduces a differentiated processing mechanism, while maintaining their respective expression advantages, it solves the problem of inconsistent original data dimensions. In the fusion process, the multi-head self-attention mechanism is applied to strengthen the context relationship capture ability between multi-modal, so that the fusion perception feature vector has the ability of cross-modal collaborative enhancement, thereby effectively improving the robustness and generalization ability of the overall environmental perception, and providing a high-quality input basis for subsequent strategy optimization and task decision.

[0127] In one embodiment, the above step S30 comprises:

[0128] S301, performing state space definition processing on the fusion perception feature vector to generate a perception state space;

[0129] S302, mapping the environmental information at the current time to the perception state space to determine the current perception state;

[0130] S303, defining a sensor parameter adjustment action including adjusting the camera focal length action, modifying the microphone gain action, and adjusting the radar scanning range action;

[0131] S304, defining a working mode switching action including switching the infrared imaging mode action and activating the laser ranging mode action;

[0132] S305, defining a data collection frequency adjustment action including a high-definition video collection frame rate increasing action and an audio sampling frequency decreasing action;

[0133] S306, generating a perception action space based on the sensor parameter adjustment action, the working mode switching action and the data collection frequency adjustment action;

[0134] S307, constructing a perception strategy network including a deep neural network layer and an attention mechanism layer;

[0135] S308, inputting the current perception state into the perception strategy network to obtain an action probability distribution;

[0136] S309, selecting a preset number of actions with high probability values in the action probability distribution to generate a candidate action set;

[0137] S310, verifying whether the candidate actions in the candidate action set are within the perception action space and meet physical executable conditions to generate valid candidate actions;

[0138] S311, determining a reward expectation value of each valid candidate action;

[0139] S312, selecting a valid candidate action corresponding to the maximum reward expectation value as a training action.

[0140] In the embodiment, the fusion perception feature vector needs to be converted into an expression form suitable for the input of the strategy network. First, the fusion perception feature vector needs to be processed by state space definition. The core of the processing is to construct a low-dimensional state space with distinguishability, generalizability and physical interpretability, so that the perception features can be mapped into the state space and used for decision-making. In implementation, the fusion perception feature vector can be dimensionally compressed by a self-encoder, or a state index mapping relationship can be established after clustering. The generated perception state space needs to maintain the geometric structure relationship between states, so that the subsequent strategy learning can recognize the state similarity.

[0141] Subsequently, the observation representation extracted from the fusion perception feature vector at the current time is mapped to the defined state space to determine the current perception state. The mapping can be realized by a parameterized network, or a projection method such as principal component projection, t-SNE dimensionality reduction and remapping can be used to ensure that the perception state expression can maintain compression and task relevance. The current perception state is an abstract expression of the environment, which is used for subsequent training action generation.

[0142] To realize multi-dimensional and multi-scale training action expression, an operable perception action space needs to be defined. First, in the dimension of sensor parameter adjustment, actions such as camera focal length adjustment, microphone gain adjustment, radar scanning range setting, etc. are defined. These actions belong to the category of physical parameter adjustment and have clear execution channels and execution feedback, which are deployable in the robot control system. Secondly, in the working mode level, actions such as infrared imaging mode switching, laser ranging mode activation, etc. are defined. These actions involve sensor mode level configuration and are often used to adapt to the identification needs in low light, long distance or special scenes. Further, in the data acquisition frequency dimension, actions such as video acquisition frame rate improvement, audio sampling rate reduction, etc. are defined. By adjusting the sampling rate, the perception quality and resource consumption are balanced. In implementation, various actions are modeled in a parameterized vector manner and serve as the basic component of the perception action space, building an enumerable and constrained action set.

[0143] The structure of the perception strategy network needs to be compatible with both state abstraction and action policy generation, so an architecture containing deep neural network layers and attention mechanism layers is selected. The neural network layer captures the nonlinear change relationship of state expression through hierarchical stacking, and the attention mechanism layer is used to dynamically focus on the key dimensions in the state vector, improving the response ability to complex state features. The network structure can include multiple layers of fully connected structure plus multi-head attention module, each head focusing on different types of input subspace to enhance expression diversity.

[0144] After inputting the current perception state into the above strategy network, a probability distribution over the complete perception action space can be output, representing the relative probability of each training action being adopted. The probability distribution is a real-valued vector, and each element corresponds to the adoption weight of an action. Based on this probability distribution, a number of actions with high probability values are selected to form a candidate action set. This set is used for subsequent effectiveness verification and final action determination, reducing the search space and improving sampling efficiency.

[0145] To ensure the executability of the actions in the candidate action set in the actual scene, it is also necessary to verify whether each action exists in the perception action space and meets the physical executable conditions. The execution condition verification usually includes action parameter range check, sensor state compatibility verification and current resource occupation analysis, and finally selects the effective candidate actions. Each effective candidate action calculates the reward expectation value according to its expected performance in the current state. The reward expectation value is generated by the environment feedback modeler, which can be based on the simulation environment or historical experience samples, and is obtained by predicting the reward size brought by the execution result.

[0146] Finally, from all valid candidate actions, the action corresponding to the maximum reward expectation value is selected as the training action, which will be used for subsequent interaction execution and experience sample generation, ensuring that the policy update process focuses on the most valuable behavior selection.

[0147] The embodiment constructs a multi-dimensional and constraintable perception action space, and combines a state mapping mechanism and a joint decision framework of a policy network, so that the selection process of the training action has high task relevance and physical executability. In the execution process, the action probability distribution is generated by the policy network, combined with action screening, effectiveness verification and reward expectation evaluation, to form a closed-loop action generation link including abstraction, screening and evaluation. This method effectively improves the execution value density of the training action, speeds up the policy convergence rate, reduces the invalid exploration steps, and enhances the policy stability and perception adaptability of the agent in a complex environment.

[0148] In one embodiment, the above step S40 comprises:

[0149] S401, executing the training action;

[0150] S402, receiving environment feedback information including immediate reward signal and state transition information;

[0151] S403, recording the current perception state when the training action is executed;

[0152] S404, generating experience sample data units based on the current perception state, training action, immediate reward signal and state transition information;

[0153] S405, verifying the integrity of the experience sample data units;

[0154] S406, storing the experience sample data units that pass the integrity verification into the experience replay buffer.

[0155] In the embodiment, the training action is the direct behavior output after the perception policy network completes the decision process, which is determined by the perception state input and the policy network weight. The action may include parameter adjustment, working mode switching or sampling frequency adjustment of the perception device, and needs to be sent through the underlying hardware control interface and device driver protocol to realize the control signal transmission. The state synchronization record should be kept during the action execution process to form a complete data structure with the subsequent environment feedback information.

[0156] After the training action is actually performed, the environment response mechanism will immediately generate feedback information. The feedback information needs to include two key elements, one is the immediate reward signal, which is used to evaluate the contribution of the current action to the overall objective function; the second is the state transition information, which represents the change of the environment state after the action is performed. The generation of the immediate reward signal can be captured by the monitoring module in the external environment, such as collecting the change of device power consumption, the degree of improvement of image quality indicators, the change of audio recognition accuracy, etc., and converting it into a numerical form feedback to the system. The state transition information is based on the perception state expression of the subsequent time window in the perception data stream, and the difference modeling is performed with the original state to form the state evolution path. This path not only represents the short-term action impact, but also provides traceability of state transition for subsequent policy update.

[0157] In order to sequentially bind the training action and the feedback information, the perception state at the action execution time needs to be recorded. This operation synchronously locks between the action generated by the policy network and the execution, ensures that the perception state strictly corresponds to the action timestamp, and prevents information mismatch. On this basis, the current perception state, training action, immediate reward signal and state transition information four-tuple are combined into an experience sample data unit. The data unit structure is clear, information is closed loop, and has all the elements needed for effective back propagation in subsequent policy training.

[0158] The generated experience sample data unit needs to be verified for integrity before storage. The verification content includes whether the field is missing, whether the data structure conforms to the pre-defined format, whether the value domain is within the physically reasonable range, whether the perception state and the transition state are different time segments, whether the training action exists in the perception action space, etc. If the verification is passed, it means that the experience sample data unit has logical consistency, physical interpretability and policy training applicability.

[0159] The experience sample data unit is written into the experience replay buffer after being verified. The buffer is a ring storage structure that supports asynchronous reading and index access, supports concurrent writing and priority sampling strategy. The buffer can be designed based on the priority experience replay mechanism, which maintains a sample error degree ordering index to realize the sampling enhancement of important experience units during policy training. In addition, it can also remove expired samples through a regular sliding window mechanism to maintain the freshness and representativeness of sample distribution. The structure and management strategy of the experience replay buffer directly affect the training stability and sample efficiency of the policy network, so it should be dynamically adjusted according to the training frequency, sample complexity, memory capacity and other parameters.

[0160] The embodiment forms structured experience sample data units by combining the training action, the perception state, the immediate reward signal and the state transition information, ensures the integrity of the experience sample data units through a verification mechanism, and further stores the experience sample data units in a controllable experience replay buffer. While realizing the reinforcement learning training closed loop, the embodiment effectively improves the structural rigor and use efficiency of the experience data. The processing flow guarantees the traceability of the training sample, the causal consistency of the action-state-reward link and the controllability of the sample quality in the dynamic environment, so that the subsequent policy network iterative training can continuously obtain stable and high-quality experience data sources from real interactions, thereby accelerating policy convergence, suppressing overfitting phenomenon and improving environmental adaptability.

[0161] In one embodiment, the above step S50 comprises:

[0162] S501, randomly sampling a plurality of experience sample data units from the experience replay buffer as training sample data;

[0163] S502, extracting the current perception state, training action, immediate reward signal and state transition information in the training sample data;

[0164] S503, determining the action prediction value of the perception policy network for the current perception state;

[0165] S504, calculating the target action value corresponding to the state transition information;

[0166] S505, determining the difference value between the action prediction value and the target action value;

[0167] S506, determining the policy network loss based on the difference value;

[0168] S507, based on the policy network loss, iteratively updating the network parameters of the perception policy network through a backpropagation operation until a convergence condition is met.

[0169] In the embodiment, sampling training sample data from the experience replay buffer is a process of constructing a training batch based on historical interaction data. The experience replay buffer supports sampling strategies such as priority index, time window and random sampling. In the training process, a plurality of experience sample data units are usually obtained as training sample data by using uniform random sampling or priority sampling mechanism. Each data unit structure includes four parts: current perception state, training action, immediate reward signal and state transition information, which constitute a complete state-action-reward-successor state quadruple.

[0170] After the training sample data is extracted, the structure field thereof is first unpacked and format-converted, the current perception state, training action, immediate reward signal and state transition information in the batch sample are extracted into tensor form respectively, and the dimensions are unified for subsequent neural network calculation. The current perception state is input into the policy network as the input of the policy network, the training action is used to supervise the prediction of the action space distribution, the state transition information is used to generate the action value estimation in the subsequent state, and the reward signal is used to construct the target action value.

[0171] In the policy network prediction process, the current perception state of each training sample is input into the perception policy network, the corresponding action probability distribution or action value function output is calculated, and the action prediction value in the current state is extracted. The expression form of the action prediction value depends on the output structure of the policy network, which can be a policy distribution vector, a Q value vector or an action probability and value function pair under the Actor-Critic structure.

[0172] In order to obtain the target action value, the expected value estimation based on the subsequent state represented by the state transition information and the reward signal is required. The calculation process of the target action value usually adopts the time difference method, such as TD(0), TD(λ) or DoubleQ-Learning method. In the Q-learning structure, the value can be composed by weighting the immediate reward with the maximum action Q value of the subsequent state. If the policy adopts a distributed reinforcement learning structure, the target action value can also include multiple quantile distribution parameters.

[0173] The action prediction value and the target action value are compared to calculate the difference value between them. The difference value can be measured by loss functions such as mean square error, Huber loss function, KL divergence, etc., to form the policy network loss. The loss function definition should have gradient continuity, error penalty stability and robustness to extreme noise, so as to ensure the convergence stability and reasonable update direction of the network parameters in high-dimensional space.

[0174] After the policy network loss is generated, the network parameter update is performed using the back propagation operation. Back propagation calculates the gradient of the loss with respect to each network parameter through automatic differentiation mechanism, and combines gradient descent method (such as Adam, RMSProp, SGD, etc.) to perform weight update operation. In the training process, small batch iteration is adopted, and each round of parameter update is based on the error guided adjustment of the current batch sample. The network parameters gradually converge to the optimal solution in high-dimensional space.

[0175] The training iteration is continuously performed until a convergence condition is met. The convergence condition can include that a loss function is lower than a threshold, that a parameter update amplitude is lower than a threshold, that a loss function variation amplitude is insufficient within a fixed number of rounds, and the like, and is dynamically adjusted in combination with a training stability constraint. In the training process, a sliding window average mechanism, an early stopping strategy, and a multi-period cross-validation mechanism can also be used to monitor network stability and overfitting risk, to improve the generalization ability and convergence quality of the final policy network.

[0176] The embodiment realizes the ability to continuously optimize the perception strategy from historical interaction samples by constructing a complete processing chain for sampling training sample data from the experience replay buffer, and constructing a loss function based on the difference value, and iteratively updating the network parameters of the perception strategy network using the backpropagation mechanism. The training mechanism avoids the time correlation between samples, enhances the strategy generalization ability, effectively improves the adaptability and response accuracy of the perception strategy to complex dynamic environments, and enables the perception system to have more stable dynamic adjustment ability and higher efficient learning convergence ability in the multi-source data background.

[0177] In one embodiment, the above step S60 includes:

[0178] S601, loading an updated perception strategy network;

[0179] S602, obtaining a task configuration of a target environment perception task;

[0180] S603, collecting multi-source perception data of a current environment;

[0181] S604, preprocessing the multi-source perception data of the current environment to generate current preprocessed perception data;

[0182] S605, extracting a plurality of single-modal feature vectors from the current preprocessed perception data and fusing them to generate a current fused perception feature vector;

[0183] S606, determining a task perception state based on the current fused perception feature vector;

[0184] S607, inputting the task perception state into the updated perception strategy network to obtain a probability distribution of a task action;

[0185] S608, determining an optimal target task action according to the task configuration and the probability distribution of the task action.

[0186] In this embodiment, loading the updated perception policy network refers to loading the policy network model that has been optimized and converged through multiple training rounds from the storage medium into the current task processing flow after the policy training is completed for the inference stage to call. The model structure usually includes input layer, deep neural network layer, attention mechanism layer, action distribution prediction layer and other calculation modules, and the network parameters and model weights need to be initialized synchronously during the loading process to ensure the accuracy and stability of the policy output. The loading operation can be combined with the model version control mechanism to select the target policy, supporting dynamic selection of the most suitable policy instance according to the task attributes in a multi-policy scenario.

[0187] The task configuration of the target environment perception task is to clearly define the target constraints and execution requirements of the current perception task. The task configuration usually includes structured parameters such as target scene information, priority perception channel, action space constraint, policy objective function weight, task execution frequency limit, etc. The task configuration can be input by the upper control module, or dynamically generated by the historical task scheduling record, and used as an important condition reference for subsequent action selection in the current task execution.

[0188] The operation of collecting multi-source perception data of the current environment relies on the multi-modal sensor array deployed in the target scene, including visual camera, microphone array, radar, temperature and humidity sensor, light and air pressure detector, etc. Various sensors collect original perception data at the current timestamp according to the synchronous trigger signal to ensure the timing consistency and physical coupling integrity in the subsequent data processing process.

[0189] The pre-processing of the multi-source perception data of the current environment generates the current pre-processed perception data, including image filtering, audio denoising, distance parameter calibration and environment variable normalization, etc. The processing goal is to eliminate random noise and hardware offset in the collected data, so that the subsequent feature extraction mechanism receives more consistent and high-quality perception input. The pre-processing process needs to follow the same algorithm flow and parameter configuration as the training phase to maintain the uniformity of the perception process in the training-inference closed loop.

[0190] The operation of extracting and fusing multiple single-modal feature vectors from the current pre-processed perception data inherits the perception encoding path constructed in the training phase. The visual perception data is processed by convolution network and attention mechanism to generate visual feature vector, the audio data is processed by time series feature analysis mechanism to generate audio feature vector, and the distance and environment data are respectively processed by spatial modeling and parameter abstraction mechanism to generate distance feature vector and environment parameter feature vector. After the above multiple single-modal feature vectors are processed by the multi-head attention mechanism in the fusion network, the current fused perception feature vector is generated, which is used as the basis for high-dimensional representation of the task state.

[0191] The process of determining the task perception state based on the current fusion perception feature vector includes the invocation of a state space mapping mechanism. The state space has been constructed in the training phase and used for reinforcement learning policy optimization. In the inference phase, the current fusion perception feature vector is projected into the state space to generate the task perception state. This state structure retains the comprehensive encoding of multi-source environmental information and is consistent with the policy function domain formed in the training phase in terms of spatial distribution, ensuring the structural compatibility of the policy network input data.

[0192] Inputting the task perception state into the updated perception policy network to obtain the probability distribution of the task action is the core operation of the policy inference module. The policy network takes the task perception state as input, and after deep network transformation and nonlinear mapping, outputs the probability estimate value of each action in the task action space. This probability distribution reflects the expected execution priority of each action under the current state, and is consistent in structure with the policy mapping learned in the training phase.

[0193] The process of determining the optimal target task action based on the task configuration and the probability distribution of the task action integrates the dual constraints of policy network output and task actual demand. First, a set of candidate actions with high probability values is selected according to the action probability ranking, and then actions that do not meet the execution requirements are filtered based on the action feasibility constraints, environmental physical conditions, and device execution boundaries included in the task configuration. Finally, the effective action with the largest probability value is selected from the candidate actions that meet the conditions as the optimal task action for perception execution, realizing the joint satisfaction of policy optimality and physical executability.

[0194] Example: In the scenario of a hospital intensive care unit, a set of intelligent medical robot systems with multi-modal perception capabilities is deployed. The system needs to undertake key tasks including patient state monitoring, abnormal behavior warning, and medical staff assistance response. To cope with the problems of variable patient physiological state, complex nursing behavior, and frequent sensor interference in the intensive care unit, the system needs to have the ability of dynamically adaptive perception strategy generation and execution.

[0195] The system first acquires multi-source perception data in the ward environment through its integrated multi-type sensors. The perception data includes high-definition camera captured visual image sequences, microphone array captured environmental audio signals, ultrasonic radar obtained spatial distance data, as well as environmental parameter data such as temperature and humidity, air quality, and carbon dioxide concentration. The collected data is aligned according to the set time synchronization protocol to ensure the time consistency of subsequent feature processing. The visual images are filtered by median filtering to remove image disturbance spots and shading, the audio signals are filtered by frequency domain filtering to remove background mechanical noise, and the distance information and environmental parameters are calibrated by a unified calibration module to eliminate device bias, finally forming an aligned data set containing four types of perception channels.

[0196] Subsequently, the system extracts specific single-modal features from the four types of perception data respectively. Visual images are processed by a deployed multi-scale convolutional neural network to extract clear human posture and item distribution features, and further utilize attention mechanism to enhance the expression of local change areas; audio signals are processed by MFCC acoustic modeling and GRU time series analysis to extract periodic breathing sound and device alarm sound time series patterns; radar echoes are converted into spatial frequency spectrum through short-time Fourier transform, and then extract the micro-motion features of the patient's bedside area; environmental parameters are processed by standard normalization and feature reconstruction mechanism to extract air flow and temperature change trends. These feature vectors are jointly encoded into a fusion perception feature vector in a multi-head attention fusion network, which fully expresses the state information of the current medical scene.

[0197] The system maps the fusion perception feature vector into a defined perception state space to generate the current task perception state. The construction of the state space considers multiple dimensions such as adjustable devices (e.g. camera angle, microphone gain), operating modes (e.g. infrared night vision mode, laser body temperature scanning), sampling strategies (e.g. high frame rate video recording, audio compression) in the medical scene. After receiving the task perception state, the perception strategy network outputs a set of action probability distributions. In a scene where the current patient has abnormal limb movements but the audio is quiet, the strategy network may output a high probability value for the combined action of "increasing video frame rate and switching to night vision mode". The system verifies that this action is feasible within the current hardware configuration and execution space, and conforms to the high reward feedback records in the strategy history data, and finally selects this action as the training action for subsequent strategy optimization.

[0198] During the strategy training process, the system automatically executes the training action, collects the immediate reward signals (such as recognition accuracy improvement) and state transition information (such as patient posture state update) during the response period, and records the current perception state to form an experience sample data unit. After integrity verification, these data units are stored in the experience replay buffer to provide stable sample sources for subsequent iterative learning.

[0199] The training module periodically samples multiple experience sample data units from the experience replay buffer, extracts the current perception state, action, reward, and transition information, and obtains the action prediction value through forward inference of the strategy network. At the same time, the target action value is constructed according to the actual state transition and reward. The system calculates the difference between the prediction value and the target value, propagates the strategy network error based on the difference, and iteratively updates the network parameters until the convergence condition is met, obtaining the optimized perception strategy network.

[0200] The trained perception strategy network is loaded for actual perception task inference. When the system needs to respond to a new medical task (for example: night automatic patrol ward task), the task configuration and the perception data of the current environment are synchronized to input the inference process. The system completes the perception data preprocessing, feature fusion, state inference, and finally selects the optimal task action (such as adjusting the camera to the maximum night vision sensitivity and reducing the audio sampling rate to reduce redundancy) through the optimized strategy network, effectively executing efficient and accurate perception control behavior.

[0201] In the business outlets of financial institutions, intelligent customer service terminals with environmental perception and interaction capabilities are deployed, which are used to realize intelligent queuing guidance, customer identity auxiliary verification, abnormal behavior monitoring and teller risk reminding functions. In the face of the characteristics of large number of customers, complex behavior patterns, high environmental noise, etc., the system needs to have self-adaptive adjustment of perception ability, especially in peak hours or when risk events occur, it can timely perceive changes and optimize responses.

[0202] After the system starts, it synchronously collects visual images (for recognizing faces and behaviors), audio data (collecting customer conversations and background noise), distance information (judging customer proximity and queue length), and environmental parameter data (detecting light, temperature, noise level, etc.) using the built-in multi-type sensor module. All perception data is filtered, denoised, calibrated, etc. to form time-synchronized preprocessed perception data, providing a cleaned basis for subsequent analysis.

[0203] In the feature extraction stage, the system captures the standing posture features, emotional expressions, and action behaviors of the queuing customers based on visual images through convolutional networks, and focuses on possible risky actions (such as repeated walking, emotional agitation, etc.) through attention mechanisms. The audio data is processed through spectrogram and time series modeling to analyze speaking frequency and tone changes, which helps to judge customer emotions and complaint tendencies. Distance data is processed in the frequency domain to identify spatial tension and queue area density. Environmental parameter data is used to infer whether there are interference conditions such as over-temperature, over-brightness, abnormal noise, etc. These single-modal feature vectors are effectively fused under the multi-head attention mechanism to construct a fused perception feature vector, improving the context perception ability.

[0204] The system maps the fused perception feature vector into a defined state space to identify the current perception state of the point scene. For example, at a certain time, the system identifies that there are many people queuing in front of counter A, the customer's mood fluctuates greatly, the noise level is high, and the clerk system reacts slowly. In view of this state, the system has a predefined perception action space, including camera angle adjustment (better focus on the queuing area), microphone gain adjustment (to cope with the noise environment), data sampling frequency improvement (to speed up risk identification), and other action combinations. The perception strategy network outputs multiple action candidate sets based on state input, and through probability sorting, execution condition checking and reward evaluation, finally selects the optimal action combination as the training action.

[0205] The system executes the selected training action, such as adjusting the angle to focus on the customer's face and increasing the voice recognition sampling rate, while listening to feedback signals such as customer identification accuracy changes, mood fluctuation trends, and clerk reaction delays. The system structures the perception state, training action, reward signal (such as successful risk warning), state change, etc. during this period into experience sample data and stores it in the experience replay buffer.

[0206] The training phase samples multiple sample units from the replay buffer, extracts training state, action and reward information, and calculates the difference between the predicted value of the current strategy network and the expected action value. Through back propagation, the strategy network parameters are iteratively adjusted to optimize future perception response capabilities, ensuring continuous learning and adaptation to different business environment changes.

[0207] When the updated perception strategy network is used for formal deployment tasks, such as when the branch manager starts the "peak business risk monitoring" task, the system loads the latest strategy model, real-time collects multi-source perception data and completes fusion processing, infers the task perception state, and quickly generates task action probability distribution, combined with task configuration to automatically trigger the optimal action combination, such as "improve audio processing accuracy and activate clerk assistant pop-up window", effectively assisting artificial service response.

[0208] Through the above process, the system can automatically identify and respond to complex environmental state changes in financial business scenarios, especially in the early stages of risk events, to provide efficient assistant decision-making capabilities for clerks, reduce the burden of artificial identification, and improve the intelligent level of point service and customer satisfaction.

[0209] The embodiment can complete the reasoning and selection of task actions in a highly matched manner by loading the updated perception strategy network in the environmental perception task, combining the current task configuration, the real-time collected multi-source perception data and the fusion feature state. The processing flow realizes the dynamic adaptation capability of the perception task in different environments, and improves the accuracy and response efficiency of the task decision. Through the structural consistency maintenance of the training process and the reasoning process, the closed loop is closed from the strategy optimization to the task landing, so that the perception system can maintain the optimal perception performance and stability output in the continuously evolving environment.

[0210] In an embodiment, a task action generation device is provided, which corresponds to the task action generation method in the above embodiments. Referring to Figure 3 , Figure 3 A functional module schematic diagram of a preferred embodiment of the task action generation device of the present application is shown in FIG. 1. The functional modules include a perception data preprocessing module 10, a feature fusion extraction module 20, a perception strategy generation module 30, an experience sample generation module 40, a strategy network updating module 50 and a task action reasoning module 60. The detailed descriptions of the functional modules are as follows:

[0211] The perception data preprocessing module 10 is configured to acquire multi-source perception data, and preprocess the multi-source perception data to generate preprocessed perception data.

[0212] The feature fusion extraction module 20 is configured to extract a plurality of single-modal feature vectors from the preprocessed perception data, and fuse the plurality of single-modal feature vectors to generate a fusion perception feature vector.

[0213] The perception strategy generation module 30 is configured to determine a perception state based on the fusion perception feature vector, and generate a training action according to the perception state by using a perception strategy network.

[0214] The experience sample generation module 40 is configured to execute the training action and receive environmental feedback information, generate experience sample data based on the training action and the environmental feedback information, and store the experience sample data in an experience replay buffer.

[0215] The strategy network updating module 50 is configured to sample training sample data from the experience replay buffer, and iteratively update network parameters of the perception strategy network according to the training sample data.

[0216] The task action reasoning module 60 is configured to process a target environmental perception task by using the updated perception strategy network, and generate a target task action.

[0217] In an embodiment, the perception data preprocessing module 10 is specifically configured to:

[0218] Synchronously collecting, by a plurality of sensors, multi-source perception data including visual perception data, audio perception data, distance perception data and environmental parameter perception data;

[0219] Applying median filtering processing to the visual perception data to generate denoised visual perception data;

[0220] Applying a calibration model processing to the distance perception data and the environmental parameter perception data to generate calibrated distance perception data and calibrated environmental parameter perception data;

[0221] Performing noise reduction processing on the audio perception data to generate denoised audio perception data;

[0222] Performing spatio-temporal alignment on the denoised visual perception data, the denoised audio perception data, the calibrated distance perception data and the calibrated environmental parameter perception data to generate aligned multi-source perception data;

[0223] Combining the aligned multi-source perception data to generate preprocessed perception data.

[0224] In an embodiment, the feature fusion extraction module 20 is specifically configured to:

[0225] Extracting visual perception data from the preprocessed perception data and processing the visual perception data through a convolution network to generate visual basic features;

[0226] Processing the visual basic features through a self-attention mechanism to generate a visual feature vector;

[0227] Extracting audio perception data from the preprocessed perception data and performing acoustic feature extraction processing on the audio perception data to generate audio basic features;

[0228] Processing the audio basic features through a time series analysis mechanism to generate an audio feature vector;

[0229] Extracting distance perception data from the preprocessed perception data and performing time-frequency conversion processing on the distance perception data to generate distance frequency domain features;

[0230] Processing the distance frequency domain features through a spatial feature extraction mechanism to generate a distance feature vector;

[0231] Extracting environmental parameter perception data from the preprocessed perception data and performing normalization processing on the environmental parameter perception data to generate normalized environmental parameters;

[0232] Processing the normalized environmental parameters through a parameter feature extraction mechanism to generate an environmental parameter feature vector;

[0233] input the visual feature vector, the audio feature vector, the distance feature vector and the environmental parameter feature vector into a fusion network, and calculate feature correlation weights through a multi-head self-attention mechanism of the fusion network;

[0234] fuse the visual feature vector, the audio feature vector, the distance feature vector and the environmental parameter feature vector according to the feature correlation weights, and generate a fused perception feature vector.

[0235] In an embodiment, the perception strategy generation module 30 is specifically configured to:

[0236] perform state space definition processing on the fused perception feature vector, and generate a perception state space;

[0237] map environmental information at a current time to the perception state space, and determine a current perception state;

[0238] define a sensor parameter adjustment action including an action of adjusting a camera focal length, an action of modifying a microphone gain, and an action of adjusting a radar scanning range;

[0239] define a working mode switching action including an action of switching an infrared imaging mode and an action of activating a laser ranging mode;

[0240] define a data acquisition frequency adjustment action including an action of increasing a high-definition video acquisition frame rate and an action of reducing an audio sampling frequency;

[0241] generate a perception action space based on the sensor parameter adjustment action, the working mode switching action and the data acquisition frequency adjustment action;

[0242] construct a perception strategy network including a deep neural network layer and an attention mechanism layer;

[0243] input the current perception state into the perception strategy network, and obtain an action probability distribution;

[0244] select a preset number of actions with high probability values in the action probability distribution according to a preset rule, and generate a candidate action set;

[0245] verify whether candidate actions in the candidate action set are within the perception action space and meet physical executable conditions, and generate valid candidate actions;

[0246] determine a reward expectation value of each valid candidate action;

[0247] select a valid candidate action corresponding to a maximum reward expectation value as a training action.

[0248] In an embodiment, the experience sample generation module 40 is specifically configured to:

[0249] execute the training action;

[0250] receiving environment feedback information including instant reward signals and state transition information;

[0251] recording a current perception state at the time of performing the training action;

[0252] generating an experience sample data unit based on the current perception state, the training action, the instant reward signal and the state transition information;

[0253] verifying the integrity of the experience sample data unit;

[0254] storing the experience sample data unit that passes the integrity verification in an experience replay buffer.

[0255] In an embodiment, the policy network updating module 50 is specifically configured to:

[0256] randomly sampling a plurality of experience sample data units from the experience replay buffer as training sample data;

[0257] extracting the current perception state, the training action, the instant reward signal and the state transition information in the training sample data;

[0258] determining an action prediction value of the perception policy network for the current perception state;

[0259] calculating a target action value corresponding to the state transition information;

[0260] determining a difference value between the action prediction value and the target action value;

[0261] determining a policy network loss based on the difference value;

[0262] iteratively updating network parameters of the perception policy network through a backpropagation operation based on the policy network loss until a convergence condition is met.

[0263] In an embodiment, the task action inference module 60 is specifically configured to:

[0264] loading the updated perception policy network;

[0265] obtaining a task configuration of a target environment perception task;

[0266] collecting multi-source perception data of a current environment;

[0267] preprocessing the multi-source perception data of the current environment to generate current preprocessed perception data;

[0268] extracting a plurality of single-modal feature vectors from the current preprocessed perception data and fusing them to generate a current fused perception feature vector;

[0269] The task perception state is determined based on the current fused perception feature vector;

[0270] The task perception state is input into the updated perception policy network to obtain the probability distribution of task actions;

[0271] Based on the task configuration and the probability distribution of task actions, the optimal target task action is determined.

[0272] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a task action generation method on the server side.

[0273] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a task action generation method on the user side.

[0274] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0275] Acquire multi-source sensing data and preprocess the multi-source sensing data to generate preprocessed sensing data;

[0276] Multiple single-modal feature vectors are extracted from the preprocessed sensing data, and the multiple single-modal feature vectors are fused to generate a fused sensing feature vector;

[0277] determine a perception state based on the fusion perception feature vector, and generate a training action according to the perception state by using a perception policy network;

[0278] execute the training action and receive environment feedback information, generate experience sample data based on the training action and the environment feedback information, and store the experience sample data into an experience replay buffer;

[0279] sample training sample data from the experience replay buffer, and iteratively update network parameters of the perception policy network according to the training sample data;

[0280] process a target environment perception task by using the updated perception policy network, and generate a target task action.

[0281] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0282] obtain multi-source perception data, and pre-process the multi-source perception data to generate pre-processed perception data;

[0283] extract a plurality of single-modal feature vectors from the pre-processed perception data, and fuse the plurality of single-modal feature vectors to generate a fusion perception feature vector;

[0284] determine a perception state based on the fusion perception feature vector, and generate a training action according to the perception state by using a perception policy network;

[0285] execute the training action and receive environment feedback information, generate experience sample data based on the training action and the environment feedback information, and store the experience sample data into an experience replay buffer;

[0286] sample training sample data from the experience replay buffer, and iteratively update network parameters of the perception policy network according to the training sample data;

[0287] process a target environment perception task by using the updated perception policy network, and generate a target task action.

[0288] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement are described above with reference to the method embodiments, and the related descriptions of the server side and the user side are not repeated here.

[0289] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0290] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0291] It should be noted that if non-company software tools or components appear in the embodiments of the present application, they are only used for example introduction and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A task action generation method characterized by comprising: The method comprises the following steps: obtaining multi-source perception data, and preprocessing the multi-source perception data to generate preprocessed perception data; extracting a plurality of single-modal feature vectors from the preprocessed perception data, and fusing the plurality of single-modal feature vectors to generate a fused perception feature vector; determining a perception state based on the fused perception feature vector, and generating a training action according to the perception state by using a perception strategy network, comprising: performing state space definition processing on the fused perception feature vector to generate a perception state space; mapping environment information at a current time to the perception state space to determine a current perception state; defining sensor parameter adjustment actions including adjusting camera focal length actions, modifying microphone gain actions, and adjusting radar scanning range actions; defining working mode switching actions including switching infrared imaging mode actions and activating laser ranging mode actions; defining data acquisition frequency adjustment actions including increasing high-definition video acquisition frame rate actions and reducing audio sampling frequency actions; generating a perception action space based on the sensor parameter adjustment actions, the working mode switching actions, and the data acquisition frequency adjustment actions; constructing a perception strategy network comprising a deep neural network layer and an attention mechanism layer; inputting the current perception state into the perception strategy network to obtain an action probability distribution; selecting a preset number of actions with high probability values in the action probability distribution to generate a candidate action set; verifying whether the candidate actions in the candidate action set are within the perception action space and meet physical executable conditions to generate valid candidate actions; determining a reward expectation value of each valid candidate action; and selecting a valid candidate action corresponding to the maximum reward expectation value as the training action; executing the training action and receiving environment feedback information, generating experience sample data based on the training action and the environment feedback information, and storing the experience sample data into an experience replay buffer; sampling training sample data from the experience replay buffer, and iteratively updating network parameters of the perception strategy network according to the training sample data; processing a target environment perception task by using the updated perception strategy network to generate a target task action.

2. The task action generation method of claim 1, wherein, obtaining multi-source perception data, and preprocessing the multi-source perception data to generate preprocessed perception data, comprising: synchronously collecting multi-source perception data including visual perception data, audio perception data, distance perception data, and environmental parameter perception data by using a plurality of sensors; applying median filter processing to the visual perception data to generate denoised visual perception data; applying a calibration model to the distance perception data and the environmental parameter perception data to generate calibrated distance perception data and calibrated environmental parameter perception data; performing noise reduction processing on the audio perception data to generate denoised audio perception data; spatiotemporally aligning the denoised visual perception data, the denoised audio perception data, the calibrated distance perception data, and the calibrated environmental parameter perception data to generate aligned multi-source perception data; combining the aligned multi-source perception data to generate preprocessed perception data.

3. The task action generation method of claim 1, wherein, extracting a plurality of single-modal feature vectors from the pre-processed perception data, and fusing the plurality of single-modal feature vectors to generate a fused perception feature vector, comprising: extracting visual perception data from the pre-processed perception data, and processing the visual perception data through a convolution network to generate visual base features; processing the visual base features through a self-attention mechanism to generate a visual feature vector; extracting audio perception data from the pre-processed perception data, and performing acoustic feature extraction processing on the audio perception data to generate audio base features; processing the audio base features through a time series analysis mechanism to generate an audio feature vector; extracting distance perception data from the pre-processed perception data, and performing time-frequency conversion processing on the distance perception data to generate distance frequency domain features; processing the distance frequency domain features through a spatial feature extraction mechanism to generate a distance feature vector; extracting environmental parameter perception data from the pre-processed perception data, and performing normalization processing on the environmental parameter perception data to generate normalized environmental parameters; processing the normalized environmental parameters through a parameter feature extraction mechanism to generate an environmental parameter feature vector; inputting the visual feature vector, audio feature vector, distance feature vector, and environmental parameter feature vector into a fusion network, and calculating feature correlation weights through a multi-head self-attention mechanism of the fusion network; fusing the visual feature vector, audio feature vector, distance feature vector, and environmental parameter feature vector according to the feature correlation weights to generate a fused perception feature vector.

4. The task action generating method of claim 1, wherein, performing the training action and receiving environmental feedback information, generating experience sample data based on the training action and environmental feedback information, and storing the experience sample data into an experience replay buffer, comprising: performing the training action; receiving environmental feedback information including an immediate reward signal and state transition information; recording a current perception state when the training action is performed; generating an experience sample data unit based on the current perception state, training action, immediate reward signal, and state transition information; verifying the integrity of the experience sample data unit; storing the experience sample data unit that passes the integrity verification into the experience replay buffer.

5. The task action generation method of claim 1, wherein, sampling training sample data from the experience replay buffer and iteratively updating network parameters of the perception policy network according to the training sample data, comprising: randomly sampling a plurality of experience sample data units from the experience replay buffer as training sample data; extracting a current perception state, training action, immediate reward signal, and state transition information from the training sample data; determining an action prediction value of the perception policy network for the current perception state; calculating a target action value corresponding to the state transition information; determining a difference value between the action prediction value and the target action value; determining a policy network loss based on the difference value; iteratively updating network parameters of the perception policy network through a backpropagation operation based on the policy network loss until a convergence condition is met.

6. The task action generating method of claim 1, wherein, processing a target environmental perception task through the updated perception policy network to generate a target task action, comprising: loading the updated perception policy network; Obtaining a task configuration of a target environment perception task; Collecting multi-source perception data of a current environment; Preprocessing the multi-source perception data of the current environment to generate current preprocessed perception data; Extracting multiple single-modal feature vectors from the current preprocessed perception data and fusing the multiple single-modal feature vectors to generate a current fused perception feature vector; Determining a task perception state based on the current fused perception feature vector; Inputting the task perception state into the updated perception policy network to obtain a probability distribution of a task action; Determining an optimal target task action according to the task configuration and the probability distribution of the task action.

7. A task action generating apparatus characterized by comprising: The task action generation apparatus comprises: A perception data preprocessing module configured to obtain multi-source perception data and preprocess the multi-source perception data to generate preprocessed perception data; A feature fusion extraction module configured to extract multiple single-modal feature vectors from the preprocessed perception data and fuse the multiple single-modal feature vectors to generate a fused perception feature vector; A perception policy generation module configured to determine a perception state based on the fused perception feature vector and generate a training action according to the perception state using a perception policy network, including: performing state space definition processing on the fused perception feature vector to generate a perception state space; mapping environment information at a current time to the perception state space to determine a current perception state; defining sensor parameter adjustment actions including adjusting camera focal length actions, modifying microphone gain actions, and adjusting radar scanning range actions; defining working mode switching actions including switching infrared imaging mode actions and activating laser ranging mode actions; defining data collection frequency adjustment actions including increasing high-definition video collection frame rate actions and reducing audio sampling frequency actions; generating a perception action space based on the sensor parameter adjustment actions, the working mode switching actions, and the data collection frequency adjustment actions; constructing a perception policy network comprising a deep neural network layer and an attention mechanism layer; inputting the current perception state into the perception policy network to obtain an action probability distribution; selecting a preset number of actions with high probability values from the action probability distribution to generate a candidate action set; verifying whether the candidate actions in the candidate action set are within the perception action space and meet physical executable conditions to generate valid candidate actions; determining a reward expectation value of each valid candidate action; and selecting a valid candidate action corresponding to the maximum reward expectation value as a training action; An experience sample generation module configured to execute the training action and receive environment feedback information, generate experience sample data based on the training action and the environment feedback information, and store the experience sample data in an experience replay buffer; A policy network updating module configured to sample training sample data from the experience replay buffer and iteratively update network parameters of the perception policy network according to the training sample data; A task action inference module configured to process a target environment perception task through the updated perception policy network to generate a target task action.

8. A computer device, comprising: The computer device comprises a memory, a processor, and a task action generation program stored in the memory and executable on the processor, and the task action generation program, when executed by the processor, implements the steps of the task action generation method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a task action generation program, and the task action generation program, when executed by the processor, implements the steps of the task action generation method according to any one of claims 1-6.

Images