Picture and video identification and analysis method based on end-side multi-mode large model

Abstract

The invention provides a picture and video recognition and analysis method based on an end-side multi-mode large model, and relates to the technical field of artificial intelligence, comprising the following steps: inputting a picture or a video into the end-side multi-mode large model, adopting Monte Carlo tree search to extract features and generate candidate recognition results; and constructing a feature semantic graph by using a graph neural network to calculate a confidence score, and selecting a highest score result and outputting the highest score result to a display interface. According to the method, the identification precision can be improved, the computing resource consumption is reduced, and the multimedia content analysis capability is enhanced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for image and video recognition and analysis based on edge multimodal large models. Background Technology

[0002] With the rapid development of artificial intelligence technology, image and video recognition and analysis technologies based on multimodal large models have been widely used in fields such as smart devices, security monitoring, and autonomous driving. Traditional image and video recognition mainly relies on cloud servers for processing; data needs to be uploaded to the cloud, undergo complex calculations, and then the results are returned to the terminal device. With the improvement of edge computing power, it has become possible to deploy multimodal large models on terminal devices for real-time image and video recognition and analysis. This not only reduces network transmission latency but also effectively protects user privacy data.

[0003] Current edge image and video recognition technologies primarily employ deep learning methods, using models such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to extract and analyze features from image and video data. Common techniques include image classification and object detection based on deep convolutional neural networks (CNNs), video behavior recognition based on recurrent neural networks (RNNs), and multimodal feature fusion based on transformer architectures. While these methods have achieved good recognition results in specific scenarios, they still face many challenges in complex and ever-changing real-world application environments.

[0004] Traditional image and video recognition methods typically employ fixed feature extraction paths, lacking the ability to adaptively analyze image and video content and struggling to effectively handle diverse information in complex scenes. Recognition accuracy drops significantly when faced with complex environments such as changes in lighting, occlusion, and angle.

[0005] Existing edge-side multimodal models suffer from high computational complexity when processing high-dimensional feature information, making it difficult to achieve efficient inference on resource-constrained terminal devices. This results in models either being limited in scale, affecting recognition performance, or running slowly, impacting user experience, and failing to meet real-time requirements.

[0006] Existing technologies are insufficient in the comprehensive evaluation of multiple candidate results. They often rely on simple confidence thresholds or voting mechanisms to filter results, lacking a deep understanding and mining of the semantic relationships between candidate results. This leads to easy misidentification in complex scenarios, especially when it is necessary to comprehensively consider the contextual information of the scenario, resulting in low accuracy. Summary of the Invention

[0007] This invention provides a method for image and video recognition and analysis based on a multimodal large model on the edge side, which can solve the problems in the prior art.

[0008] A first aspect of this invention provides a method for image and video recognition and analysis based on an edge-side multimodal large model, comprising: The image or video data to be processed is input into a multimodal large model pre-deployed on the device. The sampling probability of multiple sampling paths is calculated based on the probability distribution function. Monte Carlo tree search is performed on the image or video data according to the sampling probability. Each node of the Monte Carlo search tree contains a state value function and an action policy function. The state value function and action policy function of the node are updated iteratively in an asynchronous parallel manner. The search path with the largest search value is selected for feature extraction, and multiple candidate recognition results are generated. A feature semantic map is constructed based on the multiple candidate recognition results. The feature semantic map is iteratively propagated and calculated using a graph neural network to obtain node vector representations. The confidence score of each candidate recognition result is calculated based on the node vector representations. The candidate recognition result with the highest confidence score is selected as the final recognition result. The final recognition result is output to the display interface of the device, and the image scene information, target detection information and motion analysis information are presented on the display interface.

[0009] Calculating the sampling probability of multiple sampling paths based on a probability distribution function, and performing a Monte Carlo tree search on the image or video data according to the sampling probability, includes: The image or video data is divided into multiple image blocks. The prior probability distribution of each image block is calculated based on the variational inference method. A probability sampling space is constructed according to the prior probability distribution. Markov chain Monte Carlo sampling is performed in the probability sampling space to generate multiple sampling paths, and the sampling probability of each sampling path is calculated. Based on the sampling probability, a Monte Carlo search tree is constructed for the image or video data. The search path with the largest search value in the Monte Carlo search tree is selected for feature extraction, and multiple candidate recognition results are generated.

[0010] A probability sampling space is constructed based on the prior probability distribution. Markov chain Monte Carlo sampling is performed in the probability sampling space to generate multiple sampling paths, and the sampling probability of each sampling path is calculated, including: A probability density function is constructed for the prior probability distribution, and the prior probability distribution is smoothed by the kernel density estimation method. A probability sampling space is constructed based on the probability density function, and the probability sampling space includes the support set and transition probability matrix of the probability distribution. A Markov chain state transition network is constructed in the probability sampling space, and the state transition network includes a state space and state transition rules. A random walk sequence is generated using the state transition rule, and Monte Carlo sampling is performed based on the random walk sequence to generate multiple sampling paths, each sampling path corresponding to a transition trajectory in the state space; Calculate the sampling probability of each sampling path, which is obtained based on the state transition probability and the stationary distribution; output the multiple sampling paths and their corresponding sampling probabilities.

[0011] Each node in the Monte Carlo search tree contains a state value function and an action policy function. The state value function and action policy function of each node are iteratively updated in an asynchronous parallel manner. The search path with the largest search value is selected for feature extraction, including: Each node in the Monte Carlo search tree is configured with a state value function and an action policy function. The state value function uses a residual network structure to calculate the value evaluation score of the node state, and the action policy function generates the action selection probability distribution of the node based on the dual policy optimization criterion. Multiple parallel worker processes are started to calculate the confidence upper bound of the node, the child node with the largest confidence upper bound is selected for expansion, the asynchronous stochastic gradient descent algorithm is used to update the node's state value function and action policy function, and the updated value assessment score and action selection probability distribution are broadcast to other worker processes through an asynchronous message passing mechanism. The Monte Carlo search tree is iteratively searched based on a preset search depth. In each iteration, the search path with the largest search value is selected, where the search value is the weighted average of the value assessment score and the normalized probability distribution of the action selection.

[0012] The configuration of state value functions and action policy functions for each node in the Monte Carlo search tree includes: A multi-layer residual connection network is constructed, and skip connection paths are set in the residual connection network. The skip connection paths are used to fuse feature representations at different levels. The value evaluation score of the node state is calculated through feature fusion. An action transition probability matrix is ​​constructed using the value assessment score, and the action transition probability matrix describes the probability distribution of node state transitions; Two optimization objectives are set for the action transition probability matrix: online policy gradient and offline value estimation. The action selection strategy is optimized alternately using a dual-policy iterative approach to generate the action selection probability distribution of the node.

[0013] Calculate the upper confidence bound of the node, select the child node with the largest upper confidence bound for expansion, and update the node's state value function and action policy function using the asynchronous stochastic gradient descent algorithm, including: Obtain the state information of the node to be expanded, and extract the feature vector from the state information; A set of child nodes is constructed based on the feature vectors, and the historical access count and historical cumulative revenue of each child node are counted. Calculate the confidence interval parameters based on the historical access count and historical cumulative revenue, and input the confidence interval parameters into the adaptive estimator; Thompson sampling is performed using the adaptive estimator to obtain the upper confidence bound of each child node in the current state. The upper confidence bound is used to balance exploration and exploitation during the node expansion process. The child nodes are sorted according to the upper confidence bound value, and the child node with the largest upper confidence bound value is selected as the expansion target node. The feature vectors of the extended target nodes are mapped to the value space and the policy space to construct a distributed computing network. Asynchronous stochastic gradient descent is performed in the distributed computing network to optimize the state value function and action policy function in parallel. The state representation of the extended target node is updated based on the optimization results, and the updated state value function and action policy function are output.

[0014] A feature semantic map is constructed based on the multiple candidate recognition results. A graph neural network is used to iteratively propagate the feature semantic map to obtain node vector representations. The confidence score of each candidate recognition result is calculated based on the node vector representations. The candidate recognition result with the highest confidence score is selected as the final recognition result. The multiple candidate recognition results are projected onto the semantic feature space to obtain semantic feature representations. A feature semantic graph is constructed in the semantic feature space, where each candidate recognition result corresponds to a node in the feature semantic graph. Edge connection relationships are dynamically constructed based on the feature similarity between nodes. The feature semantic graph is subjected to multiple rounds of information propagation. Each round of propagation includes: calculating the importance weight of the node's neighborhood, aggregating the neighborhood node information based on the importance weight, nonlinearly fusing the aggregated information with the central node information, controlling the information update ratio through a gating mechanism, iterating until the node state converges, and obtaining the final node vector representation. The confidence score of each candidate recognition result is calculated based on the node vector representation; the candidate recognition result with the highest confidence score is taken as the final recognition result.

[0015] A second aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0016] A third aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0017] The beneficial effects of this application are as follows: This invention deploys a multimodal large model on the device side and uses a Monte Carlo tree search-based approach to extract features from image or video data. This effectively balances the relationship between computing resources and recognition accuracy, achieves efficient processing on the device side, reduces dependence on the network environment, and improves system response speed and user experience.

[0018] This invention introduces a feature semantic graph structure and a graph neural network to analyze and filter candidate results. Through iterative propagation of information between nodes, it fully considers the semantic correlation between candidate recognition results, improves the accuracy and reliability of recognition results, and can better handle image and video content recognition tasks in complex scenarios.

[0019] This invention optimizes the Monte Carlo tree search process through asynchronous parallelism, significantly improving the computational efficiency of the model on edge devices. At the same time, it combines the intelligent filtering mechanism of graph neural networks to achieve high-quality recognition and analysis under limited computing resources, enabling devices to provide real-time and accurate image and video content understanding services without relying on cloud computing. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the image and video recognition and analysis method based on a multimodal large model at the edge of the present invention. Figure 2 The flowchart illustrates the probability distribution of the dual-strategy optimization action for configuring residual network state value functions in MCTS node functions according to embodiments of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0023] Figure 1This is a flowchart illustrating the image and video recognition and analysis method based on an edge-side multimodal large model according to an embodiment of the present invention. Figure 1 As shown, the method includes: The image or video data to be processed is input into a multimodal large model pre-deployed on the device. The sampling probability of multiple sampling paths is calculated based on the probability distribution function. Monte Carlo tree search is performed on the image or video data according to the sampling probability. Each node of the Monte Carlo search tree contains a state value function and an action policy function. The state value function and action policy function of the node are updated iteratively in an asynchronous parallel manner. The search path with the largest search value is selected for feature extraction, and multiple candidate recognition results are generated. A feature semantic map is constructed based on the multiple candidate recognition results. The feature semantic map is iteratively propagated and calculated using a graph neural network to obtain node vector representations. The confidence score of each candidate recognition result is calculated based on the node vector representations. The candidate recognition result with the highest confidence score is selected as the final recognition result. The final recognition result is output to the display interface of the device, and the image scene information, target detection information and motion analysis information are presented on the display interface.

[0024] In one optional implementation, calculating the sampling probabilities of multiple sampling paths based on a probability distribution function, and performing a Monte Carlo tree search on the image or video data according to the sampling probabilities includes: The image or video data is divided into multiple image blocks. The prior probability distribution of each image block is calculated based on the variational inference method. A probability sampling space is constructed according to the prior probability distribution. Markov chain Monte Carlo sampling is performed in the probability sampling space to generate multiple sampling paths, and the sampling probability of each sampling path is calculated. Based on the sampling probability, a Monte Carlo search tree is constructed for the image or video data. The search path with the largest search value in the Monte Carlo search tree is selected for feature extraction, and multiple candidate recognition results are generated.

[0025] The input image or video data is processed frame by frame. The original pixel matrix is ​​read, unified into a fixed color space and bit depth, and a reproducible gamma correction and normalization strategy is employed to generate a standardized tensor for probabilistic modeling. Video data is processed frame by frame in chronological order, retaining inter-frame indices and timestamps for subsequent cross-frame consistency constraints. Ultra-high resolution data is scaled proportionally to the target processing size, and the scaling ratio is recorded for result backfilling.

[0026] Each standardized frame is divided into multiple image blocks. The block division is based on a fixed block width and height. An overlapping sliding window strategy is used between blocks to cover the content at the block boundaries. The overlap ratio is a small value in both the horizontal and vertical directions. A unique identifier is assigned to each image block, and its starting coordinates and size in the original frame are recorded. A mapping table is established from block to frame and from block to original image coordinates for subsequent feature backfilling and candidate result relocation.

[0027] A prior probability distribution is constructed for each image patch. A variational inference method is used to approximate the latent generation variables of the image patch. The variational approximation adopts a parameterizable and decomposable form, with parameters output by a lightweight encoding network. The encoding network takes image patch pixels, gradient magnitude histograms, texture orientation histograms, and local contrast statistics as inputs and outputs prior center, scale, and shape parameters. To avoid mathematical expressions, parameter updates are described in words: During training, the dissimilarity metric between the true posterior and the variational approximation is minimized; a smaller dissimilarity indicates that the prior is closer to the true distribution. During inference, the trained encoding network is fixed, and prior parameters are directly generated from image patch features. For image patches in video frames, a temporal consistency regularization is introduced, weighted and fused with prior parameters of patches at the same spatial location in adjacent frames. The weights vary with optical flow confidence, which is evaluated through matching consistency and occlusion detection.

[0028] A probability sampling space is constructed based on the prior probability distribution of each image patch. The sampling space is organized by patch, where a point corresponds to a latent feature generation path originating from that patch. Each step of the path corresponds to a value or transition of a latent variable. To ensure sampling accessibility and coverage, a candidate action set is set for each patch. Candidate actions include three types of operations: multi-directional perturbation in the latent space, multi-scale pyramid jumps in the pixel domain, and channel subset selection in the feature domain. Each type of operation is accompanied by a preference score calculated based on prior parameters; a higher preference score indicates greater support under the prior distribution. The sampling space normalizes the actions so that the probability of an action being selected is consistent with its preference score.

[0029] Markov chain Monte Carlo sampling is performed in the probability sampling space to generate multiple sampling paths for each image patch. Each path is initialized near the prior center of the patch, and the selection of the initial point follows the high-density region given by the prior parameters, with a small amount of random perturbation to break the symmetry. In the iterative steps of the path, a candidate transition is generated based on the current path position and the candidate action set. The selection probability of the candidate transition is consistent with the action preference score, and an exploration degree controlled by adaptive temperature is superimposed. The temperature decreases during the iteration process to achieve the transition from extensive exploration to fine convergence.

[0030] To improve sampling efficiency, a multi-chain parallel and delayed acceptance mechanism is introduced. First, a low-cost approximate evaluation is used to coarsely screen candidate transitions; only those passing the coarse screening proceed to the high-cost fine evaluation. For each transition, the decision to accept the transition is made by comparing the relative support of the candidate position and the current path position under the prior knowledge and the symmetry of the proposed action. If accepted, the path advances; if rejected, a pause is recorded at the current position. To avoid path stagnation in local regions, a restart strategy is implemented. When the number of consecutive rejections exceeds a threshold, the starting point is resampled from another high-density sub-region in the prior knowledge, and subsequent segments of the same path are generated.

[0031] For each sampling path, a sampling probability is calculated. This probability is expressed as a joint probability of selection and acceptance at each step of the path, including the initial point selection probability, the probability of selecting each action proposal, and the probability of acceptance or rejection. For ease of implementation, the probability values ​​generated at each stage of path construction are recorded, and these records are multiplied sequentially at the end of the path to accumulate the sampling probability for that path. To improve numerical stability, a logarithmic accumulation method is used for storage in the implementation, and the probability is converted to ordinary probability values ​​when sorting and normalization are required. For parallel multi-chain paths, their sampling probabilities are calculated and retained separately without mixing, so that subsequent tree construction stages can perform hierarchical processing based on the link source.

[0032] Based on the sampling paths and corresponding sampling probabilities generated from all image patches, a Monte Carlo search tree is constructed. The search tree is organized as a set with patches as root nodes. Each root node represents an image patch, and the child nodes of the root node correspond to the first-order latent variable values ​​or first-order action execution results of that patch. Deeper nodes correspond to multi-step latent decision sequences. The mapping method of each sampling path in the search tree is to progressively insert or update nodes along the path, while accumulating statistical information at each node, including the number of times it has been visited, the sum of the cumulative sampling probabilities from the path, and the local quality evaluation when the path reaches that node.

[0033] Local quality assessment is derived from two types of metrics: prior consistency metrics, which measure the degree of fit between the current node state and the prior high-support region; and observation consistency metrics, which measure the degree of matching between the latent variable corresponding to the node and the observed features of the image patch. The matching degree is output by a pre-trained discriminant scorer. The search value of a node is jointly calculated by the above statistics, and a weighted approach is used to comprehensively consider exploration and utilization. The weights are adaptively adjusted at different depths of the tree, with shallower layers biased towards exploration and deeper layers biased towards utilization.

[0034] Path selection based on search values ​​is performed on the search tree. For each root node, a path is selected from the root to a leaf node. At each level, the child node with the largest current search value is selected as the next expansion direction. When a leaf node is reached or a preset depth is reached, an expansion operation is triggered. From the previously generated sampled path pool, paths with prefixes consistent with the current path are selected, and their unexpanded next steps are added to the tree as new child nodes. The corresponding sampling probability and local quality evaluation are written into the node statistics. After each expansion, the search values ​​of all traversed nodes are backtracked and updated. The update rules are adjusted synchronously based on the increase in cumulative sampling probability and the improvement in the latest local quality evaluation, so that recently high-quality paths have a stronger influence. The tree construction loop terminates when the computational budget is exhausted or the search values ​​of all root nodes converge.

[0035] The complete path with the largest search value at each root node is selected as the basis for feature extraction. Under the latent variable settings corresponding to the path, structured features are extracted from the original image blocks, including multi-scale gradient description, orientation consistency description, sparse keypoint response, channel attention-weighted semantic embedding, and boundary consistency features with neighboring blocks. For video data, under the same path settings, cross-frame trajectory features are extracted along the time axis. The trajectory is corrected using optical flow to eliminate motion-induced drift. The trajectory consistency score is written into the quality index of the path for subsequent candidate result ranking.

[0036] Based on path-driven features, multiple candidate recognition results are generated. The recognition task can be target category recognition, target location proposal, or instance segmentation contour estimation. For the category recognition task, a pre-trained classification head is input using path features, and the output is multiple high-confidence category labels and their confidence scores. For the location proposal task, the displacement components of the path in the latent space and the multi-scale jump records in the pixel domain are combined to backtrack to the original image coordinate system, generating a set of rectangular box proposals. The proposals are scored using observation consistency index and boundary consistency features.

[0037] For instance segmentation tasks, a mask decoding head is driven to generate candidate masks by utilizing path channel subset selection and orientation consistency description. Boundary stitching and stitching are performed using consistency metrics of inter-block overlap areas to eliminate the fragmentation effect caused by block partitioning. A comprehensive score is calculated for all candidate results, consisting of three parts: classification confidence, geometric reasonableness of location or mask, and a penalty term for consistency with prior knowledge, all of which are normalized. Several candidate results are output in descending order of comprehensive score, and each candidate is provided with localization information mapped to the coordinates of the original image or original video frame.

[0038] The overlapping sliding window for block partitioning can be implemented by iterating through the row and column indices in two layers. If the boundary is not a complete block, it is padded with mirror images. The encoding network can use a structure containing multiple convolutions and normalization, and outputs the prior parameters in vector form. The dataset for the training phase is prepared according to the real task scenario. The loss function includes a difference term and an observation consistency term. The optimization adopts a staged strategy: first, the discriminator scorer is frozen, the encoding network is converged separately, and then joint fine-tuning is performed.

[0039] Optical flow estimation can utilize a pyramid cascade structure, outputting pixel displacement fields and their reliability. Occlusion detection is achieved using forward-backward consistency checks. The preference score for candidate actions is constructed by multiplying or adding three parts, derived from prior center proximity, scale fit, and shape matching. Temperature exploration employs a time decay table, implemented by querying a pre-defined table based on iteration steps or generating it through an exponential decay function. The delayed acceptance coarse-screening stage can use a fast approximate scorer, taking low-resolution blocks and low-dimensional features as input and outputting a binary decision on whether to proceed to fine evaluation. Fine evaluation then invokes the complete observation consistency scorer.

[0040] The sampling path's record structure is a sequential array, containing the action index, proposal probability, acceptance event, and local quality score for each step. At the end of the path, these entries are read from beginning to end, the sampling probability is calculated, and written to the search tree node. The search tree node's data structure includes an array of child node pointers, an access count, cumulative sampling probability, cumulative quality score, and search value. In parallel execution, to avoid data races, atomic locking and segmentation strategies are used for node updates, or a per-thread local accumulation and phased merging approach is employed. Tree memory management uses a node pool and a free list for reuse, preventing fragmentation and thrashing caused by frequent allocations.

[0041] Post-processing of candidate results includes non-maximum suppression and cross-block merging. The overlap metric for location proposals uses an intersection-union (IU) threshold to filter redundant boxes. Mask results are merged within overlapping regions using pixel-level consistency voting, and boundaries are refined. For video data, temporal trajectory association is performed on candidate results from adjacent frames, and transient spurious responses are eliminated using trajectory consistency scoring constraints. The final output is returned as a structured list, with each result containing type, location or mask, overall score, associated block identifier, and frame index information.

[0042] In engineering deployment, batching and streaming concurrency strategies are adopted for large-scale image or video processing. Batching aggregates multiple blocks or frames into a single forward and scoring call, reducing overhead. Streaming concurrency, through queued decoupling, allows variational prior generation, sampling path generation, search tree expansion, and feature extraction to work in a pipeline within independent execution units. To ensure repeatability, all processes involving randomness are uniformly driven by a seedable pseudo-random number engine, with the seeding strategy combining the hash and timestamp of the input data. For ease of debugging and evaluation, the entire process outputs configurable intermediate products at key nodes, including prior parameter heatmaps, sampling path coverage maps, search tree statistical summaries, and confidence distributions of candidate results, all exported in the form of structured files and visual images.

[0043] To illustrate an application of a recognition task, the input is a video clip containing multiple moving targets. Following the process described above, each frame is divided into blocks and priors are generated. In blocks with relatively clear textures, the priors present concentrated high-support regions, and the sampled paths appear more frequently within these regions. The search tree quickly converges to a small number of high-search-value paths in these blocks. The trajectory features extracted based on these paths remain stable over time. The final generated candidate location proposals are spatially distributed around the real targets. After non-maximum suppression, a small number of high-confidence bounding boxes are output.

[0044] In blocks with weak texture or occlusion, the prior is more dispersed, the sampling path is explored more extensively, and the search tree maintains sufficient coverage by increasing the exploration weights and suppresses noise by using the consistency constraints of adjacent frames. The output candidate results have lower confidence but are still fully recorded for subsequent fusion. The entire process meets the implementation requirements for sampling and Monte Carlo tree search based on probability distribution functions for image or video data. The disclosed structure, data organization, and operation steps can be directly used by those skilled in the art to build corresponding prototypes and put them into application.

[0045] In one optional implementation, a probability sampling space is constructed based on the prior probability distribution, Markov chain Monte Carlo sampling is performed in the probability sampling space to generate multiple sampling paths, and the sampling probability of each sampling path is calculated, including: A probability density function is constructed for the prior probability distribution, and the prior probability distribution is smoothed by the kernel density estimation method. A probability sampling space is constructed based on the probability density function, and the probability sampling space includes the support set and transition probability matrix of the probability distribution. A Markov chain state transition network is constructed in the probability sampling space, and the state transition network includes a state space and state transition rules. A random walk sequence is generated using the state transition rule, and Monte Carlo sampling is performed based on the random walk sequence to generate multiple sampling paths, each sampling path corresponding to a transition trajectory in the state space; Calculate the sampling probability of each sampling path, which is obtained based on the state transition probability and the stationary distribution; output the multiple sampling paths and their corresponding sampling probabilities.

[0046] When processing prior probability distributions, the original set of data points is used as input, reflecting the prior knowledge of the phenomenon to be modeled. For the input data points, a smooth probability density function is constructed using kernel density estimation. In the kernel density estimation process, a Gaussian kernel function is chosen as the smoothing kernel, and a bell-shaped curve is placed at each data point. The bandwidth parameter is determined using the least squares cross-validation criterion; the bandwidth value that minimizes the error is selected by calculating the prediction error under different bandwidths. Specifically, the dataset is divided into a training set and a validation set. Kernel density estimates with different bandwidths are constructed on the training set, and the prediction error is calculated on the validation set. The bandwidth corresponding to the minimum error is selected.

[0047] After the probability density function is constructed, the support set of the probability distribution is determined. The boundary of the support set is determined by analyzing the effective region of the probability density function. A density threshold of one-thousandth of the total density peak value is set, and regions with density values ​​below this threshold are excluded from the support set. For multidimensional cases, boundary ranges are determined in each dimension, forming multidimensional rectangular regions as the support set. Discretization is performed within the support set, using a uniform grid partitioning method to transform the continuous space into a discrete state space. The choice of grid spacing needs to balance computational accuracy and computational complexity, and is usually set to one-hundredth to one-thousandth of the support set range.

[0048] The transition probability matrix is ​​constructed based on the transition likelihood between adjacent states. For each state in the grid, its transition probability with all neighboring states is calculated. The calculation of the transition probability considers two factors: the ratio of the spatial distance between states to the target probability density. The spatial distance uses a Euclidean distance metric; states that are closer together have a higher transition probability. The probability density ratio reflects the tendency to transition from the current state to the target state; states with higher densities are more likely to be transitioned. These two factors are combined, and normalization ensures that the sum of the out-degree transition probabilities for each state is one.

[0049] In the construction of a Markov chain state transition network, the state space consists of discretized grid points, each representing a state. The connections between states are determined based on adjacency rules, employing either von Neumann adjacency or Moore adjacency. Von Neumann adjacency only considers states directly adjacent to the current state along each coordinate axis, while Moore adjacency also includes adjacent states along the diagonal. The choice of adjacency affects the connectivity and sampling efficiency of the Markov chain; Moore adjacency provides more transition options but has relatively higher computational complexity.

[0050] The state transition rule is designed based on the density information of the target distribution and the current state information. The transition rule consists of two parts: a proposal mechanism and an acceptance mechanism. The proposal mechanism selects candidate states from the adjacent states of the current state, which can be selected through uniform random selection or weighted selection based on transition probabilities. The acceptance mechanism decides whether to accept the proposed state based on the ratio of the target probability density. Specifically, it calculates the density ratio of the proposed state to the current state; if the ratio is greater than or equal to one, the proposed state is accepted directly; if the ratio is less than one, the proposed state is used as the acceptance probability for random decision-making.

[0051] The generation of the random walk sequence begins iteratively with an initial state. The initial state can be any state within the support set, chosen randomly, or a state with a high probability density. In each iteration, the next state is generated based on the current state and the state transition rules. To ensure the memoryless nature of the Markov chain, the selection of the next state depends only on the current state and is independent of the historical state sequence. The iterative process continues until the preset sequence length is reached or the convergence condition is met.

[0052] Monte Carlo sampling generates multiple sampling paths by running a random walk process independently multiple times, using a different random seed in each run to ensure independence between paths. The number of sampling paths is determined by the required statistical precision, typically requiring hundreds to thousands of paths to obtain stable statistical results. Each path records a complete state transition sequence from the initial state to the final state, and the path length needs to be long enough to ensure a stationary distribution of the sampling chain.

[0053] The quality assessment of the sampling path is conducted using multiple statistical indicators. Stationarity is determined by comparing the statistical characteristics of the first and second halves of the path, including mean, variance, and distribution shape. If the statistical characteristics of the two halves are significantly different, it indicates that the chain has not yet reached a stationary state, requiring an increase in path length or adjustment of sampling parameters. Mixture is assessed by calculating the statistical differences between multiple paths generated from different starting points. Good mixture is characterized by paths from different starting points eventually converging to similar statistical characteristics.

[0054] The sampling probability is calculated based on the probability of each state transition in the path and the probability of each state in a stationary distribution. For a sampling path containing several states, the transition probabilities between adjacent states are calculated one by one. These transition probabilities can be directly obtained from a pre-constructed transition probability matrix. The stationary distribution probability is calculated by evaluating the density value of each state in the target probability density function. The density value needs to be normalized to ensure probabilistic properties. The total sampling probability of the path is obtained by multiplying all transition probabilities by the stationary distribution probability. To avoid underflow in numerical calculations, a logarithmic probability form is used for accumulation in actual calculations.

[0055] The output format of the sampling path includes the path's state sequence and corresponding sampling probabilities. The state sequence is stored as an array, recording the state identifier or state coordinates at each time step in the path. The sampling probabilities are given as floating-point numbers, reflecting the relative importance of the path among all paths. To facilitate subsequent analysis, the output also includes statistical summary information about the path, such as path length, state access frequency, and average state value. This statistical information helps evaluate sampling quality and perform path selection, ensuring that the paths used for subsequent inference are representative and reliable.

[0056] In one optional implementation, each node of the Monte Carlo search tree contains a state value function and an action policy function. The state value function and action policy function of the node are iteratively updated in an asynchronous parallel manner. Feature extraction is performed by selecting the search path with the largest search value, including: Each node in the Monte Carlo search tree is configured with a state value function and an action policy function. The state value function uses a residual network structure to calculate the value evaluation score of the node state, and the action policy function generates the action selection probability distribution of the node based on the dual policy optimization criterion. Multiple parallel worker processes are started to calculate the confidence upper bound of the node, the child node with the largest confidence upper bound is selected for expansion, the asynchronous stochastic gradient descent algorithm is used to update the node's state value function and action policy function, and the updated value assessment score and action selection probability distribution are broadcast to other worker processes through an asynchronous message passing mechanism. The Monte Carlo search tree is iteratively searched based on a preset search depth. In each iteration, the search path with the largest search value is selected, where the search value is the weighted average of the value assessment score and the normalized probability distribution of the action selection.

[0057] like Figure 2 As shown, the method includes: In the node configuration process of the Monte Carlo search tree, each node is assigned two core functional components: a state value function and an action policy function. The state value function is implemented using a residual network structure, which contains multiple residual blocks. Each residual block consists of a convolutional layer, a batch normalization layer, and an activation function, and the input is directly passed to the output through skip connections. Specifically, the input state information undergoes feature extraction through an initial convolutional layer, followed by deep feature learning through sixteen residual blocks. Each residual block contains two 3x3 convolutional layers with a total of 256 kernels. Skip connections fuse the block's input with the convolutional output through element-wise addition, effectively mitigating the gradient vanishing problem in deep networks. The final output layer of the network uses a fully connected layer structure, converting the feature mapping into a single value assessment score, which reflects the expected reward in the current state.

[0058] The action policy function is designed based on a dual-policy optimization criterion, comprising two components: an exploration strategy and an exploitation strategy. The exploration strategy introduces a noise mechanism to encourage the algorithm to explore the underutilized action space. The noise is generated using a Dirichlet distribution, with distribution parameters adjusted according to the dimension of the action space. The exploitation strategy selects the optimal action based on historical access statistics and value assessment results, evaluating the quality of each action by calculating its cumulative reward and access count. The fusion of the dual strategies is achieved through an adaptive weighting mechanism. The weights are dynamically adjusted based on the number of node accesses, favoring the exploration strategy when access counts are low and favoring the exploitation strategy when access counts are high. The output of the policy function is a probability distribution, with each action corresponding to a selection probability, and the sum of all probabilities equal to one.

[0059] Parallel worker processes are launched using a multi-threaded or multi-process architecture. The number of processes is determined based on the available computing resources, typically set to one to two times the number of processor cores. Each worker process maintains an independent copy of the search tree and is responsible for updating a specific subtree. Inter-process communication is achieved through shared memory or message queue mechanisms to ensure coordination and synchronization between different processes. Load balancing of worker processes is achieved through a dynamic task allocation algorithm, which monitors the computational load of each process and assigns new computational tasks to processes with lighter loads.

[0060] The calculation of the confidence upper bound comprehensively considers the node's value assessment and exploration potential. During the calculation, the average value assessment score of the node is added to the exploration reward to obtain the confidence upper bound. The exploration reward is calculated based on the number of visits to the parent node and the number of visits to the current node; the square root of the ratio of visit counts is multiplied by a preset exploration constant to obtain the exploration reward. The value of the exploration constant affects the algorithm's exploration level; a larger constant value encourages more exploration, while a smaller constant value favors utilizing known information. The child node with the largest confidence upper bound is selected for further expansion. If this node is a leaf node, a new child node is created; otherwise, the search continues downwards.

[0061] The asynchronous stochastic gradient descent algorithm employs a mini-batch gradient update approach. Each worker process maintains an independent gradient accumulation cache, triggering parameter updates when the cache reaches a preset size. Gradient calculation is based on the error between the current node's state information and the target value. The gradient of the state-value function is calculated using the mean squared error loss function, and the gradient of the action-policy function is calculated using the cross-entropy loss function. An adaptive adjustment strategy is used for the learning rate, dynamically adjusting its size based on historical gradient statistics. The learning rate is decreased when gradient changes drastically and increased when gradient changes are gradual. Parameter updates utilize a momentum mechanism, combining historical and current gradient information for a weighted average to improve the stability of parameter updates.

[0062] The asynchronous message passing mechanism achieves parameter synchronization through an event-driven approach. When a worker process completes a parameter update, it encapsulates the updated parameters into a message and broadcasts it to other processes. The message content includes a node identifier, the updated value assessment score, the action selection probability distribution, and an update timestamp. The receiving process determines the message's freshness based on the timestamp and only accepts messages newer than its local parameters. To avoid message storms, a rate limiting mechanism controls the message sending frequency, setting a minimum update interval to prevent frequent small updates from causing network congestion. Message passing uses a non-blocking method; the sending process continues executing its computational task without waiting for confirmation, improving overall computational efficiency.

[0063] The iterative search process is controlled based on a preset search depth, typically set to ten to fifty layers, adjusted according to problem complexity and computational resources. Each iteration starts from the root node and searches downwards along the path with the largest search value until a leaf node is reached or the search depth limit is reached. All nodes traversed during the search are recorded, forming a complete search path. When a leaf node is reached, if it is a terminated state, the state value is returned directly; otherwise, a value estimate is obtained through neural network evaluation.

[0064] The search value is calculated by normalizing the value assessment score and the action selection probability distribution, followed by a weighted merging. The value assessment score is normalized using a minimax normalization method, mapping the score to a range of zero to one. The action selection probability distribution is normalized to ensure the sum of all action probabilities is one, and the entropy of the distribution is calculated as a measure of policy uncertainty. The weighted merging uses a linear weighting method, with the value assessment score weighted at 0.7 and the action probability entropy value weighted at 0.3. These weights can be adjusted according to specific application scenarios. A higher search value indicates higher potential value for the path, and paths with the highest search values ​​are prioritized for in-depth exploration.

[0065] The feature extraction process extracts key information from the selected search path for subsequent decision-making. The extracted features include the state representations of each node on the path, action sequences, value assessment trends, and policy probability distribution characteristics. State representations employ vectorized encoding to convert complex state information into numerical vectors. Action sequences record all actions performed on the path, forming action trajectories. Value assessment trends are obtained by calculating the value differences between adjacent nodes, reflecting the changing patterns of the value function during the search process. Policy probability distribution characteristics include the mean, variance, and entropy of the distribution, describing the concentration and uncertainty of the policy. These extracted features provide crucial information for subsequent decision optimization and policy improvement, helping the algorithm continuously improve its search strategy and value assessment accuracy.

[0066] In one alternative implementation, configuring state value functions and action policy functions for each node of the Monte Carlo search tree includes: A multi-layer residual connection network is constructed, and skip connection paths are set in the residual connection network. The skip connection paths are used to fuse feature representations at different levels. The value evaluation score of the node state is calculated through feature fusion. An action transition probability matrix is ​​constructed using the value assessment score, and the action transition probability matrix describes the probability distribution of node state transitions; Two optimization objectives are set for the action transition probability matrix: online policy gradient and offline value estimation. The action selection strategy is optimized alternately using a dual-policy iterative approach to generate the action selection probability distribution of the node.

[0067] A multi-layer residual connection network structure is constructed for each node of a Monte Carlo search tree. This network consists of an input layer, multiple residual blocks, and an output layer. Each residual block contains two convolutional layers and a batch normalization layer, and skip connections are set to fuse feature representations from different layers. Specifically, the input layer receives the node state representation, which can be an 8×8×16 feature tensor representing the current game state. The network contains 20 residual blocks. The first convolutional layer of each residual block uses 256 3×3 convolutional kernels, and the second convolutional layer also uses 256 3×3 convolutional kernels. In each residual block, the input features pass through the first convolutional layer and the batch normalization layer, then through the second convolutional layer, and are then added to the original input features to form the skip connection structure. This design allows the network to retain feature information from earlier layers during forward propagation, effectively alleviating the gradient vanishing problem in deep network training.

[0068] To fuse feature representations from different levels, this embodiment includes additional skip connections after the 5th, 10th, and 15th residual blocks. These connections reduce the dimensionality of the output features of the corresponding residual blocks using 1×1 convolutional layers before concatenating them with the output features of the final residual block. For example, the output features of the 5th residual block are processed by a convolutional layer with 128 1×1 kernels to obtain a feature map of size 8×8×128; similarly, the outputs of the 10th and 15th residual blocks undergo similar processing. Finally, these three processed feature maps are concatenated with the output features of the 20th residual block (size 8×8×256) to form a fused feature representation of size 8×8×640.

[0069] When calculating the value assessment score of a node state through feature fusion, this embodiment employs a two-branch network: a value assessment branch and a policy branch. The value assessment branch first reduces the dimensionality of the fused features through a convolutional layer with 128 1×1 kernels, resulting in a feature map of size 8×8×128. Subsequently, global average pooling is used to convert the feature map into a 128-dimensional vector. This vector is then processed through two fully connected layers. The first fully connected layer contains 64 neurons and uses the ReLU activation function, while the second fully connected layer outputs a scalar value representing the value assessment score of the current node state. For example, for a moderately advantageous state, the value assessment score is 0.65, indicating that the current player has a 65% probability of winning.

[0070] When constructing the action transition probability matrix using the value assessment score, the policy branch processes the same fused features through a convolutional layer with 256 1×1 kernels, resulting in a feature map of size 8×8×256. This feature map is then passed through a convolutional layer with the number of output channels equal to the number of actions (e.g., for Go, 362 output channels represent 361 board positions plus one Pass action). The resulting feature map is flattened into a vector and normalized using the Softmax function to generate the action transition probability matrix. For example, in a specific state, the probability of placing a stone at a key position is 0.15, while the probabilities of other suboptimal positions are 0.08, 0.07, etc., and the sum of the probabilities of all positions is 1.

[0071] For the two optimization objectives of online policy gradient and offline value estimation set for the action transition probability matrix, this embodiment adopts a dual-policy iterative approach for alternating optimization. First, online policy gradient optimization generates training data through self-play, with each game producing a series of state-action pairs and the final game result. For each state st, the actual chosen action at and the final game result z (+1 indicates victory, -1 indicates defeat) are recorded. The loss is calculated using the difference between the action probability distribution π(a|st) predicted by the policy network and the actual chosen action at, and the difference between the predicted state value v(st) and the actual game result z. Specifically, the total loss function includes policy loss (using cross-entropy) and value loss (using mean squared error), as well as a regularization term. For example, for a training sample with a batch size of 256, the policy loss weight is set to 1.0, the value loss weight is set to 0.5, and the L2 regularization coefficient is set to 0.0001.

[0072] Offline value estimation optimization is trained using data augmented with Monte Carlo tree search. During self-play, each state `st` records not only the actual chosen action `at` and the final game result `z`, but also the policy `πMCTS(a|st)` obtained through Monte Carlo tree search. During training, the action probability distribution `π(a|st)` predicted by the policy network is no longer compared with the actual single chosen action `at`, but rather with the entire policy distribution `πMCTS(a|st)`, using KL divergence as the loss function. For example, for the same state, the initial policy network assigns a probability of 0.12 to a certain position, but after 1600 Monte Carlo simulations, the probability of that position becomes 0.25. The network learns a better policy by minimizing this difference.

[0073] Through alternating optimization of online policy gradients and offline value estimation, network parameters are continuously updated, generating increasingly accurate node state value assessments and action selection probability distributions. In practical applications, after 10,000 games of self-play training, the prediction accuracy of the policy network increased from the initial 35% to 67%, and the mean squared error of state value assessment decreased from 0.42 to 0.18, significantly improving the decision-making quality and efficiency of Monte Carlo search trees.

[0074] In one optional implementation, the confidence upper bound of the node is calculated, the child node with the largest confidence upper bound is selected for expansion, and the asynchronous stochastic gradient descent algorithm is used to update the node's state function and action policy function, including: Obtain the state information of the node to be expanded, and extract the feature vector from the state information; A set of child nodes is constructed based on the feature vectors, and the historical access count and historical cumulative revenue of each child node are counted. Calculate the confidence interval parameters based on the historical access count and historical cumulative revenue, and input the confidence interval parameters into the adaptive estimator; Thompson sampling is performed using the adaptive estimator to obtain the upper confidence bound of each child node in the current state. The upper confidence bound is used to balance exploration and exploitation during the node expansion process. The child nodes are sorted according to the upper confidence bound value, and the child node with the largest upper confidence bound value is selected as the expansion target node. The feature vectors of the extended target nodes are mapped to the value space and the policy space to construct a distributed computing network. Asynchronous stochastic gradient descent is performed in the distributed computing network to optimize the state value function and action policy function in parallel. The state representation of the extended target node is updated based on the optimization results, and the updated state value function and action policy function are output.

[0075] During the acquisition of state information for nodes to be expanded, a complete description of the current state is read from the node's storage structure. State information includes the physical configuration of the environment, historical action sequences, time step information, and other relevant contextual data. For chess games, state information includes the chessboard configuration matrix, the current round, and the number of moves already executed; for path planning scenarios, state information includes the current position coordinates, obstacle distribution, and target location. The encoding of state information employs a multi-level structure, numerically representing different types of information to form a unified state vector format.

[0076] Feature vector extraction is achieved through multi-dimensional feature engineering. Spatial features are extracted using convolutional neural networks, converting two-dimensional or three-dimensional state representations into fixed-length feature vectors. Temporal features are extracted using recurrent neural networks or long short-term memory networks to capture the temporal dependencies in the state evolution process. Statistical features are obtained by calculating statistics such as the mean, variance, maximum, and minimum values ​​of the state vectors. Semantic features are mapped from discrete state elements to continuous vector representations using a pre-trained embedding model. All feature vectors are combined into the final state feature representation through concatenation or weighted fusion. The dimension of the feature vectors is typically set between 128 and 512.

[0077] The child node set is constructed based on the legal action space of the current state. For each executable action, a corresponding child node is created and the state after executing the action is calculated. The state of a child node is calculated using a state transition function, which determines the next state based on the current state and the executed action. During the construction process, the legality of actions needs to be verified to exclude actions that violate environmental constraints or game rules. The identifier of a child node is generated using a state hash value to ensure that nodes with the same state have a unique identifier and avoid duplicate creation.

[0078] Historical access counts and historical cumulative revenue are statistically analyzed using an access counter and a revenue accumulator. Each node maintains independent statistical information, including the total number of accesses, the immediate revenue gained per access, the total cumulative revenue, and the average revenue. Access count updates employ atomic operations to ensure data consistency during concurrent accesses. Revenue information records include detailed timestamps and source information for easy statistical analysis and anomaly detection. For newly created child nodes, the initial access count is set to zero, the initial cumulative revenue is set to zero, and the average revenue is set to a preset default value.

[0079] The confidence interval parameters are calculated based on Bayesian statistical principles. For each child node, the parameter estimate of the revenue distribution is calculated based on historical visit counts and cumulative revenue. Assuming the revenue follows a normal distribution, the distribution parameters are estimated using the sample mean and sample variance. The sample mean equals cumulative revenue divided by the number of visits, and the sample variance is calculated by dividing the sum of squared revenue deviations by the number of visits minus one. The width of the confidence interval is inversely proportional to the square root of the number of visits; the more visits, the narrower the confidence interval, reflecting the degree of certainty in the revenue estimate. The confidence level is set to 95%, and the corresponding critical value is obtained through table lookup or numerical calculation.

[0080] The adaptive estimator is constructed using the Thompson sampling framework of a multi-armed slot machine. The estimator maintains prior parameters for the payoff distribution of each child node, including prior distribution parameters for the mean and variance. The prior distribution adopts a normal-inverse gamma distribution, which is the conjugate prior of the normal distribution, facilitating Bayesian updates. The adaptability of the estimator is reflected in its dynamic adjustment of the prior parameters based on new observation data, ensuring that the estimation results better reflect the actual payoff distribution. Parameter updates employ Bayesian rules, combining the prior distribution with the observation likelihood to obtain the posterior distribution.

[0081] During Thompson sampling, samples are randomly drawn from the posterior distribution of each child node as the estimated reward for that node. The sampling process employs a standard random number generation algorithm to ensure the randomness and statistical properties of the samples. For a normal distribution, Box-Muller transform or Ziggurat algorithm is used to generate normal random numbers. The estimated reward obtained from the sampling serves as the upper confidence bound for that child node in its current state, reflecting the expected reward considering uncertainty. The calculation of the upper confidence bound also considers an exploration reward term, which is proportional to the logarithm of the number of times the node is visited, encouraging the algorithm to explore nodes that are visited less frequently.

[0082] Child nodes are sorted in descending order based on their upper confidence bounds. The sorting algorithm used is either quicksort or mergesort, ensuring a linear-logarithmic time complexity. During sorting, if two nodes have the same upper confidence bound, a secondary sorting criterion is used to distinguish them, such as creation time or lexicographical order of node identifiers. After sorting, the child node at the top is selected as the target node for expansion; this node has the highest upper confidence bound, representing the most promising direction for exploration.

[0083] The mapping from feature vectors to the value space and policy space is achieved through neural network transformations. The value space mapping uses a fully connected network to map feature vectors into scalar value outputs. The network structure contains three to five hidden layers, each with 256 to 512 neurons, and uses ReLU or a variant thereof as the activation function. The policy space mapping also uses a fully connected network, but the output dimension is equal to the action space size, and the output layer uses a softmax activation function to ensure that the output is a probability distribution. The two mapping networks can share the first few feature extraction layers, reducing the number of parameters and improving training efficiency.

[0084] The distributed computing network is constructed using either a parameter server architecture or a decentralized ring communication architecture. In the parameter server architecture, dedicated parameter server nodes store and update global parameters, while worker nodes retrieve the latest parameters from the parameter server and upload gradient updates. In the decentralized architecture, each computing node maintains a copy of the parameters, achieving parameter synchronization through communication with neighboring nodes. Network fault tolerance is ensured through redundant design and fault detection mechanisms, allowing other nodes to continue operating normally when one node fails.

[0085] The asynchronous stochastic gradient descent operation employs an asynchronous parallel strategy. Each worker process independently calculates the gradient and updates parameters asynchronously, without waiting for other processes to complete their calculations. Gradient calculation is based on mini-batch samples, with the batch size determined by memory capacity and computing power, typically set between 32 and 256. An adaptive adjustment strategy, such as the Adam or RMSprop algorithm, is used to dynamically adjust the learning rate of each parameter based on historical gradient statistics. To avoid gradient explosion, gradient clipping techniques are employed to limit the magnitude of the gradient within a preset threshold.

[0086] The parallel optimization of the state value function and action policy function employs a multi-task learning framework. The two functions share the underlying feature extraction network but use different output heads and loss functions. The optimization objective of the state value function is to minimize the mean squared error between the predicted value and the actual reward, while the optimization objective of the action policy function is to maximize the policy gradient or minimize the cross-entropy loss. The loss functions of the two tasks are combined using a weighted summation, with the weight parameters adjusted according to task importance and convergence speed. An alternating update strategy is used during optimization, randomly selecting one task for parameter updates in each iteration to balance the learning progress of the two tasks.

[0087] The state representation of the target node is updated based on optimized function parameters. During the update process, the new state value function is applied to the node's feature vector to calculate the updated value estimate. The action policy function is then applied to the feature vector to obtain the updated action selection probability distribution. Node statistics also need to be updated accordingly, including average value, visit count, and last update time. The updated state value function and action policy function are stored in a serialized format for easy retrieval and use later. Simultaneously, update information is communicated to relevant parent and child nodes via a message passing mechanism to maintain the consistency of the search tree structure.

[0088] In one optional implementation, a feature semantic map is constructed based on the multiple candidate recognition results. A graph neural network is used to iteratively propagate and calculate the node vector representations of the feature semantic map. The confidence score of each candidate recognition result is calculated based on the node vector representations. The candidate recognition result with the highest confidence score is selected as the final recognition result. The multiple candidate recognition results are projected onto the semantic feature space to obtain semantic feature representations. A feature semantic graph is constructed in the semantic feature space, where each candidate recognition result corresponds to a node in the feature semantic graph. Edge connection relationships are dynamically constructed based on the feature similarity between nodes. The feature semantic graph is subjected to multiple rounds of information propagation. Each round of propagation includes: calculating the importance weight of the node's neighborhood, aggregating the neighborhood node information based on the importance weight, nonlinearly fusing the aggregated information with the central node information, controlling the information update ratio through a gating mechanism, iterating until the node state converges, and obtaining the final node vector representation. The confidence score of each candidate recognition result is calculated based on the node vector representation; the candidate recognition result with the highest confidence score is taken as the final recognition result.

[0089] In the semantic feature space projection process of candidate recognition results, a pre-trained language model is used to encode each candidate result. For text-type candidate results, the text is converted into a sequence of word vectors through a word embedding layer, and then a context-related semantic representation is extracted through a bidirectional long short-term memory network or a transformer encoder. The encoder output is aggregated into a fixed-dimensional vector representation through average pooling or attention pooling mechanisms, with the vector dimension set to 512 or 768 dimensions. For image-type candidate results, a convolutional neural network is used to extract visual features. The network structure contains multiple convolutional blocks and pooling layers, and finally, a feature vector is obtained through global average pooling. The semantic feature space is constructed by mapping high-dimensional features to a low-dimensional space through principal component analysis or t-SNE dimensionality reduction techniques, while maintaining the relative distance relationship between features.

[0090] The semantic feature graph is constructed based on the semantic similarity relationships between candidate recognition results. In the graph structure, each candidate recognition result is a node, and the initial feature vector of each node is set as its projection representation in the semantic feature space. The edge connections between nodes are determined by calculating the cosine similarity between feature vectors. Edge connections are established between node pairs with similarity exceeding a preset threshold. The threshold is determined using cross-validation and is typically set between 0.5 and 0.8. For node pairs with high similarity, the edge weight is set to the similarity value; for node pairs with low similarity, no edge connection is established to reduce graph complexity. The adjacency matrix of the graph is stored in a sparse matrix format to improve storage efficiency and computational performance.

[0091] The construction of dynamic edge connections employs an adaptive threshold adjustment mechanism. In the initial stages of graph construction, a lower similarity threshold is used to ensure graph connectivity. As information propagates, the threshold is gradually increased to filter weakly correlated connections. Threshold adjustments are based on the graph's connectivity index and node degree distribution. When isolated nodes or connected components appear, the threshold is lowered to increase connections; when the graph is too dense and affects computational efficiency, the threshold is increased to reduce connections. Edge weight updates utilize an exponential decay mechanism. The newly calculated similarity is weighted and averaged with historical weights, with a decay factor set to 0.9.

[0092] In the multi-round iterative process of information propagation, each round includes four steps: neighborhood importance calculation, information aggregation, nonlinear fusion, and gating update. The calculation of neighborhood importance weights is based on an attention mechanism. For each neighboring node of the central node, the attention score between its feature vector and the central node's feature vector is calculated. The attention score is calculated through a multi-head attention mechanism, involving a linear transformation of the query vector, key vector, and value vector. The query vector is generated from the central node's features, while the key and value vectors are generated from the neighboring node features. The multi-head attention mechanism employs eight attention heads, each with a 64-dimensional dimension, focusing on different aspects of the features.

[0093] The aggregation of neighborhood node information is achieved using a weighted summation method. For each neighbor of the central node, the feature vector of the neighbor node is multiplied by its corresponding importance weight, and then all weighted feature vectors are summed to obtain the aggregated information. The aggregation process considers the influence of edge weights, using the attention weight multiplied by the edge weight as the final aggregation weight. To maintain numerical stability, the aggregation weights are normalized to ensure that the weights sum to one. The dimension of the aggregated information is consistent with the node feature vectors, facilitating subsequent fusion operations.

[0094] The nonlinear fusion process combines the aggregated neighborhood information with the original information of the central node. This fusion operation is implemented using a multilayer perceptron, comprising two hidden layers and one output layer. The number of neurons in the hidden layers is set to twice the feature dimension, and the activation function is either ReLU or GELU. The input layer receives the concatenated feature vector, containing features from the central node and aggregated neighborhood features. The parameters of the fusion network are trained using backpropagation, and the loss function is either mean squared error or cross-entropy loss. The fused feature vector serves as an intermediate representation of the nodes for subsequent gating updates.

[0095] The gating mechanism employs a gated recurrent unit structure, comprising an update gate and a reset gate. The update gate controls the degree to which new information influences the node state, while the reset gate controls the degree to which historical information is retained. The gates are calculated using the sigmoid activation function, with inputs being the current node's features and the concatenated feature vector. The output of the update gate ranges from zero to one; a larger value indicates the adoption of more new information, while a smaller value indicates the retention of more historical information. The reset gate selectively forgets historical information during candidate update calculations, avoiding information redundancy and the vanishing gradient problem.

[0096] The convergence of node states is determined based on the change in the node's feature vector between two consecutive iterations. For each node, the Euclidean distance between the feature vector of the current iteration and that of the previous iteration is calculated. Convergence is considered achieved when the distances between all nodes are less than a preset threshold. The convergence threshold is typically set between 0.01 and 0.001, with the specific value determined based on the application's precision requirements. To avoid infinite iterations, a maximum number of iterations is limited, typically between ten and fifty. Convergence checks are performed after each round of propagation, and the propagation process stops when the convergence condition or the maximum number of iterations is met.

[0097] The final node vector representation is obtained through node feature vectors after multiple rounds of information propagation. The converged node vector contains information about all relevant nodes in the graph, reflecting the relative position and importance of the candidate recognition result in the entire candidate set. The quality of the node vector is evaluated through visualization analysis and cluster analysis. High-quality node vectors should make semantically similar candidate results closer together in the vector space, and semantically different candidate results farther apart.

[0098] The confidence score is calculated based on node vector representations and a pre-trained confidence evaluation network. The network employs a multilayer perceptron structure, taking node vector representations as input and outputting a confidence score between zero and one. The network contains three hidden layers, each with 256 neurons, using ReLU activation. The output layer uses a sigmoid activation function to ensure the output remains within a reasonable range. The network's training data comes from historical recognition tasks, including feature representations of candidate results and their corresponding true confidence labels.

[0099] The confidence score calculation also considers the topological location of candidate results in the graph. Nodes with higher centrality generally have higher confidence scores because they are connected to more relevant candidate results. Centrality is calculated using a weighted combination of degree centrality and betweenness centrality. Degree centrality reflects the number of direct connections a node has, while betweenness centrality reflects the bridging role a node plays in the graph. The weights were determined experimentally, typically with a degree centrality weight of 0.6 and a betweenness centrality weight of 0.4.

[0100] The final recognition result is selected based on the overall confidence score ranking of the candidate results, with the candidate result having the highest confidence score being chosen as the final recognition result output. In cases of equal scores, secondary judgment criteria are used for selection, such as the candidate result's original score, generation time, or lexicographical order. The selection process also includes a rationality check to verify whether the final result meets the task's basic constraints, such as format requirements and length limits. After passing the check, the final recognition result and its corresponding confidence score are output, providing a reference for subsequent applications and analysis.

[0101] A second aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0102] A third aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0103] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image and video recognition and analysis based on edge-side multimodal large model, characterized in that, include: The image or video data to be processed is input into a multimodal large model pre-deployed on the device. The sampling probability of multiple sampling paths is calculated based on the probability distribution function. Monte Carlo tree search is performed on the image or video data according to the sampling probability. Each node of the Monte Carlo search tree contains a state value function and an action policy function. The state value function and action policy function of the node are updated iteratively in an asynchronous parallel manner. The search path with the largest search value is selected for feature extraction, and multiple candidate recognition results are generated. A feature semantic map is constructed based on the multiple candidate recognition results. The feature semantic map is iteratively propagated and calculated using a graph neural network to obtain node vector representations. The confidence score of each candidate recognition result is calculated based on the node vector representations. The candidate recognition result with the highest confidence score is selected as the final recognition result. The final recognition result is output to the display interface of the device, and the image scene information, target detection information and motion analysis information are presented on the display interface.

2. The method according to claim 1, characterized in that, Calculating the sampling probability of multiple sampling paths based on a probability distribution function, and performing a Monte Carlo tree search on the image or video data according to the sampling probability, includes: The image or video data is divided into multiple image blocks. The prior probability distribution of each image block is calculated based on the variational inference method. A probability sampling space is constructed according to the prior probability distribution. Markov chain Monte Carlo sampling is performed in the probability sampling space to generate multiple sampling paths, and the sampling probability of each sampling path is calculated. Based on the sampling probability, a Monte Carlo search tree is constructed for the image or video data. The search path with the largest search value in the Monte Carlo search tree is selected for feature extraction, and multiple candidate recognition results are generated.

3. The method according to claim 2, characterized in that, A probability sampling space is constructed based on the prior probability distribution. Markov chain Monte Carlo sampling is performed in the probability sampling space to generate multiple sampling paths, and the sampling probability of each sampling path is calculated, including: A probability density function is constructed for the prior probability distribution, and the prior probability distribution is smoothed by the kernel density estimation method. A probability sampling space is constructed based on the probability density function, and the probability sampling space includes the support set and transition probability matrix of the probability distribution. A Markov chain state transition network is constructed in the probability sampling space, and the state transition network includes a state space and state transition rules. A random walk sequence is generated using the state transition rule, and Monte Carlo sampling is performed based on the random walk sequence to generate multiple sampling paths, each sampling path corresponding to a transition trajectory in the state space; Calculate the sampling probability of each sampling path, which is obtained based on the state transition probability and the stationary distribution; output the multiple sampling paths and their corresponding sampling probabilities.

4. The method according to claim 1, characterized in that, Each node in the Monte Carlo search tree contains a state value function and an action policy function. The state value function and action policy function of each node are iteratively updated in an asynchronous parallel manner. The search path with the largest search value is selected for feature extraction, including: Each node in the Monte Carlo search tree is configured with a state value function and an action policy function. The state value function uses a residual network structure to calculate the value evaluation score of the node state, and the action policy function generates the action selection probability distribution of the node based on the dual policy optimization criterion. Multiple parallel worker processes are started to calculate the confidence upper bound of the node, the child node with the largest confidence upper bound is selected for expansion, the asynchronous stochastic gradient descent algorithm is used to update the node's state value function and action policy function, and the updated value assessment score and action selection probability distribution are broadcast to other worker processes through an asynchronous message passing mechanism. The Monte Carlo search tree is iteratively searched based on a preset search depth. In each iteration, the search path with the largest search value is selected, where the search value is the weighted average of the value assessment score and the normalized probability distribution of the action selection.

5. The method according to claim 4, characterized in that, The configuration of state value functions and action policy functions for each node in the Monte Carlo search tree includes: A multi-layer residual connection network is constructed, and skip connection paths are set in the residual connection network. The skip connection paths are used to fuse feature representations at different levels. The value evaluation score of the node state is calculated through feature fusion. An action transition probability matrix is ​​constructed using the value assessment score, and the action transition probability matrix describes the probability distribution of node state transitions; Two optimization objectives are set for the action transition probability matrix: online policy gradient and offline value estimation. The action selection strategy is optimized alternately using a dual-policy iterative approach to generate the action selection probability distribution of the node.

6. The method according to claim 4, characterized in that, Calculate the upper confidence bound of the node, select the child node with the largest upper confidence bound for expansion, and update the node's state value function and action policy function using the asynchronous stochastic gradient descent algorithm, including: Obtain the state information of the node to be expanded, and extract the feature vector from the state information; A set of child nodes is constructed based on the feature vectors, and the historical access count and historical cumulative revenue of each child node are counted. Calculate the confidence interval parameters based on the historical access count and historical cumulative revenue, and input the confidence interval parameters into the adaptive estimator; Thompson sampling is performed using the adaptive estimator to obtain the upper confidence bound of each child node in the current state. The upper confidence bound is used to balance exploration and exploitation during the node expansion process. The child nodes are sorted according to the upper confidence bound value, and the child node with the largest upper confidence bound value is selected as the expansion target node. The feature vectors of the extended target nodes are mapped to the value space and the policy space to construct a distributed computing network. Asynchronous stochastic gradient descent is performed in the distributed computing network to optimize the state value function and action policy function in parallel. The state representation of the extended target node is updated based on the optimization results, and the updated state value function and action policy function are output.

7. The method according to claim 1, characterized in that, A feature semantic map is constructed based on the multiple candidate recognition results. A graph neural network is used to iteratively propagate the feature semantic map to obtain node vector representations. The confidence score of each candidate recognition result is calculated based on the node vector representations. The candidate recognition result with the highest confidence score is selected as the final recognition result. The multiple candidate recognition results are projected onto the semantic feature space to obtain semantic feature representations. A feature semantic graph is constructed in the semantic feature space, where each candidate recognition result corresponds to a node in the feature semantic graph. Edge connection relationships are dynamically constructed based on the feature similarity between nodes. The feature semantic graph is subjected to multiple rounds of information propagation. Each round of propagation includes: calculating the importance weight of the node's neighborhood, aggregating the neighborhood node information based on the importance weight, nonlinearly fusing the aggregated information with the central node information, controlling the information update ratio through a gating mechanism, iterating until the node state converges, and obtaining the final node vector representation. The confidence score of each candidate recognition result is calculated based on the node vector representation; the candidate recognition result with the highest confidence score is taken as the final recognition result.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images