Method and system for improving intelligent decision-making ability based on multi-modal learning fusion

Abstract

The invention provides a method and a system for improving intelligent decision-making ability based on multi-modal learning fusion. The method comprises the following steps: acquiring chessboard image data, strategy text data and historical match chess manual data; processing the chessboard image data by using a convolutional neural network, and extracting spatial features; processing the Go related strategy text by using a natural language processing model, and extracting semantic features of the text; extracting historical chess manual characteristics, namely common strategies and decision modes, from the history game chess manual data; extracting features of different modes by using a multi-mode neural network model, and introducing an attention mechanism to fuse and weight the features of the different modes to generate comprehensive feature representation; based on the fused comprehensive feature representation, a comprehensive decision network model is used for strategy selection and optimization; the method can generate a more accurate and comprehensive Go strategy, not only improves the ability of Go AI to understand and cope with complex situations, but also has the application potential of future multi-modal information fusion in other strategic game fields.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and machine learning, and specifically relates to a method and system for improving intelligent decision-making capabilities based on multimodal learning fusion. Background Art

[0002] Multimodal fusion learning strategy technology refers to integrating multiple modal information from different sensors or data sources into a unified learning framework to enhance information processing and decision-making capabilities. The main technologies and methods are feature fusion: integrating features from different modalities to form a more comprehensive and rich feature representation. This can be achieved through simple feature connection, weighted summation, nonlinear transformation, etc.; representation learning: using deep learning techniques such as convolutional neural networks, recurrent neural networks, or autoencoders to learn high-level representations of each modality. These representations can be abstractions and compressions of raw data, which help to more effectively process and fuse multimodal information; integrated learning: integrating the prediction results of multiple learners or models from different modalities to improve overall performance. Integration methods can include voting, weighted averaging, stacking models, etc.; multi-task learning: learning multiple tasks simultaneously under a unified framework, each task may depend on different modal information. This method can extract modal shared information by sharing some neural network layers to enhance learning efficiency and generalization ability.

[0003] The existing application areas and advantages of multimodal fusion learning strategy technology are in computer vision, combining visual and language information for tasks such as image description generation and visual question answering. In natural language processing, text and voice information are integrated for tasks such as speech recognition and sentiment analysis. In intelligent systems, multimodal information is used to enhance robot perception and decision-making capabilities, such as autonomous driving and intelligent interactive systems.

[0004] As a complex strategic board game, Go has always been a hot topic in artificial intelligence research. Existing Go AI systems mainly rely on a single Go game mode, that is, only inputting the board state into the deep neural network for processing. Although this method has improved the performance of Go AI to a certain extent, there are still some technical problems:

[0005] Limited to a single modality: Strategy learning is only done through Go games, ignoring other valuable information sources such as text, historical game data, etc. This single-modality learning method limits the breadth and depth of the Go AI system's strategic decision-making, limiting its adaptability in dealing with different situations and strategy changes.

[0006] Lack of comprehensive use of multi-source information: Failure to effectively integrate and utilize information from different modalities for decision-making. The single source of information leads to a single and stereotyped strategy, making it difficult to make innovative and effective decisions when faced with new and unseen situations.

[0007] Insufficient information extraction and feature representation: Existing methods have limited information extraction and feature representation capabilities in complex situations, making it difficult to cope with complex strategies in high-level games. In a complex game environment, key information may be scattered across multiple data sources and multiple time points, and existing methods find it difficult to effectively integrate this information to make accurate decisions.

[0008] Technical problem solving: (1) Single modality limitation: By introducing text and historical chess data, we can break through the limitation of relying on a single modality and improve our ability to understand complex situations. (2) Insufficient comprehensive utilization of information: Through multimodal information fusion, we can achieve comprehensive utilization of visual, text and historical chess data to improve decision-making quality. (3) Insufficient feature representation: We can use multimodal neural networks and attention mechanisms to enhance information extraction and feature representation capabilities and improve our ability to respond to complex situations. Summary of the invention

[0009] In order to solve the problems existing in the prior art, the present invention provides a method for improving intelligent decision-making ability based on multimodal learning fusion to solve the above problems in the prior art. By introducing multimodal learning, multi-source information such as vision, text and historical chess data can be comprehensively utilized to improve the decision-making ability and strategy diversity of Go AI.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is: a method for improving intelligent decision-making ability based on multimodal learning fusion, comprising the following steps:

[0011] Obtain chessboard image data, strategy text data, and historical game record data;

[0012] Use convolutional neural networks to process chessboard image data and extract spatial features; use natural language processing models to process Go-related strategy texts and extract text semantic features; extract historical game features, i.e. common strategies and decision-making patterns, from historical game data;

[0013] Use a multimodal neural network model to extract features of different modalities, introduce an attention mechanism to fuse and weight features of different modalities, and generate a comprehensive feature representation;

[0014] Based on the fused comprehensive feature representation, a comprehensive decision network model is used for strategy selection and optimization.

[0015] Furthermore, obtaining chessboard image data, strategy text data, and historical game record data includes:

[0016] Denoising, enhancing, resizing and normalizing chessboard image data, which includes various chessboard states in different games, normalizing image pixel values ​​and adjusting all images to the same size;

[0017] Use the natural language processing model to parse and process the strategy text data, clean and annotate the game records and comments, perform word segmentation and grammar analysis, remove irrelevant characters, punctuation marks, and HTML tags, and use the word segmenter provided by the BERT model to split the strategy text data into words or subwords;

[0018] Format the historical game record data into a unified structure and convert the game record data into a two-dimensional chessboard state matrix, where each element represents the chess piece at the corresponding position.

[0019] Furthermore, the chessboard image data processing in multimodal information fusion includes:

[0020] The chessboard image data is input into the convolutional neural network. After multiple layers of convolution and pooling operations, the spatial features of the chessboard state are extracted. The convolutional neural network architecture: set the input layer, set the dimension of the input image, set the convolution layer, apply multiple convolution kernels to perform convolution operations on the input image, extract local features in the image, set the convolution kernel size, the number of convolution kernels, use the ReLU activation function, and each convolution operation generates a feature map;

[0021] The feature map is pooled in the pooling layer, and the maximum pooling is used to reduce the dimension of the feature map. Multiple convolutional layers and pooling layers are stacked alternately to flatten the feature map after multiple convolutions and pooling, and convert it into a one-dimensional vector. One or more fully connected layers are used to further combine and process the extracted features.

[0022] Use the cross entropy loss and Adam optimizer in the classification task to train the convolutional neural network model, set the evaluation index accuracy to evaluate the model performance, use the training data set to train the model, and iteratively update the model parameters;

[0023] The Go board image data is processed by a convolutional neural network to obtain the spatial characteristics of the board state.

[0024] Furthermore, the strategic text data processing in multimodal information fusion includes:

[0025] Input the policy text data into the BERT model, load the BERT tokenizer to tokenize and format the policy text data, input the formatted input data into the BERT model for processing, and the output includes the hidden states of all layers and the pooled output of the last layer. Using the hidden state of the last layer of the BERT model or the hidden states of all layers, the pooled output can be used as the semantic representation of the entire policy text data.

[0026] Furthermore, the historical data processing steps include:

[0027] The input layer obtains historical chess data and preprocesses it to obtain a two-dimensional chessboard state matrix. Multi-layer convolution operations are performed on the two-dimensional chessboard state matrix to extract the spatial features of the chessboard; the feature map output by the convolution layer is flattened and connected to the fully connected layer for the final feature representation.

[0028] Furthermore, the multimodal neural network construction includes:

[0029] The image features, text semantic features and historical chess features are fused to form a comprehensive feature representation, and a comprehensive decision network is constructed based on the fused features. In the comprehensive decision network: convolutional neural network is used to process image data and extract spatial features; natural language processing is used to process text data and extract text semantic features; a dedicated network model structure is used to process chess data and extract chess game state features; a fully connected layer and an output layer are added to obtain a multimodal neural network model.

[0030] Furthermore, the introduction of attention mechanism into the multimodal neural network model includes:

[0031] Visual features, text features and historical chess features are prepared as inputs of the attention mechanism. All features are concatenated together to form a feature vector. The importance weight of each feature is calculated using dense layers and activation functions. Different features are integrated according to the calculated attention weights through the weighted summation method to form a comprehensive feature representation. The comprehensive feature representation is subsequently processed, and further fully connected layers and decision layers are added to obtain a multimodal neural network model that includes an attention mechanism.

[0032] Furthermore, based on the integrated feature representation after fusion, the strategy selection and optimization using the integrated decision network model includes:

[0033] Combine Monte Carlo tree search with policy network, dynamically adjust the parameters of policy network, introduce adversarial network, and simulate opponents of different levels;

[0034] Monte Carlo tree search is combined with the policy network. Starting from the root node, the tree structure is initialized. Each node stores the board state, number of visits, and total value information. Starting from the root node, child nodes are selected based on the upper confidence bound until a leaf node is reached.

[0035] During the game, adjust the exploration constant and learning rate; when training the policy network, introduce opponent data of different styles so that the policy network can learn and adapt to multiple strategies; evaluate the opponent's strategy changes in real time, and dynamically adjust the parameters of the policy network based on the evaluation results.

[0036] Furthermore, it also includes evaluating and adjusting the model through self-play and expert feedback:

[0037] Use multimodal data to evaluate strategies and ensure the consistency and effectiveness of the strategies in different data modalities.

[0038] Evaluation indicators: Win rate and average error rate are used to evaluate the strategic effect of the model;

[0039] Expert feedback: Invite Go experts to evaluate the model's strategy and provide feedback and suggestions;

[0040] Model adjustment: Adjust and optimize model parameters based on evaluation results and expert feedback.

[0041] On the other hand, based on the concept of the method, the present invention also provides a system for improving intelligent decision-making ability based on multimodal learning fusion, including an input module, a data processing module, a multimodal fusion module and a strategy generation module;

[0042] The input module is used to obtain the Go board image of the game, the chess record data of the historical game and the strategy text data;

[0043] The data processing module is used to pre-process the chessboard image data of the game, the chess record data of the historical game, and the strategy text data to obtain the chessboard image data, the chess record data of the historical game, and the strategy text data that meet the feature fusion requirements;

[0044] The multimodal fusion module is used to fuse the features of chessboard image data, historical game record data and strategy text data, comprehensively analyze the current game status, and integrate the feature vectors of chess record, image and text data into a unified feature representation through feature splicing and feature transformation;

[0045] The strategy generation module is used to generate strategy recommendations and improvement suggestions based on the unified feature representation, provide suggestions for the next move, and explain the recommended strategy.

[0046] Compared with the prior art, the present invention has at least the following beneficial effects: the present invention realizes the comprehensive utilization of multi-source information by integrating visual, textual and historical chess data, adopts multimodal neural network for information extraction and feature representation, and adopts attention mechanism and decision network for comprehensive decision-making based on the fusion results of multimodal information, thereby improving the diversity and adaptability of strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Flow chart of Embodiment 1 of the present invention

[0048] Figure 2 It is a structural diagram of the multimodal fusion network. DETAILED DESCRIPTION

[0049] The present invention is further described below with reference to the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0050] Example 1

[0051] In this embodiment, a multimodal Go learning AI system is constructed. The system is integrated into the Go teaching application and can analyze the user's game board in real time, provide strategy suggestions and improvement suggestions. The system uses image recognition technology to automatically identify the user's chess piece position, and combines chess record data and strategy text for comprehensive analysis, thereby improving the user's learning effect. The following is a detailed process description, see Figure 1 :

[0052] (1) System architecture

[0053] The multimodal learning Go AI system includes:

[0054] Input module: Users use cameras to capture images of the Go board in real-time, upload or input historical game records, and provide relevant strategy text data, such as chess reviews and game analysis.

[0055] Data processing module:

[0056] Image data processing, using image recognition technology to identify the position and type of chess pieces. Normalize the chessboard image data, perform denoising, enhancement, resizing and normalization. The image should include various chessboard states in different games, normalize the image pixel values ​​to the range of [0, 1], and adjust all images to the same size.

[0057] Game record data processing: parsing and processing game record data, extracting game status, eliminating incomplete or erroneous game record data, and ensuring data quality.

[0058] Text data processing, analyzing strategy texts, extracting key information. Perform pre-processing such as word segmentation, stop word removal, and semantic analysis on text data to extract key information.

[0059] Multimodal fusion module: used to fuse the features of chessboard image data, historical game record data and strategy text data, and comprehensively analyze the current game status. The feature vectors of chess record, image and text data are integrated into a unified feature representation through feature splicing and feature transformation.

[0060] Strategy generation module: used to generate strategy recommendations and improvement suggestions based on the fused feature data. Provide suggestions for the next move and explain the recommended strategy.

[0061] Output module: used to display real-time analysis results and strategy recommendations to users, and provide detailed improvement suggestions and learning resources.

[0062] As an example, when processing image data, the user uses the built-in camera function of the application to capture the image of the Go board of the current game, grayscales the image, converts the color image into a grayscale image, reduces the amount of data, and performs normalization to standardize the image size and pixel value range.

[0063] When extracting image features, a convolutional neural network is used to extract chessboard features from the image, automatically identify the position of chess pieces in the image and their type (black or white), and use the PCA dimensionality reduction algorithm to reduce the dimension of high-dimensional image features. The most important image features are retained to improve computational efficiency, and image feature vectors for multimodal fusion are generated and prepared for input into the fusion network.

[0064] The chess record data processing includes: users uploading or inputting chess record data of historical games, removing incomplete or erroneous chess record records, ensuring the quality and consistency of the data, unifying the coding method, and standardizing the chess record data.

[0065] The chess record feature extraction includes: converting each chessboard state into a feature vector, each feature vector contains the chess piece position and state of the current chess game; using principal component analysis (PCA) dimensionality reduction technology to reduce the dimension of the feature vector, select the most useful features for strategy generation, reduce computational complexity and noise, and generate a processed chess record feature vector, which will be input into a multimodal fusion network for further processing.

[0066] Text data processing includes user input related strategy text data, such as chess reviews, game analysis and study notes. The text is segmented to break the text into independent words. Stop words and irrelevant words are removed to reduce noise. Stem extraction is performed to restore words to their root form to improve the consistency of text data.

[0067] Text feature extraction includes: extracting text features using the BERT method of the natural language processing technology model, generating word vectors or sentence vectors to represent key information in the text. Through feature selection, the most important text features are retained, the dimension of the feature vector is reduced, the computational complexity and redundant information are reduced, and a text feature vector for multimodal fusion is generated, which is used as the input of the fusion network.

[0068] During multimodal fusion and strategy generation, the input layer is used to receive processed chess record features, image features, and text features, with each feature serving as a different part of the input layer.

[0069] The multimodal fusion layer uses a fully connected layer or an attention mechanism to fuse the features of different modalities, comprehensively process the features, generate a unified feature representation, and retain the important information of each modality.

[0070] Multiple hidden layers are used for deep learning and feature abstraction. Each hidden layer contains several neurons, which perform nonlinear transformation and feature extraction, and refine high-level features layer by layer.

[0071] The output layer is used to output the fused feature vector for generating Go strategies. Based on the comprehensive features, the next move is recommended and a detailed strategy explanation is provided.

[0072] The application interface displays the analysis results of the current chessboard in real time, provides suggestions for the next move, and explains the recommended strategy to help users understand and improve; based on the game analysis, it provides detailed improvement suggestions, points out the user's mistakes and areas for improvement. It also provides relevant learning resources, such as Go books, teaching videos, and case studies, to help users further improve their Go skills.

[0073] Through multimodal learning and integration of chess records, images and text data, the system can fully understand the user's game situation and provide intelligent strategy suggestions and improvement opinions. The system has broad application prospects in Go teaching applications and can significantly improve the user's learning effect and understanding of Go strategies.

[0074] Example 2

[0075] In this embodiment, we use the existing Go game database, open source image processing library and natural language processing technology to build a deep learning model with integrated features, and continuously optimize the model strategy through self-playing and expert evaluation. The specific steps are as follows:

[0076] S1. Data acquisition and preprocessing

[0077] S101, acquisition and preprocessing of chess record data

[0078] Get game record data: Get hundreds of thousands of game data from the public Go game record database and store it in SGF (SmartGame Format) format.

[0079] Data cleaning: Remove incomplete or erroneous game records to ensure data quality and consistency.

[0080] Data standardization: Based on the unified coding method, the chess record data is standardized. The chessboard state of each step is converted into a feature vector (a two-dimensional array represents the chessboard, black pieces are 1, white pieces are -1, and empty spaces are 0).

[0081] Feature extraction and dimensionality reduction: Use dimensionality reduction techniques such as PCA to extract key features and reduce feature dimensions.

[0082] S102, acquisition and preprocessing of image data

[0083] Acquire image data: collect image data of the Go board, which comes from real-time shooting or historical records.

[0084] Image preprocessing: Use the open source image processing library OpenCV to grayscale and normalize image data to improve image quality.

[0085] Image feature extraction: Use convolutional neural networks to extract chessboard features in images and automatically learn the positions and types of chess pieces on the board.

[0086] Feature selection and dimensionality reduction: PCA is used to reduce the dimensionality of high-dimensional image features and retain the most important image features.

[0087] S103, acquisition and preprocessing of text data

[0088] Go strategy text data provides additional strategy information and analysis, which helps the model understand the decision-making logic of the players. Go-related text materials are collected from Go books, online articles, Go review websites and other sources. Go strategy descriptions, game analysis and game review need to be parsed and processed using natural language processing (NLP) technology, clean and annotate game records and annotations, and perform word segmentation and grammar analysis. First, perform text cleaning operations to remove irrelevant characters, punctuation marks, and HTML tags. Then split the text into words or subwords. For the BERT model, use its own tokenizer for word segmentation.

[0089] Obtain text data: Collect text data related to Go, including chess reviews, strategy analysis, and game records.

[0090] Strategic text preprocessing: segment text, remove stop words, extract stems, and unify text formats.

[0091] Text feature extraction: Use natural language processing technology to extract text features and generate word vectors or sentence vectors.

[0092] Feature selection and dimensionality reduction: Through feature selection technology, the most important text features are retained and the dimension of the feature vector is reduced.

[0093] S2, modality data fusion and model training (see Figure 2 )

[0094] S201, multimodal feature fusion

[0095] Input layer: receives the preprocessed chess record feature vector, image feature vector and text feature vector, and each feature vector is a different part of the input layer.

[0096] Fusion layer: Use a fully connected layer or attention mechanism to fuse the features of different modalities to generate a unified feature representation and retain the important information of each modality.

[0097] Hidden layer: Multiple hidden layers are used for deep learning and feature abstraction. Each hidden layer contains several neurons, which perform nonlinear transformation and feature extraction, and refine high-level features layer by layer.

[0098] S202, model training

[0099] Training dataset: Use large-scale chess game, image and text datasets to train the model.

[0100] Optimization algorithm: The back propagation algorithm is used to optimize the model parameters, and the cross entropy loss function and mean square error (MSE) are used as the loss function.

[0101] Validation and testing: Use the validation set to evaluate model performance and adjust parameters to improve accuracy and generalization.

[0102] S3, model optimization and policy improvement

[0103] S301, self-playing training

[0104] The model plays games with itself to generate new chess record data, further enrich the training set, use reinforcement learning algorithms to improve model strategies, and optimize the model through continuous self-playing.

[0105] S302, Expert Evaluation and Feedback

[0106] Invite Go experts to evaluate the model's strategy, provide opinions and suggestions for improvement, adjust the model structure and parameters based on expert feedback, and optimize strategy generation.

[0107] S4, System Integration and Application

[0108] S401, real-time analysis and strategy recommendations

[0109] The camera function is integrated into the application to capture the Go board in real time, automatically identify the position and type of chess pieces, generate strategy suggestions for the next move based on the current state of the game, and provide explanations.

[0110] S402, Teaching and Learning

[0111] Provide users with detailed improvement suggestions, point out errors and areas for optimization, and provide relevant learning resources such as Go books, teaching videos and case studies to help users improve their Go skills.

[0112] Through multimodal data fusion and deep learning technology, the system can fully understand the Go game board and provide intelligent strategy suggestions and improvement opinions. The system has broad application prospects in the fields of Go teaching, strategy research and artificial intelligence game, and can significantly improve the user's learning effect and understanding of Go strategy.

[0113] A method for improving Go strategy decision-making ability by using multimodal learning fusion technology, comprising the following steps:

[0114] Data collection and preprocessing, obtaining Go board image data, Go strategy text data, and historical game record data;

[0115] Multimodal information fusion: (1) Visual information processing: Use convolutional neural networks (CNNs) to process chessboard image data and extract spatial features; (2) Text information processing: Use natural language processing (NLP) models (such as BERT) to process Go-related strategy texts and extract text semantic features; (3) Historical game data processing: Collect and analyze a large amount of historical game data and extract common strategies and decision-making patterns;

[0116] Feature extraction and representation: (1) Construct a multimodal neural network that includes visual, text, and historical data processing modules to extract features from information in different modalities; (2) Use the attention mechanism to fuse and weight features from different modalities to generate a comprehensive feature representation;

[0117] Decision-making mechanism optimization: (1) Strategy optimization: Based on the integrated feature representation after fusion, a comprehensive decision network (such as a reinforcement learning model) is constructed to select and optimize strategies; (2) Strategy evaluation: Through self-game and expert feedback, strategies are continuously evaluated and adjusted to improve overall performance;

[0118] As an example, the data collection and preprocessing steps include:

[0119] Acquiring visual data: The acquisition and processing of Go board image data helps enhance the model's understanding of the actual game. The state image of the Go board can be preprocessed through image processing technology to denoise, enhance, resize and normalize the board image data. The Go board image data should include various board states in different games. The image pixel values ​​should be normalized to the range of [0, 1] to facilitate neural network processing. All images should be resized to the same size to ensure the consistency of the input data.

[0120] Obtain strategy text data: Go-related strategy text data provides additional strategy information and analysis, which helps the model understand the decision-making logic of the players. Go-related text materials are collected from sources such as Go books, online articles, and Go review websites. Go strategy descriptions, game analysis, and game review are parsed and processed using natural language processing (NLP) technology, and game records and annotations are cleaned and annotated, and word segmentation and grammar analysis are performed. Specifically, the text is first cleaned to remove irrelevant characters, punctuation, and HTML tags, and then the text is segmented into words or subwords using the BERT model's built-in tokenizer.

[0121] Historical game record data: Game record data is the basis of Go strategy research. In order to obtain high-quality game record data, hundreds of thousands of historical games can be collected from multiple public game record databases, including game records of professional players and amateur masters. The data should include player information, game records (sequence of chessboard states) and game results. The collected game record data is formatted into a unified structure and stored in SGF form. Then, data conversion is performed to convert the game record data in SGF format into a two-dimensional chessboard state matrix acceptable to the machine learning model, where each element represents the chess piece (black, white or empty) at the corresponding position.

[0122] Optionally, the chessboard image data processing step in the multimodal information fusion includes:

[0123] The Go board image data is input into the convolutional neural network (CNN). After multiple layers of convolution and pooling operations, the spatial features of the board state are extracted. After the image data is preprocessed, the convolutional neural network architecture is designed. First, the input layer is set, and the dimension of the input image is set to (256, 256, 3), where the first and second numbers represent the height and width of the image, and the third number represents the number of channels; then the convolution layer is set, and multiple convolution kernels are applied to perform convolution operations on the input image to extract local features in the image. The appropriate convolution kernel size and number of convolution kernels are selected, and the ReLU activation function is used to increase the nonlinearity of the network. Each convolution operation generates a feature map.

[0124] Set the commonly used pooling kernel size to 2×2, perform pooling operations, use Max Pooling to reduce the dimension of the feature map and retain the main features. Use multiple convolutional layers and pooling layers to stack alternately to gradually extract more advanced features. The features extracted by each convolutional layer will become more and more complex and abstract. Flatten the feature map after multiple convolutions and pooling and convert it into a one-dimensional vector. Add one or more fully connected layers after multiple convolutional layers and pooling layers to further combine and process the extracted features. Use the cross entropy loss in the classification task as the loss function and the Adam optimizer to train the model. Set the accuracy as the evaluation indicator to evaluate the model performance. Use the training dataset to train the model, iteratively update the model parameters, and optimize the feature extraction and decision-making capabilities. Set the training parameters as the number of training rounds (epochs) and batch size (batchsize).

[0125] Through the above steps, the Go board image data can be effectively extracted from the Go board state through multi-layer convolution and pooling operations of the convolutional neural network. These features will provide a solid foundation for subsequent multimodal information fusion and strategy optimization.

[0126] The steps of text information processing in multimodal information fusion include:

[0127] The Go strategy text data is input into the natural language processing (NLP) model BERT to parse the text data and extract the text semantic features of the strategy description. BERT is a pre-trained language representation model that can understand the contextual information of the text through a bidirectional Transformer architecture. The pre-trained BERT model can efficiently extract the text semantic features of the text. Specifically: First, use the Hugging Face Transformers library to load the BERT tokenizer. The text data is tokenized by the BERT tokenizer and converted into an input format that the model can process. The formatted input data is input into the BERT model to extract the text semantic features of the text. The output of the BERT model includes the hidden states of all layers and the pooled output of the last layer. Then, according to the task requirements, choose to use the hidden states of the last layer or the hidden states of all layers. The pooled output can be used as the semantic representation of the entire input sequence for subsequent feature fusion and decision models.

[0128] Through the above steps, the Go strategy text data is processed by the BERT model to extract the text semantic features of the text description. After the text semantic features are integrated with the chessboard image data and historical game data, they provide Go AI with richer and more comprehensive information and improve AI's strategic decision-making ability.

[0129] Optionally, the historical data processing steps in multimodal information fusion include:

[0130] After preprocessing the historical chess record data, a dedicated network model structure is constructed to extract chess game features.

[0131] The dedicated network model structure is based on a convolutional neural network, in which the input layer: accepts a 19×19 two-dimensional chessboard state matrix; the convolution layer: uses multi-layer convolution operations to extract the spatial features of the chessboard; the activation function: uses the ReLU activation function to introduce nonlinear features; the pooling layer: reduces the size of the feature map and reduces the amount of calculation through maximum pooling or average pooling; the fully connected layer: flattens the feature map output by the convolution layer and connects it to the fully connected layer for the final feature representation.

[0132] Then, a dedicated network model is trained. First, the preprocessed chess record data is used to generate training samples. Each sample includes the chessboard state and the corresponding next move position. The label data is the next move position, which can be represented as a 19×19 two-dimensional array with only one position being 1 and the rest being 0.

[0133] The cross entropy loss function is used to evaluate the gap between the model output and the true label, and the Adam optimizer is used to optimize the model parameters. The training process iterates the model through multiple epochs and adjusts the parameters to minimize the loss function. When evaluating the model, the test data uses an independent test data set to evaluate the model performance. The evaluation indicators use accuracy and average error rate to evaluate the strategy effect of the model.

[0134] Furthermore, when optimizing the model, hyperparameter adjustment is performed based on the evaluation results to adjust the model's hyperparameters, such as the number of convolutional layers, convolution kernel size, and learning rate. Data enhancement is used to increase the amount and diversity of training data and improve the generalization ability of the model. Under the optimized conditions, further model training is performed to improve model performance.

[0135] Through the above steps, the dedicated network model structure can effectively process Go game data, extract game state features, and provide a solid foundation for the optimization of multimodal learning fusion Go strategy. This method has high accuracy and generalization ability and can adapt to complex and changeable Go game environments.

[0136] Furthermore, multimodal data fusion integrates the feature vectors of chess records, images and text data into a unified feature representation through feature concatenation and feature transformation. The detailed steps are as follows:

[0137] In the feature splicing stage, the feature vectors extracted from the chess records, images, and strategy text data should maintain the same dimension and feature representation, and the feature vectors of each modality should be directly spliced ​​into a high-dimensional feature vector in order. In this step, feature splicing does not require additional processing of the features, but only requires connecting them in series to ensure that the dimensions of the feature vectors of each modality are unified. To determine the splicing order, you can sort according to the importance of the data and the contribution of the modal features. As an example, when the length of the chess record feature vector is N1, the length of the image feature vector is N2, and the length of the text feature vector is N3, the length of the spliced ​​feature vector is N=N1+N2+N3.

[0138] In the feature transformation stage, multiple fully connected layers are stacked to perform linear transformation and nonlinear mapping on the concatenated feature vectors to extract higher-level and more abstract features. The activation function ReLU is used to introduce nonlinearity to increase the model's expressiveness and feature diversity. A self-attention mechanism is added to the concatenated feature vectors so that the model can adaptively focus on the correlation of different parts of the features, thereby improving the expressiveness and generalization capabilities of the features.

[0139] In the stage of integrating feature representation, the feature vector processed by the fully connected layer and the self-attention mechanism is regarded as the integrated multimodal feature representation, which contains the characteristics and interaction information of each modal data. The integrated features are analyzed to understand the information contained therein and ensure the quality and rationality of the features. According to the requirements of subsequent tasks, the dimension and range of the feature vector are adjusted to match the input requirements of downstream tasks.

[0140] Through feature concatenation and feature transformation, different information sources of multimodal data can be effectively integrated and extracted.

[0141] The construction of multimodal neural network in the feature extraction and representation step includes:

[0142] Before building a multimodal neural network, the visual, textual and historical chess record data have been preprocessed, and image features, textual semantic features and historical chess record features have been extracted. The visual, textual and historical chess record features are then fused to form a comprehensive feature representation. Based on the fused features, a comprehensive decision network is built to select and optimize strategies. The model is trained using the training data and its performance is evaluated. Convolutional neural networks are used to process image data and extract spatial features. Natural language processing is used to process text data and extract text semantic features. A dedicated network model structure is used to process chess record data and extract game state features. A fully connected layer and an output layer are added to obtain a multimodal neural network model.

[0143] Through the above steps, a multimodal neural network model including visual, text and historical game data processing modules was constructed. This multimodal neural network model can extract features from information of different modalities and form a comprehensive feature representation by fusing these features for Go strategy optimization and decision-making.

[0144] In the integrated decision network, the input layer receives feature vectors extracted from chess records, images, and text data as the input of the network. Ensure that the dimension and representation of the feature vectors of each modality are unified so that they can be smoothly input into the network. The hidden layer processes the input feature vectors through multiple fully connected layers to extract higher-level feature representations. Each fully connected layer contains the activation function ReLU and regularized Dropout. The image feature vector is convolved to retain spatial information. A pooling layer is added after the convolution layer to reduce the size of the feature map. The feature vectors processed by the fully connected layer or convolution layer are combined to form a higher-dimensional and richer feature representation. The output layer generates the probability distribution of the next move position and the chess game evaluation. For the probability distribution of the next move position, the softmax function is used to output the probability of each position. The chess game evaluation is a single value that represents the score of the current situation.

[0145] As a further optimization, the attention mechanism is introduced into the multimodal neural network model as follows:

[0146] The attention mechanism is used to weight and integrate image features, text features, and historical chess features to form a comprehensive feature representation. The attention mechanism is used in neural networks to selectively focus on important information. By calculating the importance weights of different features, the attention mechanism can dynamically adjust the weights of input features to form a more effective comprehensive feature representation. Specifically: First, the visual features, text features, and historical chess features are used as inputs to the attention mechanism, all features are spliced ​​together to form a large feature vector, and the importance weights of each feature are calculated using dense layers and activation functions. Through the weighted summation method, different features are integrated according to the calculated attention weights to form a comprehensive feature representation. The comprehensive feature representation is processed, further fully connected layers and decision layers are added, and a multimodal neural network model containing the attention mechanism is constructed and compiled.

[0147] Through the above steps, a multimodal neural network model with attention mechanism is realized. The multimodal neural network model with attention mechanism can extract features from visual, textual and historical chess data, and weightedly integrate these features through the attention mechanism to form a comprehensive feature representation.

[0148] Based on the integrated feature representation after fusion, the strategy selection and optimization using the integrated decision network model include:

[0149] Search algorithm fusion submodule: Combine Monte Carlo Tree Search (MCTS) with the policy network to improve the accuracy and efficiency of decision-making; Adaptive policy adjustment submodule: Dynamically adjust the parameters of the policy network to enhance adaptability. Adversarial training submodule: Introduce adversarial networks, simulate opponents of different levels, and conduct adversarial training to improve the robustness of the decision-making mechanism.

[0150] Monte Carlo Tree Search (MCTS) is combined with the policy network to initialize the tree structure starting from the current board state, that is, from the root node. Each node stores information such as the board state, number of visits, and total value. Starting from the root node, child nodes are selected based on the upper confidence bound (UCB1) until a leaf node is reached. The UCB1 formula is as follows:

[0151] UCB1=Q(v)+c*√((ln(N)) / (n(v)))

[0152] Where Q(v) is the average value of node v, N is the number of visits to the parent node, n(v) is the number of visits to node v, and c is the exploration constant.

[0153] Expand: If a leaf node is not terminated and can be expanded, child nodes are generated from that node and added to the tree.

[0154] Simulation: Start from the leaf node, simulate random games or use the strategy network to play games until the game ends and get the results.

[0155] Feedback: The simulation results are fed back to all nodes passed through, and the number of node visits and total value are updated.

[0156] Adaptive strategy adjustment, in the selection and simulation stages, combines the predicted value (probability distribution and value estimation) of the policy network to guide the decision of MCTS. The specific approach is to use the output of the policy network as the prior probability and adjust the Q(v) value in the UCB1 formula. During the game, adjust the exploration constant c and the learning rate; when training the policy network, introduce opponent data of different styles so that the policy network can learn and adapt to multiple strategies. In the actual game, evaluate the opponent's strategy changes in real time, and dynamically adjust the parameters of the policy network according to the evaluation results to enhance adaptability. Use the generative adversarial network (GAN) generation method to generate virtual opponents of different levels and styles. The policy network plays against the virtual opponent, and through continuous confrontation, the policy network's ability to cope with different strategies is improved. Through multiple rounds of adversarial training, the robustness and generalization ability of the policy network are gradually improved.

[0157] On the basis of the above optimization, the present invention also evaluates and adjusts the model through self-play and expert feedback, including:

[0158] Use multimodal data to evaluate strategies and ensure the consistency and effectiveness of the model's strategies under different data modalities: Evaluation indicators: Use indicators such as win rate and average error rate to evaluate the model's strategy effects; Expert evaluation: Invite Go experts to evaluate the model's strategies and provide feedback and suggestions; Model adjustment: Adjust and optimize model parameters based on evaluation results and expert feedback.

[0159] The above contents are only for explaining the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for improving intelligent decision-making ability based on multimodal learning fusion, characterized in that: The following steps are involved: Obtain chessboard image data, strategy text data, and historical game record data; Use convolutional neural network to process chessboard image data and extract spatial features; Use the natural language processing model to process Go-related strategy texts and extract text semantic features; extract historical game features, namely common strategies and decision-making patterns, from historical game data; Use a multimodal neural network model to extract features of different modalities, introduce an attention mechanism to fuse and weight features of different modalities, and generate a comprehensive feature representation; Based on the fused comprehensive feature representation, a comprehensive decision network model is used for strategy selection and optimization.

2. The method for improving intelligent decision-making ability based on multimodal learning fusion according to claim 1 is characterized in that: Obtaining chessboard image data, strategy text data, and historical game record data includes: Denoising, enhancing, resizing and normalizing chessboard image data, which includes various chessboard states in different games, normalizing image pixel values ​​and adjusting all images to the same size; Use the natural language processing model to parse and process the strategy text data, clean and annotate the game records and comments, perform word segmentation and grammar analysis, remove irrelevant characters, punctuation marks, and HTML tags, and use the word segmenter provided by the BERT model to split the strategy text data into words or subwords; Format the historical game record data into a unified structure and convert the game record data into a two-dimensional chessboard state matrix, where each element represents the chess piece at the corresponding position.

3. The method for improving intelligent decision-making ability based on multimodal learning fusion according to claim 1 is characterized in that: The chessboard image data processing in multimodal information fusion includes: The chessboard image data is input into the convolutional neural network. After multiple layers of convolution and pooling operations, the spatial features of the chessboard state are extracted. The convolutional neural network architecture: set the input layer, set the dimension of the input image, set the convolution layer, apply multiple convolution kernels to perform convolution operations on the input image, extract local features in the image, set the convolution kernel size, the number of convolution kernels, use the ReLU activation function, and each convolution operation generates a feature map; The feature map is pooled in the pooling layer, and the maximum pooling is used to reduce the dimension of the feature map. Multiple convolutional layers and pooling layers are stacked alternately to flatten the feature map after multiple convolutions and pooling, and convert it into a one-dimensional vector. One or more fully connected layers are used to further combine and process the extracted features. Use the cross entropy loss and Adam optimizer in the classification task to train the convolutional neural network model, set the evaluation index accuracy to evaluate the model performance, use the training data set to train the model, and iteratively update the model parameters; The Go board image data is processed by a convolutional neural network to obtain the spatial characteristics of the board state.

4. The method for improving intelligent decision-making ability based on multimodal learning fusion according to claim 1 is characterized in that: Strategic text data processing in multimodal information fusion includes: Input the policy text data into the BERT model, load the BERT tokenizer to tokenize and format the policy text data, input the formatted input data into the BERT model for processing, and the output includes the hidden states of all layers and the pooled output of the last layer. Using the hidden state of the last layer of the BERT model or the hidden states of all layers, the pooled output can be used as the semantic representation of the entire policy text data.

5. The method for improving intelligent decision-making ability based on multimodal learning fusion according to claim 1 is characterized in that: The historical data processing steps include: The input layer obtains historical chess data and preprocesses it to obtain a two-dimensional chessboard state matrix. Multi-layer convolution operations are performed on the two-dimensional chessboard state matrix to extract the spatial features of the chessboard; the feature map output by the convolution layer is flattened and connected to the fully connected layer for the final feature representation.

6. The method for improving intelligent decision-making ability based on multimodal learning fusion according to claim 1 is characterized in that: Multimodal neural network construction includes: Image features, text semantic features and historical chess record features are fused to form a comprehensive feature representation. Based on the fused features, a comprehensive decision network is constructed. In the comprehensive decision network: convolutional neural network is used to process image data and extract spatial features; natural language processing is used to process text data and extract text semantic features; a dedicated network model structure is used to process chess record data and extract chess game state features; a fully connected layer and an output layer are added to obtain a multimodal neural network model.

7. The method for improving intelligent decision-making ability based on multimodal learning fusion according to claim 1 is characterized in that: Introducing the attention mechanism into the multimodal neural network model includes: Visual features, text features and historical chess features are prepared as inputs of the attention mechanism. All features are concatenated together to form a feature vector. The importance weight of each feature is calculated using dense layers and activation functions. Different features are integrated according to the calculated attention weights through the weighted summation method to form a comprehensive feature representation. The comprehensive feature representation is subsequently processed, and further fully connected layers and decision layers are added to obtain a multimodal neural network model that includes an attention mechanism.

8. The method for improving intelligent decision-making ability based on multimodal learning fusion according to claim 1 is characterized in that: Based on the integrated feature representation after fusion, the strategy selection and optimization using the integrated decision network model include: Combine Monte Carlo tree search with policy network, dynamically adjust the parameters of policy network, introduce adversarial network, and simulate opponents of different levels; Monte Carlo tree search is combined with the policy network. Starting from the root node, the tree structure is initialized. Each node stores the board state, number of visits, and total value information. Starting from the root node, child nodes are selected based on the upper confidence bound until a leaf node is reached. During the game, adjust the exploration constant and learning rate; when training the policy network, introduce opponent data of different styles so that the policy network can learn and adapt to multiple strategies; evaluate the opponent's strategy changes in real time, and dynamically adjust the parameters of the policy network based on the evaluation results.

9. The method for improving intelligent decision-making ability based on multimodal learning fusion according to claim 1 is characterized in that: It also includes evaluating and adjusting the model through self-play and expert feedback: Use multimodal data to evaluate strategies and ensure the consistency and effectiveness of the strategies in different data modalities. Evaluation indicators: Win rate and average error rate are used to evaluate the strategic effect of the model; Expert feedback: Invite Go experts to evaluate the model's strategy and provide feedback and suggestions; Model adjustment: Adjust and optimize model parameters based on evaluation results and expert feedback.

10. A system for improving intelligent decision-making capabilities based on multimodal learning fusion, characterized in that: It includes input module, data processing module, multimodal fusion module and strategy generation module; The input module is used to obtain the Go board image of the game, the chess record data of the historical game and the strategy text data; The data processing module is used to pre-process the chessboard image data of the game, the chess record data of the historical game, and the strategy text data to obtain the chessboard image data, the chess record data of the historical game, and the strategy text data that meet the feature fusion requirements; The multimodal fusion module is used to fuse the features of chessboard image data, historical game record data and strategy text data, comprehensively analyze the current game status, and integrate the feature vectors of chess record, image and text data into a unified feature representation through feature splicing and feature transformation; The strategy generation module is used to generate strategy recommendations and improvement suggestions based on the unified feature representation, provide suggestions for the next move, and explain the recommended strategy.

Images