Multi-modal analysis method based on intelligent robot
By training a single-modal decision model and combining regularized loss function and double-factor verification, the optimal decision feature group is generated for multimodal analysis of intelligent robots, which solves the problem of rational decision-making in complex environments and improves the perception model General availability and depth of decision making.
Patent Information
- Application Number
- CN202411182491.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-31
AI Technical Summary
The existing intelligent robot multimodal analysis schemes have insufficient training depth in single-modal perception model and the integration depth of multimodal decision-making schemes is not high, making it difficult to ensure the rationality of application decisions in complex environments.
By obtaining data of each perceived mode of the intelligent robot, training a single-modal decision model, building a set of decision-making feature types, and ensuring the generalization of the model through regularized loss functions and double-factor verification, cross-fusion is carried out in combination with multimodal decision-making features to generate The optimal decision feature group makes comprehensive decisions.
It improves the rationality and perceptual universality of intelligent robots in complex environments, and ensures the deep integration of perceptual accuracy and multimodal decision-making in unknown scenarios.
Smart Images
Figure CN119226996A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of robot learning, and in particular to a multimodal analysis method based on an intelligent robot. Background Art
[0002] With the rapid development of artificial intelligence technology, intelligent robots are increasingly used in various fields. However, the execution of tasks in complex environments requires robots to have higher perception, understanding and decision-making capabilities. As an effective means, multimodal analysis methods can integrate information from different perception modules, such as vision, speech, touch, etc., to improve the robot's understanding and adaptability to the environment.
[0003] The existing multimodal analysis solutions for intelligent robots lack sufficient training depth for the perception models under each single mode, and the fusion depth of decision-making solutions under multiple single modes is not high, making it difficult to ensure the rationality of the application decisions of intelligent robots in complex environments. Summary of the invention
[0004] In order to solve the above technical problems, a multimodal analysis method based on an intelligent robot is provided. This technical solution solves the problem that the existing multimodal analysis scheme of the intelligent robot mentioned above has insufficient training depth for the perception model under each single modality and low fusion depth for the decision-making schemes under multiple single modalities, making it difficult to ensure the rationality of the application decision of the intelligent robot in a complex environment.
[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:
[0006] A multimodal analysis method based on an intelligent robot, comprising:
[0007] Acquire the perception modality dimension of the intelligent robot, wherein the perception modality includes visual modality, auditory modality and tactile modality, and the perception modality dimension is the total number of perception modality types of the robot;
[0008] Based on each perception modality of the intelligent robot, a single-modal decision model corresponding to the robot's perception modality is trained;
[0009] Obtain the perception data of the intelligent robot in each perception mode;
[0010] Based on the perception data of the intelligent robot in each perception mode, a single-modal decision-making model corresponding to the robot's perception mode is used to determine the single-modal decision-making plan in each perception mode;
[0011] Based on each unimodal decision scheme, construct a decision feature type set of each unimodal decision scheme and a decision feature value of each unimodal decision scheme;
[0012] The decision feature type sets of all single-mode decision solutions are combined to obtain the total set of decision feature types;
[0013] Determine at least one candidate decision feature value for each decision feature in the total set of decision feature types based on the decision feature value of each decision feature in the decision feature type set corresponding to each unimodal decision solution;
[0014] Based on at least one candidate decision feature value of each decision feature in the total set of decision feature types, cross-fusion is performed to obtain at least one preliminary screening decision feature group;
[0015] Based on the fusion analysis formula, the comprehensive decision index of each preliminary screening decision feature group is determined, and the preliminary screening decision feature group with the largest comprehensive decision index is selected as the optimal decision feature group of the intelligent robot;
[0016] Output the multimodal comprehensive decision of the intelligent robot according to the optimal decision feature group.
[0017] Preferably, the training of a single-modal decision model corresponding to each perception modality of the robot based on each perception modality of the intelligent robot specifically includes:
[0018] Using automatic or manual methods, obtain a number of sample sets under single-modal decision-making;
[0019] The sample set is divided into a training set and a validation set in a ratio of 8:2;
[0020] The training set is divided into a model training set and a model test set in a ratio of 7:3;
[0021] Repeat the step of dividing the training set into a model training set and a model test set n times to obtain n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other;
[0022] Based on n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other, construct at least one model regularization loss function;
[0023] Using a neural network, determine the preliminary unimodal decision model trained using the training set under the regularized loss function of each model;
[0024] Determine the performance of each preliminary unimodal decision model on the validation set and select the final unimodal decision model;
[0025] Among them, the expression of the model regularization loss function is:
[0026] L'(θ)=L(θ)+L∑|θ j
[0027] Among them, L is the coefficient of the regularization term, θ j is the jth parameter of the neural network, L'(θ) is the model regularization loss function, and L(θ) is the model loss function.
[0028] Preferably, constructing at least one model regularization loss function based on n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other specifically includes:
[0029] Obtaining at least one coefficient of a regularization term to be selected within a value range of the coefficient of the regularization term according to a set value step size;
[0030] Substitute the coefficient of the candidate regularization term into the model regularization loss function to obtain the candidate model regularization loss function;
[0031] Determine n first neural network models trained by using n sets of model training sets that are not completely identical to each other under the regularized loss function of each candidate model;
[0032] Calculate the loss values of n first neural network models under n groups of model test sets that are not completely identical to each other;
[0033] The loss values of n first neural network models under n groups of model test sets that are not completely identical to each other are averaged to obtain the fitting loss value of the coefficient of the regularization term to be selected;
[0034] Screening out at least one coefficient of a candidate regularization term whose fitting loss value is within a preset interval as the coefficient of the determined regularization term;
[0035] The regularization loss function of the candidate model corresponding to the coefficients of several determined regularization terms is used as the model regularization loss function.
[0036] Preferably, the step of respectively determining the performance of each preliminary unimodal decision model on the validation set and selecting the final unimodal decision model specifically includes:
[0037] Calculate the loss value of each preliminary unimodal decision model on the validation set and record it as the actual loss value;
[0038] Determine the fitting loss value of the coefficient of the regularization term of the model regularization loss function corresponding to the preliminary unimodal decision model;
[0039] The preliminary unimodal decision model with the smallest difference between the actual loss value and the fitted loss value is selected as the final unimodal decision model.
[0040] Preferably, the cross-fusion of at least one candidate decision feature value of each decision feature in the total set of decision feature types to obtain at least one preliminary screening decision feature group specifically includes:
[0041] Select and only select one candidate decision feature value from at least one candidate decision feature value of each decision feature as the preliminary screening decision feature value, and combine all the preliminary screening decision feature values into a preliminary screening decision feature group;
[0042] Repeat the above steps several times until no new initial screening decision feature group can be generated;
[0043] Output all the initial screening decision feature groups obtained.
[0044] Preferably, the comprehensive decision index of each initial screening decision feature group is determined based on the fusion analysis formula and specifically includes:
[0045] Extracting the preliminary screening decision feature value of the decision feature type in the decision feature type set of each single-modal decision solution from the preliminary screening decision feature group;
[0046] The loss value is calculated based on the decision feature value of the unimodal decision scheme and the initial screening decision feature value, and is recorded as the unimodal loss value of the initial screening decision feature group for the unimodal type;
[0047] Determine the unimodal loss value of each unimodal type of the initial screening decision feature group, and construct the unimodal loss value array A of the initial screening decision feature group j , A j =[a 1j …a ij …a mj ]; among them, A j is the unimodal loss value array of the jth initial screening decision feature group, a ij is the unimodal loss value of the jth initial screening decision feature group for the ith unimodal category, and m is the total number of unimodal categories;
[0048] The unimodal loss value arrays of all initial screening decision feature groups are constructed into a unimodal loss value matrix B. k is the total number of initial screening decision feature groups;
[0049] Based on the single-mode loss value matrix B, the comprehensive decision index of each initial screening decision feature group is calculated through the fusion analysis formula;
[0050] The fusion analysis formula is specifically:
[0051]
[0052] R j is the comprehensive decision index of the jth initial screening decision feature group, a iminfor a i1 …a ij …a ik The minimum value in a imax for a i1 …a ij …a ik The maximum value in .
[0053] Compared with the prior art, the present invention has the following beneficial effects:
[0054] The present invention proposes a multimodal analysis method based on an intelligent robot. By introducing a regularization term to construct a loss function in the model training process, it ensures that the trained perception model can better cope with different unknown scenarios. At the same time, a double verification method is designed to determine the optimal regularization coefficient, which effectively ensures the perception accuracy of the trained perception model in known scenarios and the perception versatility in unknown scenarios.
[0055] This solution also conducts a comprehensive analysis based on the decision outputs of various perception models, splits and reorganizes the decision features of several single-modal decision solutions, and substitutes the reorganized decision feature group back to the decision solution under the single modality, and calculates and fits the comprehensive decision loss of the reorganized decision feature group, which effectively ensures the depth of integration of multiple single-modal decision solutions when the robot makes decisions, and ensures the rationality of the application decision of the intelligent robot in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 Flowchart of the multimodal analysis method based on intelligent robots proposed for this scheme;
[0057] Figure 2 A flowchart of a method for training a single-modal decision model that corresponds one-to-one with the robot's perception modality in this solution;
[0058] Figure 3 A flowchart of a method for constructing at least one model regularization loss function in this solution;
[0059] Figure 4 A flow chart of the method for selecting the final single-modal decision model in this scheme;
[0060] Figure 5 A flow chart of a method for obtaining at least one initial screening decision feature group for cross-fusion in this scheme;
[0061] Figure 6 A flow chart of the method for determining the comprehensive decision-making index for each initial screening decision feature group in this scheme;
[0062] Figure 7 This is a schematic diagram of the electronic device in this solution;
[0063] Figure 8 This is a schematic diagram of the computer-readable storage medium structure in this solution. DETAILED DESCRIPTION
[0064] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.
[0065] Reference Figure 1 As shown, a multimodal analysis method based on an intelligent robot includes:
[0066] Obtain the perception modality dimension of the intelligent robot. The perception modality includes visual modality, auditory modality and tactile modality. The perception modality dimension is the total number of perception modalities of the robot.
[0067] Based on each perception modality of the intelligent robot, a single-modal decision model corresponding to the robot's perception modality is trained;
[0068] Obtain the perception data of the intelligent robot in each perception mode;
[0069] Based on the perception data of the intelligent robot in each perception mode, a single-modal decision-making model corresponding to the robot's perception mode is used to determine the single-modal decision-making plan in each perception mode;
[0070] Based on each unimodal decision scheme, construct a decision feature type set of each unimodal decision scheme and a decision feature value of each unimodal decision scheme;
[0071] The decision feature type sets of all single-mode decision solutions are combined to obtain the total set of decision feature types;
[0072] Determine at least one candidate decision feature value for each decision feature in the total set of decision feature types based on the decision feature value of each decision feature in the decision feature type set corresponding to each unimodal decision solution;
[0073] Based on at least one candidate decision feature value of each decision feature in the total set of decision feature types, cross-fusion is performed to obtain at least one preliminary screening decision feature group;
[0074] Based on the fusion analysis formula, the comprehensive decision index of each preliminary screening decision feature group is determined, and the preliminary screening decision feature group with the largest comprehensive decision index is selected as the optimal decision feature group of the intelligent robot;
[0075] Output the multimodal comprehensive decision of the intelligent robot according to the optimal decision feature group.
[0076] This solution introduces a regularization term to construct a loss function in the model training process to ensure that the trained perception model can better cope with different unknown scenarios. At the same time, a double verification method is designed to determine the optimal regularization coefficient, which effectively guarantees the perception accuracy of the trained perception model in known scenarios and the perception versatility in unknown scenarios. At the same time, a comprehensive analysis is performed based on the decision outputs of various perception models, and the decision features of several single-modal decision schemes are split and reorganized. The reorganized decision feature group is substituted back into the decision scheme under the single modality, and the comprehensive decision loss of the reorganized decision feature group is calculated and fitted, which effectively guarantees the fusion depth of multiple single-modal decision schemes when the robot makes decisions, and ensures the rationality of the application decision of the intelligent robot in a complex environment.
[0077] Reference Figure 2 As shown, based on each perception modality of the intelligent robot, training a single-modal decision model corresponding to the robot's perception modality specifically includes:
[0078] Using automatic or manual methods, obtain a number of sample sets under single-modal decision-making;
[0079] The sample set is divided into a training set and a validation set in a ratio of 8:2;
[0080] The training set is divided into a model training set and a model test set in a ratio of 7:3;
[0081] Repeat the step of dividing the training set into a model training set and a model test set n times to obtain n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other;
[0082] Based on n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other, construct at least one model regularization loss function;
[0083] Using a neural network, determine the preliminary unimodal decision model trained using the training set under the regularized loss function of each model;
[0084] Determine the performance of each preliminary unimodal decision model on the validation set and select the final unimodal decision model;
[0085] Among them, the expression of the model regularization loss function is:
[0086] L'(θ)=L(θ)+L∑|θ j
[0087] Among them, L is the coefficient of the regularization term, θ j is the jth parameter of the neural network, L'(θ) is the model regularization loss function, and L(θ) is the model loss function.
[0088] Reference Figure 3 As shown, based on n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other, constructing at least one model regularization loss function specifically includes:
[0089] Obtaining at least one coefficient of a regularization term to be selected within a value range of the coefficient of the regularization term according to a set value step size;
[0090] Substitute the coefficient of the candidate regularization term into the model regularization loss function to obtain the candidate model regularization loss function;
[0091] Determine n first neural network models trained by using n sets of model training sets that are not completely identical to each other under the regularized loss function of each candidate model;
[0092] Calculate the loss values of n first neural network models under n groups of model test sets that are not completely identical to each other;
[0093] The loss values of n first neural network models under n groups of model test sets that are not completely identical to each other are averaged to obtain the fitting loss value of the coefficient of the regularization term to be selected;
[0094] Screening out at least one coefficient of a candidate regularization term whose fitting loss value is within a preset interval as the coefficient of the determined regularization term;
[0095] The regularization loss function of the candidate model corresponding to the coefficients of several determined regularization terms is used as the model regularization loss function.
[0096] Reference Figure 4 As shown in the figure, the performance of each preliminary unimodal decision model on the validation set is determined respectively, and the final unimodal decision model is selected, including:
[0097] Calculate the loss value of each preliminary unimodal decision model on the validation set and record it as the actual loss value;
[0098] Determine the fitting loss value of the coefficient of the regularization term of the model regularization loss function corresponding to the preliminary unimodal decision model;
[0099] The preliminary unimodal decision model with the smallest difference between the actual loss value and the fitted loss value is selected as the final unimodal decision model.
[0100] The regularization coefficient is added to the loss function as a regularization term. This regularization term is a constraint or penalty on the model parameters, which aims to reduce the complexity of the model. Therefore, the size of the regularization coefficient will directly affect the value of the loss function. When the regularization coefficient is small, the model complexity is relatively high, and overfitting may be more likely to occur. At this time, the loss value may be very low on the training set, but may be high on the validation set or test set. As the regularization coefficient increases, the model complexity gradually decreases, and the overfitting phenomenon is alleviated, but it may also cause the model to be underfitted, and the loss value may be high on both the training set and the validation set / test set;
[0101] Therefore, in this scheme, a grid search method is used to determine the regularization coefficient, and the average value of the loss performance of the regularization coefficient on n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other is used as the fitting loss value of the regularization coefficient;
[0102] The preset interval of loss value is set according to the complexity of the actual application scenario of the intelligent robot. The lower the complexity of the actual application scenario of the intelligent robot, the more specific needs of the actual application scenario of the intelligent robot are encountered. At this time, the intelligent robot needs to respond to unknown scenarios less, and the overall loss value of the intelligent robot needs to be small. Therefore, the preset interval of loss value can be set in a smaller interval. On the contrary, the higher the complexity of the actual application scenario of the intelligent robot, the higher the preset interval of loss value needs to be set. By screening out the coefficient of at least one regularization term whose fitting loss value of the regularization coefficient is in the set preset interval of loss value, the preliminary selection of the regularization term is achieved.
[0103] By screening out the preliminary unimodal decision model with the smallest difference between the actual loss value and the fitted loss value, the smaller the difference between the actual loss value and the fitted loss value, the higher the stability of the preliminary unimodal decision model trained using the regularization term coefficient.
[0104] Reference Figure 5 As shown, based on at least one candidate decision feature value of each decision feature in the total set of decision feature types, cross-fusion to obtain at least one preliminary screening decision feature group specifically includes:
[0105] Select and only select one candidate decision feature value from at least one candidate decision feature value of each decision feature as the preliminary screening decision feature value, and combine all the preliminary screening decision feature values into a preliminary screening decision feature group;
[0106] Repeat the above steps several times until no new initial screening decision feature group can be generated;
[0107] Output all the initial screening decision feature groups obtained.
[0108] Reference Figure 6 As shown in the figure, based on the fusion analysis formula, the comprehensive decision indicators for each initial screening decision feature group are determined to include:
[0109] Extracting the preliminary screening decision feature value of the decision feature type in the decision feature type set of each single-modal decision solution from the preliminary screening decision feature group;
[0110] The loss value is calculated based on the decision feature value of the unimodal decision scheme and the initial screening decision feature value, and is recorded as the unimodal loss value of the initial screening decision feature group for the unimodal type;
[0111] Determine the unimodal loss value of each unimodal type of the initial screening decision feature group, and construct the unimodal loss value array A of the initial screening decision feature group j , A j =[a 1j …a ij …a mj ]; among them, A j is the unimodal loss value array of the jth initial screening decision feature group, a ij is the unimodal loss value of the jth initial screening decision feature group for the ith unimodal category, and m is the total number of unimodal categories;
[0112] The unimodal loss value arrays of all initial screening decision feature groups are constructed into a unimodal loss value matrix B. k is the total number of initial screening decision feature groups;
[0113] Based on the single-mode loss value matrix B, the comprehensive decision index of each initial screening decision feature group is calculated through the fusion analysis formula;
[0114] The specific fusion analysis formula is:
[0115]
[0116] R j is the comprehensive decision index of the jth initial screening decision feature group, a imin for a i1 …a ij …a ik The minimum value in a imax for a i1 …a ij …a ik The maximum value in .
[0117] It is understandable that in the intelligent robot decision-making system under multimodality, decision schemes under different modes may usually have the same type of decision features, while decision schemes under different modes may have different decision feature values for the same type of decision features. Based on this situation, this scheme adopts the scheme extraction and fusion method to extract the decision features of decision schemes under all modes and reorganize them into decision feature groups, and correspond the reorganized decision feature groups to the decision loss values under each mode for comprehensive analysis. By calculating the reorganized decision feature group and the ideal optimal scheme loss a imin and the ideal worst solution loss a imax The comprehensive distance between them represents the reasonable index of the scheme under multimodal fusion. The closer the comprehensive decision index of the initial screening decision feature group is to 1, the closer the initial screening decision feature group is to the ideal optimal scheme, and the better the decision.
[0118] Furthermore, the method according to the embodiment of the present application can also be performed by Figure 7 The electronic device architecture shown in FIG. Figure 7 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store a multimodal analysis method based on an intelligent robot provided in the present application. The electronic device 500 may also include a user interface 508. Of course, Figure 7 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 7 One or more components of an electronic device are shown.
[0119] Figure 8 Schematic diagram of a computer-readable storage medium structure provided by an embodiment of the present application. Figure 8 As shown, a computer-readable storage medium 600 according to an embodiment of the present application is shown. Computer-readable instructions are stored on the computer-readable storage medium 600. When the computer-readable instructions are executed by the processor, a multimodal analysis method based on an intelligent robot according to an embodiment of the present application described with reference to the above figures can be executed. The storage medium 600 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.
[0120] In summary, the advantages of the present invention are: effectively integrating the optimal solution under multimodal decision-making to achieve multimodal automated optimal decision-making of intelligent robots.
[0121] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.
Claims
1. A multimodal analysis method based on an intelligent robot, characterized in that: include: Acquire the perception modality dimension of the intelligent robot, wherein the perception modality includes visual modality, auditory modality and tactile modality, and the perception modality dimension is the total number of perception modality types of the robot; Based on each perception modality of the intelligent robot, a single-modal decision model corresponding to the robot's perception modality is trained; Obtain the perception data of the intelligent robot in each perception mode; Based on the perception data of the intelligent robot in each perception mode, a single-modal decision-making model corresponding to the robot's perception mode is used to determine the single-modal decision-making plan in each perception mode; Based on each unimodal decision scheme, construct a decision feature type set of each unimodal decision scheme and a decision feature value of each unimodal decision scheme; The decision feature type sets of all single-mode decision solutions are combined to obtain the total set of decision feature types; Determine at least one candidate decision feature value for each decision feature in the total set of decision feature types based on the decision feature value of each decision feature in the decision feature type set corresponding to each unimodal decision solution; Based on at least one candidate decision feature value of each decision feature in the total set of decision feature types, cross-fusion is performed to obtain at least one preliminary screening decision feature group; Based on the fusion analysis formula, the comprehensive decision index of each preliminary screening decision feature group is determined, and the preliminary screening decision feature group with the largest comprehensive decision index is selected as the optimal decision feature group of the intelligent robot; Output the multimodal comprehensive decision of the intelligent robot according to the optimal decision feature group.
2. The multimodal analysis method based on an intelligent robot according to claim 1, characterized in that: The training of a single-modal decision model corresponding to each perception modality of the robot based on each perception modality of the intelligent robot specifically includes: Using automatic or manual methods, obtain a number of sample sets under single-modal decision-making; The sample set is divided into a training set and a validation set in a ratio of 8:2; The training set is divided into a model training set and a model test set in a ratio of 7:3; Repeat the step of dividing the training set into a model training set and a model test set n times to obtain n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other; Based on n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other, construct at least one model regularization loss function; Using a neural network, determine the preliminary unimodal decision model trained using the training set under the regularized loss function of each model; Determine the performance of each preliminary unimodal decision model on the validation set and select the final unimodal decision model; Among them, the expression of the model regularization loss function is: L'(θ)=L(θ)+L∑|θ j | Among them, L is the coefficient of the regularization term, θ j is the jth parameter of the neural network, L'(θ) is the model regularization loss function, and L(θ) is the model loss function.
3. The multimodal analysis method based on an intelligent robot according to claim 2, characterized in that: The constructing at least one model regularization loss function based on n groups of model training sets that are not completely identical to each other and n groups of model test sets that are not completely identical to each other specifically includes: Obtaining at least one coefficient of a regularization term to be selected within a value range of the coefficient of the regularization term according to a set value step size; Substitute the coefficient of the candidate regularization term into the model regularization loss function to obtain the candidate model regularization loss function; Determine n first neural network models trained by using n sets of model training sets that are not completely identical to each other under the regularized loss function of each candidate model; Calculate the loss values of n first neural network models under n groups of model test sets that are not completely identical to each other; The loss values of n first neural network models under n groups of model test sets that are not completely identical to each other are averaged to obtain the fitting loss value of the coefficient of the regularization term to be selected; Screening out at least one coefficient of a candidate regularization term whose fitting loss value is within a preset interval as the coefficient of the determined regularization term; The regularization loss function of the candidate model corresponding to the coefficients of several determined regularization terms is used as the model regularization loss function.
4. The multimodal analysis method based on an intelligent robot according to claim 3, characterized in that: The step of respectively determining the performance of each preliminary unimodal decision model on the validation set and selecting the final unimodal decision model specifically includes: Calculate the loss value of each preliminary unimodal decision model on the validation set and record it as the actual loss value; Determine the fitting loss value of the coefficient of the regularization term of the model regularization loss function corresponding to the preliminary unimodal decision model; The preliminary unimodal decision model with the smallest difference between the actual loss value and the fitted loss value is selected as the final unimodal decision model.
5. The multimodal analysis method based on an intelligent robot according to claim 4, characterized in that: The cross-fusion step of obtaining at least one preliminary screening decision feature group based on at least one candidate decision feature value of each decision feature in the total set of decision feature types specifically includes: Select and only select one candidate decision feature value from at least one candidate decision feature value of each decision feature as the preliminary screening decision feature value, and combine all the preliminary screening decision feature values into a preliminary screening decision feature group; Repeat the above steps several times until no new initial screening decision feature group can be generated; Output all the initial screening decision feature groups obtained.
6. The multimodal analysis method of an intelligent robot according to claim 5, characterized in that: The comprehensive decision indicators for each initial screening decision feature group determined based on the fusion analysis formula specifically include: Extracting the preliminary screening decision feature value of the decision feature type in the decision feature type set of each single-modal decision solution from the preliminary screening decision feature group; The loss value is calculated based on the decision feature value of the unimodal decision scheme and the initial screening decision feature value, and is recorded as the unimodal loss value of the initial screening decision feature group for the unimodal type; Determine the unimodal loss value of each unimodal type of the initial screening decision feature group, and construct the unimodal loss value array A of the initial screening decision feature group j , A j =[a 1j …a ij …a mj ]; among them, A j is the unimodal loss value array of the jth initial screening decision feature group, a ij is the unimodal loss value of the jth initial screening decision feature group for the ith unimodal category, and m is the total number of unimodal categories; The unimodal loss value arrays of all initial screening decision feature groups are constructed into a unimodal loss value matrix B. k is the total number of initial screening decision feature groups; Based on the single-mode loss value matrix B, the comprehensive decision index of each initial screening decision feature group is calculated through the fusion analysis formula; The fusion analysis formula is specifically: R j is the comprehensive decision index of the jth initial screening decision feature group, a imin for a i1 …a ij …a ik The minimum value in a imax for a i1 …a ij …a ik The maximum value in .
7. An electronic device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal analysis method based on an intelligent robot as described in any one of claims 1-6.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal analysis method based on an intelligent robot according to any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Intelligent kitchen range hood central control gateway system
CN119713349A