Decision-making method and device based on multiple modes, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a decision-making method and device based on multiple modes, equipment and a medium. The method comprises the steps of obtaining historical text data and historical image data used for decision making, obtaining a preset decision making model and a preset decision, determining a loss value of the preset decision making model generated by the preset decision making model according to the historical text data and the historical image data, obtaining feedback data corresponding to the preset decision making, and optimizing the preset decision making model according to the feedback data. And obtaining an optimized decision model, and generating a target decision according to the optimized decision model, the historical text data and the historical image data. According to the method and the device, the decision model is optimized according to the text data and the image data, so that the decision model can learn data in different modalities, the decision generated by the decision model can help investors to provide comprehensive market analysis, and the decision accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal decision-making method, device, equipment and medium. Background Art

[0002] An investment decision-making system is a tool that uses algorithms to assist investors in making more scientific and accurate investment decisions in complex financial markets. It mainly helps investors evaluate market trends, identify investment opportunities, and provide corresponding trading recommendations by integrating market data, financial news, company announcements, economic analysis reports, and investors' historical trading records. Existing financial investment decision-making systems usually use methods such as technical analysis, fundamental analysis, and quantitative trading to achieve more efficient decision-making and risk control. However, with the rapid growth of market information and the diversification of multimodal data (such as text, charts, news, historical records, etc.), traditional financial investment decision-making systems often perform poorly when faced with massive, multi-dimensional information. Therefore, in the face of diversified data, how to improve decision-making accuracy is an urgent problem to be solved. Summary of the invention

[0003] In view of this, embodiments of the present invention provide a multimodal decision-making method, apparatus, device and medium to solve the problem of low decision-making accuracy in diversified data.

[0004] In a first aspect, an embodiment of the present invention provides a multi-modal decision-making method, the decision-making method comprising: Acquire historical text data and historical image data for decision-making; Obtaining a preset decision model and a preset decision, and determining a loss value of the preset decision generated by the preset decision model according to the historical text data and the historical image data; Acquire feedback data corresponding to the preset decision, and optimize the preset decision model according to the feedback data to obtain an optimized decision model; A target decision is generated according to the optimized decision model, the historical text data and the historical image data.

[0005] In a second aspect, an embodiment of the present invention provides a multi-modal decision-making device, the decision-making device comprising: An acquisition module, used to acquire historical text data and historical image data for decision-making; A determination module, used to obtain a preset decision model and a preset decision, and determine a loss value of the preset decision generated by the preset decision model according to the historical text data and the historical image data; An optimization module, used to obtain feedback data corresponding to the preset decision, and optimize the preset decision model according to the feedback data to obtain an optimized decision model; A generation module is used to generate a target decision based on the optimized decision model, the historical text data and the historical image data.

[0006] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the decision-making method described in the first aspect when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the decision-making method as described in the first aspect is implemented.

[0008] Compared with the prior art, the present invention has the following beneficial effects: Obtain historical text data and historical image data for decision-making, obtain a preset decision model and a preset decision, determine the loss value of the preset decision generated by the preset decision model based on the historical text data and historical image data, obtain feedback data corresponding to the preset decision, optimize the preset decision model based on the feedback data, obtain an optimized decision model, and generate a target decision based on the optimized decision model and the historical text data and historical image data. In this application, the decision model is optimized based on text data and image data, so that the decision model can learn data in different modalities, so that the decision generated by the decision model can help investors provide a full range of market analysis and improve decision accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0010] Figure 1 It is a schematic diagram of an application environment of a multi-modal decision-making method provided in an embodiment of the present application; Figure 2 is a flowchart of a multi-modal decision-making method provided by an embodiment of the present invention; Figure 3 is a structural block diagram of a multi-modal decision-making device provided by an embodiment of the present invention; Figure 4It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0012] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0013] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0014] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0016] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0018] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0019] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0020] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0021] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0022] A multi-modal decision-making method provided by an embodiment of the present invention can be applied in the following aspects: Figure 1In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0023] See also Figure 2 , is a flowchart of a multi-modal decision-making method provided by an embodiment of the present invention. The multi-modal decision-making method can be applied to Figure 1 The server in Figure 2 As shown, the multimodal decision-making method may include the following steps.

[0024] S201: Acquire historical text data and historical image data for decision-making.

[0025] In step S201, historical text data and historical image data used for decision-making are obtained, wherein the historical text data is text data related to the decision, and the historical image data is image data related to the decision.

[0026] In this embodiment, before obtaining historical text data and historical image data for decision-making, multiple financial data source information is obtained, such as stock prices, market news, company announcements, economic analysis reports, etc., and the multiple financial data source information is processed and divided into text data and image data, wherein the image data is the corresponding non-text data, and the image data includes corresponding chart data and table data, etc.

[0027] It should be noted that obtaining historical text data and historical image data for decision-making, that is, obtaining data of different modalities, so that the data of different modalities can be used as the input of the decision-making model, so that the decision-making model can learn multi-source data information and improve the output accuracy of the decision-making model.

[0028] S202: Obtain a preset decision model and a preset decision, and determine a loss value of the preset decision generated by the preset decision model according to the historical text data and the historical image data; In step S202, a preset decision model and a preset decision are obtained, wherein the preset decision model is a pre-trained decision model. The preset decision is a personalized decision generated based on historical transaction records and risk preferences, etc., and the loss value of the preset decision generated by the preset decision model is determined based on historical text data and historical image data.

[0029] In this embodiment, a preset decision model and a preset decision are obtained, wherein the preset decision model is a pre-trained decision model, which can be a decision model obtained through technical analysis, fundamental analysis, quantitative trading, etc. The preset decision is a personalized decision generated by automatically analyzing the latest market trends based on real-time monitoring of market changes and combining historical transaction records and risk preferences, such as short-term trading and long-term holding.

[0030] According to the historical text data and the historical image data, the loss value of the preset decision generated by the preset decision model is determined, that is, the difference between the decision generated according to the historical text data and the historical image data and the preset decision.

[0031] In this embodiment, the preset decision model generates different decisions based on historical text data and historical image data, each decision corresponds to a different probability value, and the corresponding loss value is calculated based on the probability value corresponding to the preset decision.

[0032] In this embodiment, the loss value of the preset decision generated by the preset decision model is determined based on historical text data and historical image data, so that the decision generated by the preset decision model is close to the decision generated based on real-time monitoring of market changes, the latest market trends, and combined with investors' historical trading records and risk preferences, thereby improving the personalization and accuracy of the decision.

[0033] Optionally, determining the loss value of a preset decision generated by a preset decision model according to the historical text data and the historical image data includes: Perform feature extraction on historical text data to obtain text features, and perform feature extraction on historical image data to obtain image features; Fuse the text features with the image features to obtain fused features; According to the fused features, the loss value of the preset decision model to generate the preset decision is determined.

[0034] In this embodiment, feature extraction is performed on historical text data to obtain text features. When feature extraction is performed on historical text data, a text encoder can be used to extract text features. For example, the text encoder is composed of a character embedding layer, a position encoding layer and a FFTBlock module (the module is composed of multiple convolutional layers and a self-attention mechanism layer). The historical text data is processed by the character embedding layer and encoded into a sequence vector of fixed dimension. The position vector is constructed by the position encoding layer. The encoded text sequence vector and the corresponding position of the position vector are added. The result of the addition is sent to the multi-layer FFTBlock module and converted into a text encoding feature of fixed length to obtain text features.

[0035] Perform feature extraction on the historical image data to obtain image features. When performing feature extraction on the historical image data, an image encoder can be used to extract image features, such as a VGG-19 neural network model. The historical image data is input into the VGG-19 neural network model to obtain a corresponding visual feature vector, i.e., image features.

[0036] The text features and the image features are fused to obtain fused features. When fusing features, the text features and the image features can be projected into the same space to obtain text features and image features of the same dimension. The text features and image features of the same dimension are spliced ​​and fused or superimposed and fused to obtain fused features.

[0037] When determining the loss value of a preset decision generated by a preset decision model based on the fused features, the fused features are input into the preset decision model, and the corresponding loss value is calculated based on the probability value of the preset decision output by the preset decision model.

[0038] Optionally, the text features are fused with the image features to obtain fused features, including: Perform spatial projection on the text features and the image features to obtain projected text features and projected image features; Calculate the attention weight value according to the correlation between the projected text features and the projected image features; According to the attention weight value, the projected text features and the projected image features are fused to obtain the fused features.

[0039] In this embodiment, the text features and the image features are spatially projected to obtain the projected text features and the projected image features, wherein the projected text features and the projected image features have the same dimensions. The projected text features and the projected image features are fused based on the attention mechanism. During fusion, the attention mechanism is used for fusion, that is, the attention weight value is calculated based on the correlation between the projected text features and the projected image features, and the attention weight value includes the weight value of the projected text features and the weight value of the projected image features.

[0040] Among them, when calculating the weight value of the projected text feature, the calculation formula is as follows: in, is the weight value of the projected text feature, is the text feature, is the image feature, and is the projection matrix, is the text feature after projection, is the image feature after projection, is a fixed value.

[0041] According to the weight value of the projected text feature, the weight value of the projected image feature is calculated. The calculation formula is as follows: in, is the weight value of the projected image feature, is the weight value of the projected text feature.

[0042] According to the attention weight value, the projected text feature and the projected image feature are feature fused to obtain a fused feature. In this embodiment, the projected text feature is multiplied by the weight value of the projected text feature to obtain a text attention feature, the projected image feature is multiplied by the weight value of the projected image feature to obtain an image attention feature, and the text attention feature and the image attention feature are feature spliced ​​or feature superimposed to obtain a fused feature.

[0043] Optionally, determining a loss value of a preset decision generated by a preset decision model according to the fused features includes: According to the fused features, determine the probability value of the preset decision model generating the preset decision; The loss value is calculated based on the probability value and the preset loss function.

[0044] In this embodiment, according to the fused features, the probability value of the preset decision model generating the preset decision is determined, that is, the fused features are input into the preset decision model, and the probability value of the preset decision model outputting the preset decision is determined. According to the probability value and the preset loss function, the loss value is calculated, wherein the formula for calculating the loss value is as follows: in, Loss value, are the parameters in the preset decision model, To make a pre-determined decision, The parameters in the preset decision model are When , the preset decision model generates the probability value of the preset decision based on the text features and image features.

[0045] It should be noted that the preset loss function is a function of the parameters in the preset decision model, and changes with the changes of the parameters in the preset decision model.

[0046] S203: Obtain feedback data corresponding to the preset decision, and optimize the preset decision model according to the feedback data to obtain an optimized decision model.

[0047] In step S203, feedback data corresponding to the preset decision is obtained, wherein the feedback data includes corresponding rate of return data and risk data. The preset decision model is optimized according to the feedback data, that is, the values ​​of the parameters in the preset decision model are adjusted to obtain an optimized decision model.

[0048] In this embodiment, feedback data corresponding to the preset decision is obtained, and the feedback data at least includes corresponding rate of return data and risk data, wherein the rate of return data is the rate of return obtained after executing the preset decision, and the risk data is the size of risk corresponding to the execution of the preset decision. When the rate of return is high, it is considered that the preset decision meets the user requirements, and when the rate of return is low, it is considered that the preset decision does not meet the user requirements.

[0049] The preset decision model is optimized according to the feedback data to obtain an optimized decision model. During the optimization, the parameters in the preset model are updated according to the feedback data so that the decision output by the optimized decision model better meets the user requirements.

[0050] Optionally, the preset decision model is optimized according to the feedback data to obtain an optimized decision model, including: Determine the update strategy of the parameters in the preset decision model based on the feedback data; The preset decision model is optimized according to the update strategy to obtain the optimized decision model.

[0051] In this embodiment, according to the feedback data, the updating strategy of the parameters in the preset decision model is determined, wherein the updating strategy includes positive incremental updating of the parameters in the preset decision model and negative decremental updating of the parameters in the preset decision model. If the updating strategy is positive incremental updating of the parameters in the preset decision model, the parameters in the preset decision model are incremented, and if the updating strategy is negative decremental updating of the parameters in the preset decision model, the parameters in the preset decision model are decremented. The preset decision model is optimized according to the updating strategy to obtain an optimized decision model.

[0052] Optionally, determining a strategy for updating parameters in a preset decision model based on the feedback data includes: If the feedback data is positive incentive, the parameters in the preset decision model are updated in a positive incremental manner; If the feedback data is a negative stimulus, the parameters in the preset decision model are updated in a negative decreasing direction.

[0053] In this embodiment, the update strategy includes positive incremental updates to the parameters in the preset decision model and negative decremental updates to the parameters in the preset decision model. If the feedback data is a positive stimulus, the parameters in the preset decision model are positively incrementally updated; if the feedback data is a negative stimulus, the parameters in the preset decision model are negatively decremented updated.

[0054] It should be noted that when judging whether the feedback data is a positive incentive or a negative incentive, it can be determined based on the rate of return and the risk in the feedback data. The incentive value can be determined based on the rate of return and the risk in the feedback data. The calculation formula of the incentive value is as follows: in, is the incentive value, is the rate of return, U is the risk, and is the weight value, where is a positive value, It is a negative value, that is, the greater the rate of return, the greater the corresponding incentive value, and the greater the risk, the smaller the corresponding incentive value. When the incentive value is greater than 0, it is a positive incentive, and when the incentive value is less than 0, it is a negative incentive.

[0055] It should be noted that the size of the rate of return and the size of the risk can be determined based on the decision output by the decision model after the last update, that is, the historical text data and the historical image data are input into the decision model after the last update, and the corresponding decision is output. According to the corresponding decision, the rate of return and risk based on the corresponding decision are calculated.

[0056] It should be noted that, when calculating the incentive value, other parameters besides the rate of return and the risk parameter may also be used for calculation, such as investment parameters, etc., which are not limited in this embodiment.

[0057] Optionally, the preset decision model is optimized according to the update strategy to obtain an optimized decision model, including: The parameters of the preset decision model are updated according to the update strategy to obtain an updated decision model; According to the historical text data and the historical image data, the updated decision model is calculated to generate an updated loss value for the preset decision; When the update loss value is greater than a preset threshold, executing the step of updating the parameters of the preset decision model according to the update strategy to obtain an updated decision model; When the update loss value is not greater than a preset threshold, the updated decision model is determined as the optimized decision model.

[0058] In this implementation, when optimizing the preset decision model according to the update strategy, the parameters of the preset decision model are updated according to the update strategy to obtain an updated decision model, wherein the parameter update formula is as follows: in, is the updated parameter, is the parameter after the last update, is the learning rate, is the loss function The gradient value of , are the parameters in the preset decision model, To make a pre-determined decision, The parameters in the preset decision model are When , the preset decision model generates the probability value of the preset decision based on the text features and image features.

[0059] Based on the historical text data and the historical image data, the updated loss value of the preset decision generated by the updated decision model is calculated. During the calculation, feature extraction is performed on the historical text data to obtain text features, and feature extraction is performed on the historical image data to obtain image features. The text features and image features are fused to obtain fused features. Based on the fused features, the probability value of the updated decision model generating the preset decision is determined, and the loss value is calculated based on the probability value and the preset loss function.

[0060] When the update loss value is greater than the preset threshold, the parameters of the decision model continue to be updated, that is, the parameters of the preset decision model are updated according to the update strategy to obtain the updated decision model. When the update loss value is not greater than the preset threshold, the parameters of the preset decision model are stopped from being updated, and the updated decision model is determined as the optimized decision model.

[0061] S204: Generate a target decision based on the optimized decision model, historical text data and historical image data.

[0062] In this embodiment, a target decision is generated based on the optimized decision model, historical text data and historical image data, that is, the fusion features after the fusion of historical text data and historical image data are input into the optimized decision model to output the target decision.

[0063] Obtain historical text data and historical image data for decision-making, obtain a preset decision model and a preset decision, determine the loss value of the preset decision generated by the preset decision model based on the historical text data and historical image data, obtain feedback data corresponding to the preset decision, optimize the preset decision model based on the feedback data, obtain an optimized decision model, and generate a target decision based on the optimized decision model and the historical text data and historical image data. In this application, the decision model is optimized based on text data and image data, so that the decision model can learn data in different modalities, so that the decision generated by the decision model can help investors provide a full range of market analysis and improve decision accuracy.

[0064] See also Figure 3 , Figure 3 is a structural block diagram of a multimodal decision-making device provided by an embodiment of the present invention, and the multimodal decision-making device is applied to the above-mentioned server. For the convenience of explanation, only the part related to the embodiment of the present application is shown. Figure 3 The decision-making device 30 includes an acquisition module 31, a determination module 32, an optimization module 33, and a generation module 34.

[0065] The acquisition module 31 is used to acquire historical text data and historical image data for decision-making.

[0066] The determination module 32 is used to obtain a preset decision model and a preset decision, and determine the loss value of the preset decision generated by the preset decision model according to the historical text data and the historical image data.

[0067] The optimization module 33 is used to obtain feedback data corresponding to the preset decision, and optimize the preset decision model according to the feedback data to obtain an optimized decision model.

[0068] The generation module 34 is used to generate a target decision based on the optimized decision model, historical text data and historical image data.

[0069] Optionally, the determination module 32 includes: The extraction unit is used to perform feature extraction on historical text data to obtain text features, and to perform feature extraction on historical image data to obtain image features.

[0070] A obtaining unit is used to perform feature fusion on the text feature and the image feature to obtain fused features; According to the fused features, the loss value of the preset decision model to generate the preset decision is determined.

[0071] Optionally, the obtaining unit includes: The projection subunit is used to perform spatial projection on the text features and the image features to obtain the projected text features and the projected image features.

[0072] The first calculation subunit is used to calculate the attention weight value according to the correlation between the projected text features and the projected image features.

[0073] The second computing subunit is used to perform feature fusion on the projected text features and the projected image features according to the attention weight value to obtain fused features.

[0074] Optionally, the optimization module 33 includes: The determination unit is used to determine the updating strategy of the parameters in the preset decision model according to the feedback data.

[0075] The optimization unit is used to optimize the preset decision model according to the update strategy to obtain the optimized decision model.

[0076] Optionally, the determining unit includes: The first updating subunit is used to perform positive incremental update on the parameters in the preset decision model if the feedback data is positive stimulation.

[0077] The second updating subunit is used to perform negative decreasing updates on the parameters in the preset decision model if the feedback data is a negative stimulus.

[0078] Optionally, the above optimization unit includes: The updating subunit is used to update the parameters of the preset decision model according to the updating strategy to obtain the updated decision model.

[0079] The calculation subunit is used to calculate the updated loss value of the preset decision generated by the updated decision model based on the historical text data and the historical image data.

[0080] The execution subunit is used to execute the step of updating the parameters of the preset decision model according to the update strategy to obtain the updated decision model when the update loss value is greater than the preset threshold.

[0081] The determination subunit is used to determine the updated decision model as the optimized decision model when the update loss value is not greater than a preset threshold.

[0082] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0083] Figure 4 FIG. 1 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned multi-modal decision-making method embodiments are implemented.

[0084] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0085] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0086] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0087] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0088] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing the computer program product.

[0089] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0090] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0091] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0092] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0093] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0094] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A decision-making method based on multimodality, characterized in that: The decision-making method includes: Acquire historical text data and historical image data for decision-making; Obtaining a preset decision model and a preset decision, and determining a loss value of the preset decision generated by the preset decision model according to the historical text data and the historical image data; Acquire feedback data corresponding to the preset decision, and optimize the preset decision model according to the feedback data to obtain an optimized decision model; A target decision is generated according to the optimized decision model, the historical text data and the historical image data.

2. The decision-making method according to claim 1, characterized in that: The step of determining the loss value of the preset decision generated by the preset decision model according to the historical text data and the historical image data includes: Performing feature extraction on the historical text data to obtain text features, and performing feature extraction on the historical image data to obtain image features; Fusing the text feature with the image feature to obtain a fused feature; According to the fused features, a loss value of the preset decision generated by the preset decision model is determined.

3. The decision-making method according to claim 2, characterized in that: The step of fusing the text feature with the image feature to obtain a fused feature includes: Performing spatial projection on the text features and the image features to obtain projected text features and projected image features; Calculating an attention weight value according to the correlation between the projected text features and the projected image features; According to the attention weight value, feature fusion is performed on the projected text feature and the projected image feature to obtain a fused feature.

4. The decision-making method according to claim 2, characterized in that: The step of determining, based on the fused features, a loss value of the preset decision model for generating the preset decision comprises: Determining, based on the fused features, a probability value of the preset decision model generating the preset decision; The loss value is calculated based on the probability value and a preset loss function.

5. The decision-making method according to claim 1, characterized in that: The step of optimizing the preset decision model according to the feedback data to obtain an optimized decision model includes: Determining an update strategy for parameters in the preset decision model according to the feedback data; The preset decision model is optimized according to the update strategy to obtain an optimized decision model.

6. The decision-making method according to claim 5, characterized in that: The update strategy includes performing positive incremental update on the parameters in the preset decision model and performing negative decremental update on the parameters in the preset decision model; Determining, based on the feedback data, a strategy for updating parameters in the preset decision model includes: If the feedback data is a positive stimulus, the parameters in the preset decision model are updated in a positive and incremental manner; If the feedback data is a negative stimulus, the parameters in the preset decision model are updated in a negative decreasing direction.

7. The decision-making method according to claim 5, characterized in that: The step of optimizing the preset decision model according to the update strategy to obtain an optimized decision model includes: Updating the parameters of the preset decision model according to the update strategy to obtain an updated decision model; Calculating, based on the historical text data and the historical image data, an update loss value of the updated decision model to generate a preset decision; When the update loss value is greater than a preset threshold, executing the step of updating the parameters of the preset decision model according to the update strategy to obtain an updated decision model; When the update loss value is not greater than a preset threshold, the updated decision model is determined as the optimized decision model.

8. A decision-making device based on multimodality, characterized in that: The decision-making device comprises: An acquisition module, used to acquire historical text data and historical image data for decision-making; A determination module, used to obtain a preset decision model and a preset decision, and determine a loss value of the preset decision generated by the preset decision model according to the historical text data and the historical image data; An optimization module, used to obtain feedback data corresponding to the preset decision, and optimize the preset decision model according to the feedback data to obtain an optimized decision model; A generation module is used to generate a target decision based on the optimized decision model, the historical text data and the historical image data.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the decision-making method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the decision-making method according to any one of claims 1 to 7 is implemented.

Images