Method, device, equipment and storage medium for determining execution instructions

Abstract

The present invention discloses a method, device, equipment and storage medium for determining an execution instruction. The method comprises: obtaining input information, wherein the input information comprises: at least one first execution instruction and at least one modal information, determining at least one second execution instruction according to the at least one modal information, and determining a target execution instruction according to the at least one first execution instruction and the at least one second execution instruction. Through the technical solution of the present invention, the embodiment of the present invention adopts a manager to obtain at least one first execution instruction and at least one modal information, to determine a second execution instruction according to the obtained multiple modal information, and finally to determine the target execution instruction based on the first execution instruction and the second execution instruction, thereby achieving a more friendly interaction method and obtaining a better interaction experience.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of vehicle technology, and in particular, to a method, device, equipment and storage medium for determining an execution instruction. Background Art

[0002] Generally speaking, modality refers to the way things happen or exist, and multimodality refers to the combination of various forms of two or more modalities. Each source or form of information can be called a modality. Multimodal fusion technology in deep learning is the process of processing different forms of data when the model completes analysis and recognition tasks. The fusion of multimodal data can provide more information for model decision-making, thereby improving the accuracy of the overall decision-making results. The purpose is to establish a model that can process and associate information from multiple modalities. The reason for fusing modalities is that different modalities have different expressions and different perspectives on things, so there are some crossovers, that is, information redundancy, and complementarity, that is, better than single features. There may even be multiple different information interactions between modalities. If multimodal information can be processed reasonably, rich feature information can be obtained.

[0003] There are two types of existing technical solutions. One is a method without multi-mode fusion, that is, after receiving the input from the input source, the operation is performed immediately. Since the multi-mode fusion method is not adopted, it may cause the phenomenon of erroneous operation after receiving the input information from the input source. The other is only through simple judgment and analysis, which makes the interactive experience worse. Summary of the invention

[0004] Embodiments of the present invention provide a method, device, equipment and storage medium for determining an execution instruction to enhance the interactive experience.

[0005] In a first aspect, an embodiment of the present invention provides a method for determining an execution instruction, comprising:

[0006] Acquiring input information, wherein the input information includes: at least one first execution instruction and at least one modal information;

[0007] determining at least one second execution instruction according to the at least one modality information;

[0008] A target execution instruction is determined according to the at least one first execution instruction and the at least one second execution instruction.

[0009] In a second aspect, an embodiment of the present invention further provides an execution instruction determination device, the device comprising:

[0010] An input information acquisition module, used to acquire input information, wherein the input information includes: at least one first execution instruction and at least one modal information;

[0011] A second execution instruction determination module, configured to determine at least one second execution instruction according to the at least one modal information;

[0012] The target execution instruction determination module is used to determine the target execution instruction according to the at least one first execution instruction and the at least one second execution instruction.

[0013] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0014] one or more processors;

[0015] A memory for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the processors implement the execution instruction determination method as described in any one of the embodiments of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an execution instruction determination method as described in any one of the embodiments of the present invention.

[0018] In the technical solution provided by the embodiment of the present invention, input information is obtained, wherein the input information includes: at least one first execution instruction and at least one modal information, and at least one second execution instruction is determined based on the at least one modal information, and then the target execution instruction is determined based on the at least one first execution instruction and the at least one second execution instruction. The phenomenon of erroneous operation after receiving input information from the input source is effectively avoided. Compared with the simple judgment and analysis method in the prior art, this embodiment needs to combine the information input by the user and the instructions generated by the modal information respectively, and make a comprehensive judgment, thereby better improving the interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a flow chart of a method for determining an execution instruction in an embodiment of the present invention;

[0021] Figure 1a is a flow chart of a method for obtaining training samples in an embodiment of the present invention;

[0022] Figure 2 is a structural diagram of an execution instruction determination device in an embodiment of the present invention;

[0023] Figure 3 is a schematic structural diagram of an electronic device in an embodiment of the present invention;

[0024] Figure 4 It is a schematic diagram of the structure of a computer-readable storage medium containing a computer program in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0026] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0027] The term “including” and its variations used in the present invention are open inclusions, that is, “including but not limited to.” The term “based on” means “based at least in part on.” The term “one embodiment” means “at least one embodiment.”

[0028] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0029] Embodiment 1

[0030] Figure 1 This is a flow chart of a method for determining an execution instruction provided by an embodiment of the present invention. This embodiment is applicable to the case of determining an execution instruction. The method can be executed by an execution instruction determination device in an embodiment of the present invention. The device can be implemented in software and / or hardware. Figure 1 As shown, the method specifically comprises the following steps:

[0031] S110: Obtain input information.

[0032] The input information obtained may be modal information or instruction information.

[0033] Specifically, the input information is obtained by the manager, which may be a controller mounted on an ECU in the car, usually but not limited to an infotainment host, a body, chassis, engine and other controllers, and an autonomous driving related controller.

[0034] Among them, the input information can be information input through the screen of controllers such as infotainment systems and air-conditioning systems, information input through voice, information input through a camera, information input through hard keys, buttons or smart surface keys on a steering wheel, air conditioning, or window, information input through an externally connected device, and information input through a built-in application.

[0035] Among them, the external connection devices may include:

[0036] Mobile phones, tablets, etc. connected to the infotainment system via USB, WIFI, Bluetooth;

[0037] Peripheral devices connected to the vehicle through the vehicle network connection (CAN network, Ethernet, etc.);

[0038] Connect normal other types of controllers, sensors, etc. through hardwired IO;

[0039] Connect to the vehicle's cloud server, etc. via the on-board Ethernet.

[0040] Among them, built-in applications may include:

[0041] The vehicle controller provides navigation and positioning information;

[0042] Road information and other navigation information transmitted from the cloud to the local area;

[0043] Weather information transmitted from the cloud to the local area;

[0044] Smart city information transmitted from the cloud to the local area.

[0045] Optionally, the input information includes: at least one first execution instruction and at least one modal information;

[0046] Among them, the first execution instruction can be an instruction generated by pressing a window button, or an instruction generated by pressing an air-conditioning button, or an instruction generated by clicking a preset area on a touch screen, or an instruction generated by clicking a switch of a vehicle-mounted terminal. The embodiment of the present invention does not limit this.

[0047] The modal information may be acquired through sensor collection or user input.

[0048] Specifically, the modal information can be:

[0049] The five senses of man: touch, hearing, sight, smell, taste;

[0050] Personal identity information: face, fingerprint, login account, etc.

[0051] Various human emotions: happy, angry, sad, wronged, crying, excited, etc.;

[0052] The medium of information: voice, video, text, graphics and pictures;

[0053] Sensors: such as radar, infrared, navigation and positioning information, temperature and humidity, air quality, odor sensors, lights, etc.;

[0054] Public information: time, time zone, weather, temperature, etc.;

[0055] Personal preferences: I like air conditioning at 25℃, I like pop music, I hate noisy environments, I hate traffic jams, etc.

[0056] Historical interaction information: crew operations and other interaction information within a certain period of time.

[0057] S120. Determine at least one second execution instruction according to the at least one modal information.

[0058] Among them, the second execution instruction can be an instruction for the window button, for example, it can be to open all the windows. The second execution instruction can also be an instruction for the air conditioner, for example, it can be to adjust the air conditioner temperature to 27 degrees. The second execution instruction can also be an instruction for the vehicle terminal, for example, it can be to adjust the sound to 35 decibels.

[0059] Specifically, the method for determining at least one second execution instruction based on the at least one modal information may be: pre-establishing a table, storing the correspondence between the modal information and the second execution instruction in the table, searching the table based on the at least one modal information, and obtaining at least one second execution instruction corresponding to the at least one modal information; the method for determining at least one second execution instruction based on the at least one modal information may also be: inputting the at least one modal information into an AI model to obtain at least one second execution instruction, wherein the AI ​​model is obtained by iteratively training a neural network model with a target sample set, and the target sample set includes: scene samples and execution instruction samples corresponding to the scene samples, and the scene samples include: at least one modal information sample.

[0060] Among them, the neural network model is described based on the mathematical model of neurons. It is a complex network system formed by a large number of simple processing units that are widely interconnected.

[0061] The neural network model may be a convolutional neural network model, or a deconvolutional neural network, or a recurrent neural network.

[0062] Optionally, determining at least one second execution instruction according to the at least one modal information may include:

[0063] The at least one modal information is input into the AI ​​model to obtain at least one second execution instruction, wherein the AI ​​model is obtained by iteratively training a neural network model through a target sample set, and the target sample set includes: a scene sample and an execution instruction sample corresponding to the scene sample, and the scene sample includes: at least one modal information sample.

[0064] It should be noted that the AI ​​model is obtained by iteratively training the neural network model with a target sample set, and the target sample set is continuously updated.

[0065] Optionally, iteratively training the neural network model through the target sample set may include:

[0066] Build a neural network model;

[0067] Inputting the at least one modal information sample into the neural network model to obtain a prediction execution instruction;

[0068] Acquire a first scene sample corresponding to the at least one modal information sample;

[0069] Obtaining a first execution instruction sample corresponding to the first scene sample;

[0070] Training the parameters of the neural network model according to the objective function formed by the predicted execution instruction and the first execution instruction sample;

[0071] Return to execute the operation of inputting the at least one modal information sample into the neural network model to obtain a prediction execution instruction until an AI model is obtained.

[0072] Specifically, the method of inputting the at least one modal information sample into the neural network model to obtain the predicted execution instruction may be: inputting at least one modal information sample into the neural network model, and the output of the neural network model is the predicted execution instruction.

[0073] Specifically, the method for obtaining the first scene sample corresponding to the at least one modal information sample is: pre-establishing a scene library, querying the scene library according to the at least one modal information sample, and obtaining the first scene sample corresponding to the at least one modal information sample.

[0074] Specifically, a method for obtaining the first execution instruction sample corresponding to the first scene sample is: querying a scene library according to the first scene sample to obtain the first execution instruction sample corresponding to the first scene sample.

[0075] Wherein, the objective function is the loss function of the neural network model.

[0076] S130. Determine a target execution instruction according to the at least one first execution instruction and the at least one second execution instruction.

[0077] For example, Figure 1a The figure is a flow chart of the training sample acquisition method in an embodiment of the present invention. The scene library contains multiple scene samples, each scene sample contains multiple modal information samples, and each scene sample corresponds to multiple execution instruction samples. Among them, post-scene operation collection refers to the process of updating the database according to the first operation after the target execution instruction is executed and the first operation is obtained within a preset time.

[0078] The technical solution of this embodiment obtains input information, wherein the input information includes: at least one first execution instruction and at least one modal information, and determines at least one second execution instruction based on the at least one modal information, and then determines the target execution instruction based on the at least one first execution instruction and the at least one second execution instruction. The phenomenon of misoperation after receiving input information is effectively avoided. Compared with the simple judgment and analysis method in the prior art, this embodiment needs to combine the information input by the user and the instructions generated by the modal information respectively, and make a comprehensive judgment, thereby better improving the interactive experience.

[0079] Optionally, after determining the target execution instruction according to the at least one first execution instruction and the at least one second execution instruction, the following steps may be performed:

[0080] The target execution instruction is executed.

[0081] Specifically, after determining the target execution instruction, the corresponding instruction is executed by the executor. The executor can be:

[0082] Infotainment system display screen: instrument screen, central control screen, co-pilot screen, rear screen, etc.;

[0083] Equipped with air conditioning, HUD, reversing surround view image, transparent A-pillar, streaming media rearview mirror, and driving recorder display screen;

[0084] Voice feedback information;

[0085] Vehicle prompt information or prompt sound, etc.;

[0086] Status lights for hard keys, buttons, or smart surface keys such as steering wheels, air conditioning, and windows;

[0087] Controllers for windows, doors, etc.

[0088] Optionally, after executing the target execution instruction, the following steps may be performed:

[0089] If the first operation is triggered within the preset time and the first operation meets the preset condition, determining the execution instruction to be added according to the first operation;

[0090] The target sample set is updated according to the execution instruction to be added.

[0091] The method for determining whether the first operation satisfies the preset condition may be: a user pre-establishes an operation list, and determines whether the first operation exists in the operation list; if the first operation exists in the operation list, the first operation satisfies the preset condition; if the first operation does not exist in the operation list, the first operation does not satisfy the preset condition. The method for determining whether the first operation satisfies the preset condition may be: a judgment model is pre-established, and the first operation is input into the judgment model; if the output result is yes, it is determined that the first operation satisfies the preset condition; if the output result is no, it is determined that the first operation does not satisfy the preset condition.

[0092] In this embodiment, the target sample set is a sample set obtained by collecting actual operations of users in different scenarios. The AI ​​model is obtained by iteratively training the neural network model with the target sample set. After executing the target execution instruction, the target sample set can also be updated according to some operations of the user within a certain period of time to further optimize the target sample set.

[0093] Specifically, the AI ​​model that inputs the modal information obtains the second execution instruction, and then determines the target execution instruction based on the first execution instruction and the second execution instruction. If the first operation is triggered within a period of time after executing the target execution instruction, information is collected for this operation. If this operation is in a list pre-set by the user, the execution instruction to be added is determined based on this operation, and then the target sample set is updated based on the execution instruction to be added.

[0094] The execution instruction to be added may be directly added to the instruction corresponding to the mode, or the instruction corresponding to the mode may be replaced by the execution instruction to be added.

[0095] Exemplarily, if there is an instruction sample corresponding to modality 1, the corresponding instruction in the original modality 1 is recorded as instruction A, and the instruction to be added is recorded as instruction B. At this time, the instruction B to be added can be directly added to the instruction corresponding to modality 1, or instruction A can be replaced with the instruction B to be added. If there is no instruction sample corresponding to modality 1, it is necessary not only to add the instruction B to be added directly to modality 1, but also to add it to the instruction corresponding to modality 1.

[0096] Embodiment 2

[0097] Figure 2 This is a schematic diagram of the structure of an execution instruction determination device provided by an embodiment of the present invention. This embodiment is applicable to the case of determining the execution instruction. The device can be implemented in software and / or hardware. The device can be integrated in any device that provides the functions of a computer, such as Figure 2 As shown, the execution instruction determination device specifically includes: an information acquisition module 210, a second execution instruction determination module 220 and a target execution instruction determination module 230.

[0098] The information acquisition module 210 is used to acquire input information, wherein the input information includes: at least one first execution instruction and at least one modal information;

[0099] A second execution instruction determining module 220, configured to determine at least one second execution instruction according to the at least one modal information;

[0100] The target execution instruction determination module 230 is configured to determine a target execution instruction according to the at least one first execution instruction and the at least one second execution instruction.

[0101] The technical solution of this embodiment, by acquiring at least one first execution instruction and at least one modal information, to determine a second execution instruction based on the acquired multiple modal information, and finally to determine the target execution instruction based on the first execution instruction and the second execution instruction, realizes a more friendly interaction method and obtains a better interaction experience.

[0102] Optionally, the second execution instruction determining module 220 includes:

[0103] The second execution instruction acquisition unit is used to input the at least one modal information into the AI ​​model to obtain at least one second execution instruction, wherein the AI ​​model is obtained by iteratively training a neural network model through a target sample set, and the target sample set includes: scene samples and execution instruction samples corresponding to the scene samples, and the scene samples include: at least one modal information sample.

[0104] Optionally, the second execution instruction acquisition unit is specifically used for:

[0105] Build a neural network model;

[0106] Inputting the at least one modal information sample into the neural network model to obtain a prediction execution instruction;

[0107] Acquire a first scene sample corresponding to the at least one modal information sample;

[0108] Obtaining a first execution instruction sample corresponding to the first scene sample;

[0109] Training the parameters of the neural network model according to the objective function formed by the predicted execution instruction and the first execution instruction sample;

[0110] Return to execute the operation of inputting the at least one modal information sample into the neural network model to obtain a prediction execution instruction until an AI model is obtained.

[0111] Optionally, also include:

[0112] The execution module is used to execute the target execution instruction after determining the target execution instruction according to the at least one first execution instruction and the at least one second execution instruction.

[0113] Optionally, also include:

[0114] An instruction determination module, configured to determine, after executing the target execution instruction, if a first operation is triggered within a preset time and the first operation satisfies a preset condition, an execution instruction to be added according to the first operation;

[0115] An updating module is used to update the target sample set according to the execution instruction to be added.

[0116] The technical solution of this embodiment obtains input information, wherein the input information includes: at least one first execution instruction and at least one modal information, and determines at least one second execution instruction based on the at least one modal information, and then determines the target execution instruction based on the at least one first execution instruction and the at least one second execution instruction. The phenomenon of misoperation after receiving input information is effectively avoided. Compared with the simple judgment and analysis method in the prior art, this embodiment needs to combine the information input by the user and the instructions generated by the modal information respectively, and make a comprehensive judgment, thereby better improving the interactive experience.

[0117] Embodiment 3

[0118] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. Figure 3 A block diagram of an electronic device 312 suitable for implementing embodiments of the present invention is shown. Figure 3 The electronic device 312 shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. The device 312 is a typical computing device for trajectory fitting functions.

[0119] like Figure 3 As shown, the electronic device 312 is in the form of a general purpose computing device. The components of the electronic device 312 may include, but are not limited to: one or more processors 316, a storage device 328, and a bus 318 connecting different system components (including the storage device 328 and the processor 316).

[0120] Bus 318 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnect (PCI) bus.

[0121] The electronic device 312 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 312, including volatile and non-volatile media, removable and non-removable media.

[0122] The storage device 328 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 330 and / or cache memory 332. The electronic device 312 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 334 may be used to read and write non-removable, non-volatile magnetic media ( Figure 3 not shown, usually called a "hard drive"). Although Figure 3 Not shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a read-only optical disk (Compact Disc-Read Only Memory, CD-ROM), a digital video disk (Digital Video Disc-Read Only Memory, DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 318 via one or more data medium interfaces. The storage device 328 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0123] A program 336 having a set (at least one) of program modules 326 may be stored, for example, in a storage device 328, such program modules 326 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 326 generally perform the functions and / or methods of the embodiments described herein.

[0124] The electronic device 312 may also communicate with one or more external devices 314 (e.g., keyboard, pointing device, camera, display 324, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 312, and / or communicate with any device that enables the electronic device 312 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 322. In addition, the electronic device 312 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN) and / or a public network, such as the Internet) through a network adapter 320. As shown, the network adapter 320 communicates with other modules of the electronic device 312 through a bus 318. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 312, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) systems, tape drives, and data backup storage systems.

[0125] The processor 316 executes various functional applications and data processing by running the program stored in the storage device 328, for example, implementing the execution instruction determination method provided in the above embodiment of the present invention, which includes:

[0126] Acquiring input information, wherein the input information includes: at least one first execution instruction and at least one modal information;

[0127] determining at least one second execution instruction according to the at least one modality information;

[0128] A target execution instruction is determined according to the at least one first execution instruction and the at least one second execution instruction.

[0129] Embodiment 4

[0130] Figure 4 Schematic diagram of the structure of a computer-readable storage medium containing a computer program in an embodiment of the present invention. The embodiment of the present invention provides a computer-readable storage medium 41 on which a computer program 410 is stored. When the program is executed by one or more processors, the execution instruction determination method provided in all the embodiments of the present application is implemented. The method includes:

[0131] Acquiring input information, wherein the input information includes: at least one first execution instruction and at least one modal information;

[0132] determining at least one second execution instruction according to the at least one modality information;

[0133] A target execution instruction is determined according to the at least one first execution instruction and the at least one second execution instruction.

[0134] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media or any combination of the above two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0135] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0136] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0137] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0138] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0139] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0142] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0143] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for determining an execution instruction, characterized in that: include: Acquire input information, wherein the input information includes: at least one first execution instruction and at least one modal information; the modal information is acquired through a sensor or input by a user; the modal information includes: People’s five senses, their identity information, their emotions, sensors, public information, personal preferences, and historical interaction information; determining at least one second execution instruction according to the at least one modality information; determining a target execution instruction according to the at least one first execution instruction and the at least one second execution instruction; executing the target execution instruction; If the first operation is triggered within the preset time and the first operation meets the preset condition, determining the execution instruction to be added according to the first operation; updating a target sample set according to the execution instruction to be added; the target sample set includes: a scene sample and an execution instruction sample corresponding to the scene sample, and the scene sample includes: at least one modality information sample; Among them, the method for judging whether the first operation meets the preset condition is: pre-establishing a judgment model, inputting the first operation into the judgment model, if the output result is yes, then determining that the first operation meets the preset condition, if the output result is no, then determining that the first operation does not meet the preset condition.

2. The method according to claim 1, characterized in that Determining at least one second execution instruction according to the at least one modal information includes: The at least one modal information is input into the AI ​​model to obtain at least one second execution instruction, wherein the AI ​​model is obtained by iteratively training a neural network model through a target sample set.

3. The method according to claim 2, characterized in that The neural network model is iteratively trained through the target sample set, including: Build a neural network model; Inputting the at least one modal information sample into the neural network model to obtain a prediction execution instruction; Acquire a first scene sample corresponding to the at least one modal information sample; Obtaining a first execution instruction sample corresponding to the first scene sample; Training the parameters of the neural network model according to the objective function formed by the predicted execution instruction and the first execution instruction sample; Return to execute the operation of inputting the at least one modal information sample into the neural network model to obtain a prediction execution instruction until an AI model is obtained.

4. An execution instruction determination device, characterized in that: include: The information acquisition module is used to acquire input information, wherein the input information includes: at least one first execution instruction and at least one modal information; the modal information is acquired through sensor collection or user input; the modal information includes: A person's five senses, a person's identity information, a person's various emotions, sensors, public information, personal preferences, and historical interaction information; a second execution instruction determination module, configured to determine at least one second execution instruction according to the at least one modal information; a target execution instruction determination module, configured to determine a target execution instruction according to the at least one first execution instruction and the at least one second execution instruction; an execution module, configured to execute the target execution instruction after determining the target execution instruction according to the at least one first execution instruction and the at least one second execution instruction; An instruction determination module, configured to determine, after executing the target execution instruction, if a first operation is triggered within a preset time and the first operation satisfies a preset condition, an execution instruction to be added according to the first operation; An updating module, configured to update a target sample set according to the execution instruction to be added; the target sample set includes: a scene sample and an execution instruction sample corresponding to the scene sample, and the scene sample includes: at least one modality information sample; Among them, the method for judging whether the first operation meets the preset condition is: pre-establishing a judgment model, inputting the first operation into the judgment model, if the output result is yes, then determining that the first operation meets the preset condition, if the output result is no, then determining that the first operation does not meet the preset condition.

5. The device according to claim 4, characterized in that The second execution instruction determination module includes: The second execution instruction acquisition unit is used to input the at least one modal information into the AI ​​model to obtain at least one second execution instruction, wherein the AI ​​model is obtained by iteratively training the neural network model through the target sample set.

6. The device according to claim 5, characterized in that The second execution instruction acquisition unit is specifically used for: Build a neural network model; Inputting the at least one modal information sample into the neural network model to obtain a prediction execution instruction; Acquire a first scene sample corresponding to the at least one modal information sample; Obtaining a first execution instruction sample corresponding to the first scene sample; Training the parameters of the neural network model according to the objective function formed by the predicted execution instruction and the first execution instruction sample; Return to execute the operation of inputting the at least one modal information sample into the neural network model to obtain a prediction execution instruction until an AI model is obtained.

7. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the processors are enabled to implement the method according to any one of claims 1 to 3.

8. A computer-readable storage medium containing a computer program, wherein the computer program is stored thereon, characterized in that: When the program is executed by one or more processors, the method according to any one of claims 1 to 3 is implemented.

Images