A model determination method, device, equipment and medium

Abstract

The present application provides a model determination method, apparatus, device and medium to obtain a command information set and a device behavior set; input the command information set and the device behavior set into an initial pre-trained model to determine the device behavior that has information similarity conditions to the command information; use the command information as sample content and the device behavior corresponding to the command information as sample pseudo-labels to perform supervised training on the initial pre-trained model to obtain a target model; the target model is used to determine the corresponding target device behavior based on the input target command information. The target model can determine the corresponding target device behavior based on the target command information issued by the user, and the user can control the target device through one or more types of commands, and the device control method is more diverse. There is no need to input specific command information, and device control can be achieved for relatively vague commands, which improves the flexibility of device control, reduces the user's operating difficulty, and improves the user experience.

Description

Technical Field

[0001] The present application relates to the field of computer model processing, and in particular to a model determination method, device, equipment and medium. Background Art

[0002] Traditional device control relies on limited interaction modes, such as physical buttons, touch screens, or simple voice commands. This single or limited interaction method limits the user's control experience. Users can only control the device through one interaction method, which lacks flexibility.

[0003] In addition, when controlling a device, the user needs to issue specific instructions to control the device to perform the corresponding action. However, the intention of the instruction designer may not be completely consistent with the subjective understanding of the actual user, which may cause the user to not understand the preset specific instructions, and the instructions issued by the user are different from the specific instructions.

[0004] For example, when the user issues a voice command of "please open the curtains", the device can open the curtains. However, if the user issues a voice command of "pull the curtains open", the device cannot execute the command to open the curtains because the command is different from the specific command set by the device, which increases the difficulty of using the device. Therefore, in the related art, the user's control method for the device is relatively single, and the control command is too rigid, which reduces the user's experience of using the device. Summary of the invention

[0005] In view of this, the purpose of this application is to provide a model determination method, device, equipment and medium, so that users can control the target device through one or more types of instructions, and the device control method is more diverse. The device control can also be achieved for relatively vague instructions, which improves the flexibility of device control, reduces the user's operating difficulty, and improves the user experience. The specific scheme is as follows:

[0006] In one aspect, the present application provides a model determination method, the method comprising:

[0007] Acquire a command information set and a device behavior set; the command information set includes a plurality of command information, the command information includes at least one of image command information and voice command information, and the device behavior set includes a plurality of device behaviors performed by the target device;

[0008] Inputting the instruction information set and the device behavior set into an initial pre-training model to determine the device behavior having information similar to the instruction information;

[0009] The instruction information is used as sample content, and the device behavior corresponding to the instruction information is used as a sample pseudo label, and supervised training is performed on the initial pre-trained model to obtain a target model; the target model is used to determine the corresponding target device behavior according to the input target instruction information.

[0010] In another aspect, an embodiment of the present application further provides a model determination device, the device comprising:

[0011] An acquisition unit, configured to acquire a command information set and a device behavior set; the command information set includes a plurality of command information, the command information includes at least one of image command information and voice command information, and the device behavior set includes a plurality of device behaviors performed by the target device;

[0012] A determination unit, configured to input the instruction information set and the device behavior set into an initial pre-training model, and determine the device behavior having information similarity conditions to the instruction information;

[0013] A training unit is used to use the instruction information as sample content and the device behavior corresponding to the instruction information as sample pseudo-label to perform supervised training on the initial pre-trained model to obtain a target model; the target model is used to determine the corresponding target device behavior based on the input target instruction information.

[0014] In another aspect, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory:

[0015] The memory is used to store program code and transmit the program code to the processor;

[0016] The processor is configured to execute the method described above according to the instructions in the program code.

[0017] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the above aspects.

[0018] The embodiment of the present application provides a model determination method, device, equipment and medium, which obtains a command information set and a device behavior set; the command information set includes multiple command information, the command information includes at least one of image command information and voice command information, and the device behavior set includes multiple device behaviors made by the target device, so that the user can control the target device through one or more types of commands, and the device control method is more diverse. The command information set and the device behavior set are input into the initial pre-training model to determine the device behavior with information similar to the command information; the command information is used as the sample content, and the device behavior corresponding to the command information is used as the sample pseudo-label to supervise the initial pre-training model to obtain the target model; the target model is used to determine the corresponding target device behavior according to the input target command information. In this way, by using the command information and the corresponding device behavior for model training, the obtained target model can determine the corresponding target device behavior according to the target command information issued by the user, without the need to input specific command information, and the device control can also be achieved for relatively vague commands, which improves the flexibility of device control, reduces the user's operating difficulty, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a flow chart of a model determination method provided in an embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram of the structure of an initial pre-training model provided in an embodiment of the present application is shown;

[0022] Figure 3 A schematic diagram of a model training process provided by an embodiment of the present application is shown;

[0023] Figure 4 A structural block diagram of a model determination device provided in an embodiment of the present application;

[0024] Figure 5 A structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below with reference to the accompanying drawings.

[0026] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0027] To facilitate understanding, a model determination method, device, equipment and medium provided in an embodiment of the present application are described in detail below with reference to the accompanying drawings.

[0028] refer to Figure 1 As shown, it is a flow chart of a model determination method provided in an embodiment of the present application, and the method may include the following steps.

[0029] S101, obtaining a command information set and a device behavior set.

[0030] In the embodiment of the present application, a command information set and a device behavior set can be obtained. The command information set may include multiple command information, and the command information set is a set composed of a large amount of command information. The command information can be understood as a command for controlling the target device. The command information may include at least one of image command information and voice command information. The image command information may be a captured image photo, and the actions performed by the user, such as gestures, may be reflected in the image command information. The voice command information may be the collected sound information, such as the words spoken by the user.

[0031] In addition, the command information can also be command information of other modes, which can be obtained through various sensors or detectors and converted into text form. For example, the current room temperature can be sensed as 25°C through a temperature sensor and converted into the text information "Current room temperature 25°C".

[0032] The command information may be image command information, that is, the captured user image is used as the command information for controlling the target device, or may be voice command information, that is, the voice emitted by the user is used as the command information. In addition, the command information may include both image command information and voice command information. In this case, when the user controls the target device, the corresponding device control may be implemented based on the captured user image and the user voice at the same time.

[0033] The device behavior set may include multiple device behaviors performed by the target device. The target device may be understood as the device to be controlled, and the device behavior may be understood as some actions performed by the device. The device behavior is set according to the target device. For example, if the target device is a smart curtain, the device behavior may be to open the curtain, close the curtain, etc.

[0034] S102, inputting the instruction information set and the device behavior set into the initial pre-training model to determine the device behavior that has information similar to the instruction information.

[0035] In an embodiment of the present application, the initial pre-trained model may be a model that requires training and adjustment, and both the instruction information set and the device behavior set may be input into the model so that the model may match the instruction information and the device behavior and determine the device behavior corresponding to the instruction information.

[0036] Among them, the command information and the device behavior have information similarity conditions, which can be understood as the command information and the device behavior both represent the same device action, and the meaning of their information is the same. For example, the device behavior "open the curtains" and the voice command information "please open the curtains" meet the information similarity condition, and the device behavior "open the curtains" and the image command information for the user to make an OK gesture meet the information similarity condition. In addition, the initial pre-trained model can be an existing model that has been trained to a certain extent, which can achieve the mutual correspondence between device behavior and command information.

[0037] When the instruction information is image instruction information, the device behavior and the image instruction information meet the information similarity condition. When the instruction information is voice instruction information, the device behavior and the voice instruction information meet the information similarity condition. When the instruction information includes both image instruction information and voice instruction information, the device behavior and the two types of instruction information meet the information similarity condition. For example, when the device behavior is "open the curtains", the corresponding instruction information is that the user makes an OK gesture and speaks "please open the curtains".

[0038] S103, taking the instruction information as sample content and the device behavior corresponding to the instruction information as sample pseudo-label, performing supervised training on the initial pre-trained model to obtain a target model.

[0039] In an embodiment of the present application, instruction information can be used as sample content, and the device behavior corresponding to the instruction information can be used as a sample label to form a sample pair, so as to train the initial pre-trained model and obtain a trained target model.

[0040] The sample pairs may be sample pairs consisting of image command information and device behavior, or sample pairs consisting of voice command information and device behavior, or sample pairs consisting of image command information, voice command information and device behavior. In this way, the trained target model can determine the corresponding device behavior based only on the collected user image, or can determine the device behavior based only on the collected user voice, or can comprehensively consider the user's overall actions, that is, combine the user's image and voice to comprehensively judge the user's intention and determine the device behavior.

[0041] When training the model, the instruction information and the corresponding correct device behavior can be used as positive samples, and the instruction information and the corresponding incorrect device behaviors can be used as negative samples. By training the model with positive and negative samples, the model can better learn the relevant knowledge of which device behavior the instruction information corresponds to, and the model training effect is better. For example, if the similarity between instruction information L1 and device behavior H4 is 83%, they can be used as positive samples, and the similarity between instruction information L1 and device behavior H2 is 5%, which can be used as negative samples.

[0042] In addition, when determining the sample pairs for training, since the accuracy of the device behavior determined by the initial pre-trained model may not be very high, the similarity between some instruction information and the device behavior may be at an intermediate level, for example, around 50%, which means that the matching results between these instruction information and the device behavior are not accurate enough. In this case, such samples can be discarded and do not participate in the model training process to avoid affecting the model training effect.

[0043] The target model can be used to determine the corresponding target device behavior based on the input target command information. That is, in the model application stage, the user can input target command information into the target model. The target command information can be at least one of target image command information and target voice command information. For example, the target command information can be an image captured by a camera, or can be voice captured by a sound collection device, etc. Of course, both images and voices can be collected at the same time as target command information.

[0044] Input the target instruction information to the target model, and the model can output the corresponding target device behavior, that is, the target instruction information and the target device behavior meet the information similarity condition, and then the target device can be controlled to execute the target device behavior, thereby realizing the corresponding control of the target device in accordance with the user's intention. That is, the trained target model can be used to determine the corresponding device behavior for the target instruction information issued by the user, and can be applied to various scenarios that require interactive control, such as smart home, medical monitoring and other scenarios, which improves the user experience.

[0045] In this way, the target model can determine the corresponding device behavior based on the collected voice commands, and can also determine the corresponding device behavior based on the collected image commands. It can realize multimodal input and is not limited to a single modality, thereby improving the input diversity of the model and expanding the use scenarios of the model.

[0046] In addition, by using the trained target model to determine the device behavior, the user does not need to input specific control commands. The model can learn the relevant command knowledge of various device behaviors and can better "understand" the command information issued by the user. For more vague command information, the model can also determine the accurate device behavior. For example, for the voice command "open the curtain", it can also accurately determine that the device behavior is to open the curtain, which greatly improves the accuracy of determining device behavior, makes full use of representation learning technology, can adapt to the user's personalized input, and does not need the user to repeatedly interact with the target device when the target device cannot perform the correct device behavior, thereby reducing the cost of use.

[0047] In addition, when both voice command information and image command information can trigger the target device to perform an action, the image and voice can be comprehensively considered. In particular, when the device behavior corresponding to the image command information is different from the device behavior corresponding to the voice command information, by comprehensively considering the image and voice and treating them as a whole command information, the device behavior corresponding to the command information can be determined, thereby more comprehensively considering the actions performed by the user (the voice and the image taken), further improving the accuracy of the device behavior.

[0048] In short, users can control the target device through one or more types of instructions, and the device control methods are more diverse. By using instruction information and corresponding device behavior for model training, the target model obtained can determine the corresponding target device behavior based on the target instruction information issued by the user, without the need to input specific instruction information. Device control can also be achieved for relatively vague instructions, which improves the flexibility and robustness of device control, reduces the user's operating difficulty, and improves the user experience.

[0049] In a possible implementation, when the instruction information includes image instruction information or voice instruction information, the instruction information set and the device behavior set are input into the initial pre-trained model to determine the device behavior that has information similar to the instruction information, which may specifically include steps S1021-S1024.

[0050] S1021, extracting features of the instruction information and device behavior through the initial pre-training model to obtain an instruction information feature identifier corresponding to the instruction information and a device behavior feature identifier corresponding to the device behavior.

[0051] In the embodiment of the present application, when the instruction information is image instruction information or voice instruction information, there is only one device control method at this time, and the user can make corresponding actions or make voice to control the target device to perform corresponding device behaviors.

[0052] The initial pre-trained model can extract features from instruction information to obtain instruction information feature identifiers. Instruction information feature identifiers can be used to identify the corresponding instruction information, and can subsequently replace instruction information to determine whether the information similarity condition is met. The model can also extract features from device behaviors to obtain corresponding device behavior feature identifiers. Each device behavior feature identifier also uniquely identifies a device behavior. Among them, device behaviors can be displayed in the form of text.

[0053] S1022, calculating the similarity between the instruction information feature identifier and the device behavior feature identifier through the initial pre-training model to obtain the similarity between each instruction information feature identifier and each device behavior feature identifier.

[0054] In an embodiment of the present application, the instruction information feature identifier and the device behavior feature identifier are features of the same dimension and are in the same feature space. The initial pre-trained model can calculate the similarity of these two feature identifiers to obtain the similarity.

[0055] For each instruction information feature identifier, calculate the similarity between it and each device behavior feature identifier. Similarly, for each device behavior feature identifier, calculate the similarity between it and each instruction information feature identifier. The higher the similarity, the more the device behavior matches the instruction information, and the more accurate the correspondence between the two.

[0056] S1023: Use the device behavior feature identifier corresponding to the target instruction information feature identifier with the highest similarity as the target device behavior feature identifier corresponding to the target instruction information feature identifier.

[0057] In the embodiment of the present application, each instruction information feature identifier has similarity with each device behavior feature identifier. For one instruction information feature identifier, it has different similarities with different device behavior feature identifiers, some with high similarities and some with low similarities. For example, the similarity between instruction information feature identifier A and device behavior feature identifier 1 is 90%, and the similarity between instruction information feature identifier A and device behavior feature identifier 2 is 40%.

[0058] Specifically, any instruction information feature identifier can be used as a target instruction information feature identifier. For the target instruction information feature identifier, if its similarity with a device behavior feature identifier 1 is the highest, it means that the device behavior feature identifier 1 is the most matching target instruction information feature identifier, and the device behavior feature identifier 1 can be recorded as the target device behavior feature identifier.

[0059] S1024, determining that the target device behavior corresponding to the target device behavior feature identifier and the target instruction information corresponding to the target instruction information feature identifier have information similarity conditions.

[0060] Specifically, the device behavior corresponding to the target device behavior feature identifier can be recorded as the target device behavior, and the instruction information corresponding to the target instruction information feature identifier can be recorded as the target instruction information, so that the target device behavior and the target instruction information meet the information similarity condition. In this way, for each instruction information, the corresponding device behavior can be determined, so that they can be used as sample pairs to train the initial pre-trained model.

[0061] It can be understood that for the same device behavior, it can be used as a sample pseudo-label for multiple command information. For example, for the device behavior "open the curtains", it corresponds to both the voice command information "please open the curtains" and the voice command information "open the curtains". That is, for these two voice command information, the device behavior "open the curtains" is the device behavior with the highest similarity determined by them respectively. In this way, the same device behavior can correspond to more diverse command information, and the training sample size is larger, which can enable the model to better learn the relevant knowledge of the device behavior "open the curtains" and achieve better training effect.

[0062] In this way, when the command information is a single modality, such as voice command information or image command information, determining sample pairs by feature comparison can improve the efficiency and accuracy of the comparison, reduce the amount of calculation, and determine the correct device behavior for each command information.

[0063] In a possible implementation, when the instruction information includes image instruction information and voice instruction information, the instruction information set and the device behavior set are input into the initial pre-trained model to determine the device behavior that has information similar to the instruction information, which may specifically include steps S201-S205.

[0064] S201, extracting features of image command information, voice command information and device behavior through an initial pre-trained model to obtain image command information feature identifiers corresponding to the image command information, voice command information feature identifiers corresponding to the voice command information, and device behavior feature identifiers corresponding to the device behavior.

[0065] In an embodiment of the present application, when the command information includes image command information and voice command information, that is, when it is multimodal input, the collected user images and the voice uttered by the user can be comprehensively considered to determine the corresponding device behavior, thereby generating sample pairs for model training.

[0066] Specifically, the initial pre-trained model can extract features from the input image command information to obtain the corresponding image command information feature identifier, and extract features from the input voice command information to obtain the corresponding voice command information feature identifier. The model can also extract features from device behaviors to obtain corresponding device behavior feature identifiers.

[0067] S202, calculating the similarity between the image command information feature identifier and the device behavior feature identifier through the initial pre-training model to obtain a first similarity between each image command information feature identifier and each device behavior feature identifier, and calculating the similarity between the voice command information feature identifier and the device behavior feature identifier to obtain a second similarity between each voice command information feature identifier and each device behavior feature identifier.

[0068] In the embodiment of the present application, for the collected images including the user and the voice uttered by the user, the corresponding device behavior needs to be determined. For the image command information, the initial pre-trained model can calculate the similarity between the image command information feature identifier and the device behavior feature identifier to obtain the first similarity between each image command information feature identifier and each device behavior feature identifier.

[0069] For the voice command information, the initial pre-training model can calculate the similarity between the voice command information feature identifier and the device behavior feature identifier to obtain a second similarity between each voice command information feature identifier and each device behavior feature identifier.

[0070] S203 , for each target device behavior feature identifier, integrate each corresponding first similarity and each corresponding second similarity to obtain a plurality of integrated similarities.

[0071] In the embodiment of the present application, in order to accurately determine the corresponding image instruction information and voice instruction information for each device behavior, similarity integration processing can be performed to obtain integrated similarity. The integrated similarity corresponds to an integrated feature identifier, and the integrated feature identifier includes an image instruction information feature identifier and a voice instruction information feature identifier.

[0072] The target device behavior feature identifier can be any device behavior feature identifier. For the target device behavior feature identifier M, an image command information feature identifier T1 and a voice command information feature identifier Y1 can be combined as a whole integrated feature identifier Z1. The target device behavior feature identifier M and the image command information feature identifier T1 have a first similarity S1, the target device behavior feature identifier M and the voice command information feature identifier Y1 have a second similarity S2, and the integrated similarity corresponding to the integrated feature identifier Z1 is S1+ S2=88%.

[0073] For the target device behavior feature identifier M, if another image command information feature identifier T2 and another voice command information feature identifier Y3 are combined as a whole integrated feature identifier Z2. The target device behavior feature identifier M and the image command information feature identifier T2 have a first similarity S3, the target device behavior feature identifier M and the voice command information feature identifier Y3 have a second similarity S4, and the integrated similarity corresponding to the integrated feature identifier Z2 is S3+ S4=26%.

[0074] That is, for each target device behavior, each first similarity is integrated with each second similarity, thereby obtaining a plurality of integrated similarities.

[0075] S204: Using the integration feature identifier corresponding to the target device behavior feature identifier with the highest integration similarity as the target integration feature identifier corresponding to the target device behavior feature identifier.

[0076] Specifically, if the integration similarity is higher, it means that the integration feature identifier corresponding to the integration similarity is more matched with the target device behavior feature identifier, then the integration feature identifier corresponding to the highest integration similarity can be determined. For example, the integration feature identifier Z1 corresponds to the target device behavior feature identifier M, then the integration feature identifier Z1 can be recorded as the target integration feature identifier.

[0077] In this way, for each device behavior feature identifier, a corresponding integrated feature identifier can be determined, that is, a group of image command information feature identifiers and voice command information feature identifiers can be determined.

[0078] S205, determining that the target image instruction information and the target voice instruction information corresponding to the target integration feature identifier and the target device behavior corresponding to the target device behavior feature identifier have information similarity conditions.

[0079] Specifically, the target image command information feature identifier in the target integrated feature identifier, the corresponding target image command information, and the target voice command information corresponding to the target voice command information feature identifier can be taken as a whole sample content, and the target device behavior can be taken as the corresponding sample pseudo-label, and the two have information similarity conditions.

[0080] In this way, in a pair of samples, one device behavior corresponds to one image command information and one voice command information, which can improve the accuracy of the training sample pairs. By determining the sample pairs through feature comparison, the amount of calculation can be greatly reduced and the calculation efficiency can be improved.

[0081] In one possible implementation, in step S201, feature extraction is performed on the voice command information through the initial pre-trained model to obtain a voice command information feature identifier corresponding to the voice command information. Specifically, voice-to-text conversion is performed on the voice command information through the initial pre-trained model to obtain text command information; feature extraction is performed on the text command information through the initial pre-trained model to obtain a text command information feature identifier, and the text command information feature identifier is used as the voice command information feature identifier.

[0082] Specifically, when voice command information is collected, it needs to be converted into text command information so that the model can process the text. The initial pre-trained model can convert voice into text, which can be achieved specifically through the voice-to-text conversion module in the initial pre-trained model.

[0083] The voice command information is input into the model to obtain the corresponding text command information, and then the text command information is subjected to feature extraction to obtain the text command information feature identifier corresponding to the text command information, which is also the voice command information feature identifier. In this way, the processing efficiency of the voice command can be improved, so that the model can better understand the voice command and extract more accurate feature identifiers.

[0084] In a possible implementation, the initial pre-training model may be a text-image pre-training model (Constrastive Language-Image Pre-training, CLIP), which may include an image encoder, a first text encoder, and a second text encoder. Figure 2 As shown, it is a schematic diagram of the structure of an initial pre-training model provided in an embodiment of the present application.

[0085] Specifically, the feature extraction process may be to extract features from the image instruction information through an image encoder to obtain an image instruction information feature identifier corresponding to the image instruction information. Since the device behavior is represented by text, the device behavior may be feature extracted through a first text encoder to obtain a device behavior feature identifier corresponding to the device behavior. For the collected voice instruction information, the voice instruction information may be converted into text through an initial pre-trained model to obtain text instruction information, and the text instruction information may be feature extracted using a second text encoder to obtain a voice instruction information feature identifier.

[0086] When the instruction information is in only one mode, such as image instruction information or text instruction information, when performing similarity integration, the similarity of the missing instruction information in another mode may be recorded as 0, and then similarity integration may be performed.

[0087] In this way, feature extraction can be performed more accurately through the image encoder and text encoder to improve the accuracy of feature identification. Different text encoders are used to extract features for device behavior and text instruction information respectively. In this way, during the model fine-tuning process, more targeted adjustments can be made to each text encoder to improve the quality of model training.

[0088] In one possible implementation, since in related technologies, training samples are often uploaded to the cloud for training and analysis during model training, this centralized data processing method poses a risk of privacy leakage. The user's sensitive information may be obtained by unauthorized third parties during data transmission and processing, threatening the user's information security.

[0089] Specifically, the instruction information stored on each terminal device may not be uploaded to the cloud for processing, and model training may be performed only on the respective terminal devices to avoid leakage of user information. The instruction information set includes at least a first instruction information subset stored on a first terminal device and a second instruction information subset stored on a second terminal device, wherein the first instruction information subset includes a plurality of first instruction information and the second instruction information subset includes a plurality of second instruction information.

[0090] That is to say, for the first terminal device, the instruction information stored therein can be recorded as the first instruction information, and the set of the first instruction information can be recorded as the first instruction information subset; for the second terminal device, the instruction information stored therein can be recorded as the second instruction information, and the set of the second instruction information can be recorded as the second instruction information subset. The instruction information set includes at least the first instruction information subset and the second instruction information subset. Of course, it can also include instruction information subsets stored on other terminal devices. The description will not be expanded here, and only the first instruction information subset and the second instruction information subset will be taken as examples for description.

[0091] The first device behavior corresponding to the first instruction information is determined by a first initial pre-trained model located on the first terminal device, and the second device behavior corresponding to the second instruction information is determined by a second initial pre-trained model located on the second terminal device.

[0092] Specifically, the first terminal device has a first initial pre-trained model, and the second terminal device has a second initial pre-trained model. The two models can be the same, can both be initial pre-trained models, or can be different. On the first terminal device, the first initial pre-trained model can perform similarity matching on the first instruction information, thereby determining the first device behavior corresponding to each first instruction information. Similarly, on the second terminal device, the second initial pre-trained model can perform similarity matching on the second instruction information, thereby determining the second device behavior corresponding to each second instruction information.

[0093] It can be understood that, although the first instruction information subset and the second instruction information subset are different, the device behaviors stored on each terminal device are the same, that is, the device behavior sets are consistent.

[0094] In step S103, the instruction information is used as sample content, and the device behavior corresponding to the instruction information is used as a sample pseudo label, and supervised training is performed on the initial pre-trained model to obtain a target model, which may specifically include steps S1031-S1033.

[0095] S1031, obtaining a first gradient from a first terminal device and a second gradient from a second terminal device.

[0096] In an embodiment of the present application, if each terminal device only uses a subset of instruction information stored in itself for training, due to the small amount of data in the instruction information subset, the model learns less knowledge about which device behavior the instruction information corresponds to, and the quality of each trained model is poor, and it is not possible to determine the corresponding device behavior for the instruction information input by the user. Therefore, it is necessary to summarize the relevant knowledge learned by each terminal device. The first gradient is obtained by supervised training of the first initial pre-trained model using the first instruction information and the first device behavior. That is, the first instruction information can be used as the sample content of the first initial pre-trained model, and the first device behavior can be used as the corresponding sample pseudo-label to perform supervised training and obtain a loss function. According to the loss function, the first gradient can be calculated so that the model can be fine-tuned using the first gradient.

[0097] Similarly, the second gradient is obtained by supervised training the second initial pre-trained model using the second instruction information and the second device behavior. The second instruction information is used as the sample content of the second initial pre-trained model, and the second device behavior corresponding to the second instruction information is used as the sample pseudo label. The loss function is obtained through model training, and then the second gradient is determined.

[0098] In this way, each terminal device can use its own stored training samples for training to obtain its own gradient.

[0099] S1032, determining an integrated gradient according to the first gradient and the second gradient.

[0100] In an embodiment of the present application, in order to make the trained target model more accurate, the gradients of each terminal device can be summarized, and each terminal device can upload its own gradient to the same server for integration processing to obtain an integrated gradient, which is recorded as an integrated gradient.

[0101] Specifically, the integrated gradient is determined according to the first gradient and the second gradient. Specifically, the first gradient and the second gradient are weighted averaged to obtain the integrated gradient. Of course, other methods are also possible, which will not be listed here one by one.

[0102] S1033, sending the integrated gradient to the first terminal device and the second terminal device, so that the first terminal device uses the integrated gradient to fine-tune the first initial pre-trained model to obtain the first target model, and the second terminal device uses the integrated gradient to fine-tune the second initial pre-trained model to obtain the second target model.

[0103] Specifically, the integrated gradient can be sent to each terminal device so that each terminal device can use the integrated gradient to adjust its own model to obtain a trained target model, that is, a first target model is obtained on a first terminal device, and a second target model is obtained on a second terminal device. The trained target models on each terminal device are the same, that is, the model parameters of the first target model and the second target model are completely consistent.

[0104] In this way, since the integrated gradient integrates the gradients calculated by each terminal device, that is, it is obtained by integrating the training samples of each terminal device, the integrated gradient includes all the training samples on all terminal devices, and learns the knowledge of the correlation between each instruction information and the corresponding device behavior. In other words, the model is trained based on all the instruction information, and the quality of the target model is better.

[0105] refer to Figure 3 As shown, a schematic diagram of a model training process provided in an embodiment of the present application, on a first terminal device, a first instruction information subset and a device behavior set stored therein can be input into a first initial pre-trained model, and a first gradient is output; on a second terminal device, a second instruction information subset and a device behavior set stored therein can be input into a second initial pre-trained model, and a second gradient is output; the first gradient and the second gradient are uploaded to a server, the server integrates them, and transmits the integrated gradient back to the first initial pre-trained model and the second initial pre-trained model for parameter adjustment.

[0106] This application uses federated learning technology to perform model training and parameter updates locally on the terminal device, and summarizes the relevant knowledge learned by each terminal device by uploading gradients to the server. This can not only ensure that a large amount of instruction information is used for training to improve the model training effect, but also avoid the leakage of user information involved in the instruction information by uploading gradients instead of directly uploading instruction information, and avoid the user's privacy data from being uploaded to the cloud, thereby greatly improving the security of user information.

[0107] Based on the above model determination method, the present application embodiment also provides a model determination device, referring to Figure 4 As shown, it is a structural block diagram of a model determination device provided in an embodiment of the present application, and the device may include:

[0108] The acquisition unit 201 is used to acquire a command information set and a device behavior set; the command information set includes multiple command information, the command information includes at least one of image command information and voice command information, and the device behavior set includes multiple device behaviors performed by the target device;

[0109] A determination unit 202, configured to input the instruction information set and the device behavior set into an initial pre-training model, and determine the device behavior having information similarity conditions with the instruction information;

[0110] The training unit 203 is used to use the instruction information as sample content and the device behavior corresponding to the instruction information as sample pseudo-label to perform supervised training on the initial pre-trained model to obtain a target model; the target model is used to determine the corresponding target device behavior based on the input target instruction information.

[0111] Optionally, when the instruction information includes image instruction information or voice instruction information, the determining unit is used to:

[0112] Extracting features of the instruction information and the device behavior through the initial pre-training model to obtain an instruction information feature identifier corresponding to the instruction information and an device behavior feature identifier corresponding to the device behavior;

[0113] Calculating the similarity between the instruction information feature identifier and the device behavior feature identifier by using the initial pre-training model to obtain the similarity between each instruction information feature identifier and each device behavior feature identifier;

[0114] The device behavior feature identifier corresponding to the target instruction information feature identifier with the highest similarity is used as the target device behavior feature identifier corresponding to the target instruction information feature identifier;

[0115] It is determined that the target device behavior corresponding to the target device behavior feature identifier and the target instruction information corresponding to the target instruction information feature identifier have information similarity conditions.

[0116] Optionally, when the instruction information includes image instruction information and voice instruction information, the determining unit is used to:

[0117] Extracting features of the image command information, the voice command information and the device behavior through the initial pre-training model to obtain an image command information feature identifier corresponding to the image command information, a voice command information feature identifier corresponding to the voice command information, and a device behavior feature identifier corresponding to the device behavior;

[0118] Performing similarity calculation on the image instruction information feature identifier and the device behavior feature identifier through the initial pre-training model to obtain a first similarity between each of the image instruction information feature identifier and each of the device behavior feature identifiers, and performing similarity calculation on the voice instruction information feature identifier and the device behavior feature identifier to obtain a second similarity between each of the voice instruction information feature identifier and each of the device behavior feature identifiers;

[0119] For each target device behavior feature identifier, each corresponding first similarity and each second similarity are integrated to obtain a plurality of integrated similarities; the integrated similarities correspond to integrated feature identifiers, and the integrated feature identifiers include one of the image instruction information feature identifiers and one of the voice instruction information feature identifiers;

[0120] The integration feature identifier corresponding to the highest integration similarity of the target device behavior feature identifier is used as the target integration feature identifier corresponding to the target device behavior feature identifier;

[0121] It is determined that the target image instruction information and the target voice instruction information corresponding to the target integration feature identifier have information similarity conditions with the target device behavior corresponding to the target device behavior feature identifier.

[0122] Optionally, the determining unit is configured to:

[0123] Performing voice-to-text conversion on the voice instruction information through the initial pre-trained model to obtain text instruction information;

[0124] The text instruction information is subjected to feature extraction through the initial pre-training model to obtain a text instruction information feature identifier, and the text instruction information feature identifier is used as the voice instruction information feature identifier.

[0125] Optionally, the initial pre-trained model includes an image encoder, a first text encoder, and a second text encoder, and the determining unit is used to:

[0126] Extracting features of the image instruction information by the image encoder to obtain an image instruction information feature identifier corresponding to the image instruction information;

[0127] Extracting features of the device behavior by using the first text encoder to obtain a device behavior feature identifier corresponding to the device behavior;

[0128] Performing voice-to-text conversion on the voice instruction information through the initial pre-trained model to obtain text instruction information;

[0129] The second text encoder is used to perform feature extraction on the text instruction information to obtain a feature identifier of the voice instruction information.

[0130] Optionally, the instruction information set includes at least a first instruction information subset stored on the first terminal device, and a second instruction information subset stored on the second terminal device, the first instruction information subset includes a plurality of first instruction information, and the second instruction information subset includes a plurality of second instruction information;

[0131] The first device behavior corresponding to the first instruction information is determined by a first initial pre-trained model located on the first terminal device, and the second device behavior corresponding to the second instruction information is determined by a second initial pre-trained model located on the second terminal device;

[0132] The training unit is used to:

[0133] Acquire a first gradient from the first terminal device and a second gradient from the second terminal device; the first gradient is obtained by supervised training the first initial pre-trained model using the first instruction information and the first device behavior, and the second gradient is obtained by supervised training the second initial pre-trained model using the second instruction information and the second device behavior;

[0134] determining an integrated gradient according to the first gradient and the second gradient;

[0135] The integrated gradient is sent to the first terminal device and the second terminal device, so that the first terminal device uses the integrated gradient to fine-tune the first initial pre-trained model to obtain a first target model, and the second terminal device uses the integrated gradient to fine-tune the second initial pre-trained model to obtain a second target model.

[0136] Optionally, the training unit is used to:

[0137] A weighted average is performed on the first gradient and the second gradient to obtain the integrated gradient.

[0138] In another aspect, the present application provides a computer device, referring to Figure 5 , which is a structural diagram of a computer device provided in an embodiment of the present application, wherein the computer device includes a processor 310 and a memory 320:

[0139] The memory 320 is used to store program codes and transmit the program codes to the processor 310;

[0140] The processor 310 is used to execute the method provided in the above embodiment according to the instructions in the program code.

[0141] The computer device may include a terminal device or a server, and the aforementioned apparatus may be configured in the computer device.

[0142] On the other hand, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, and the computer program is used to execute the method provided by the above embodiment.

[0143] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by program instruction hardware, and the above program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the above storage medium can be at least one of the following media: read-only memory (English: Read-only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0144] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0145] The above is only a preferred implementation of the present application. Although the present application has been disclosed as a preferred embodiment, it is not intended to limit the present application. Any technician familiar with the art can use the above disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present application without departing from the scope of the technical solution of the present application, or modify it into an equivalent embodiment of equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application still falls within the scope of protection of the technical solution of the present application.

Claims

1. A model determination method, characterized in that: The method comprises: Acquire a command information set and a device behavior set; the command information set includes a plurality of command information, the command information includes at least one of image command information and voice command information, and the device behavior set includes a plurality of device behaviors performed by the target device; Inputting the instruction information set and the device behavior set into an initial pre-training model to determine the device behavior having information similar to the instruction information; The instruction information is used as sample content, and the device behavior corresponding to the instruction information is used as a sample pseudo label, and the initial pre-trained model is supervised and trained to obtain a target model; the target model is used to determine the corresponding target device behavior according to the input target instruction information; When the instruction information includes image instruction information and voice instruction information, inputting the instruction information set and the device behavior set into an initial pre-training model, and determining the device behavior having information similarity conditions with the instruction information, including: Extracting features of the image command information, the voice command information and the device behavior through the initial pre-training model to obtain an image command information feature identifier corresponding to the image command information, a voice command information feature identifier corresponding to the voice command information, and a device behavior feature identifier corresponding to the device behavior; Performing similarity calculation on the image instruction information feature identifier and the device behavior feature identifier through the initial pre-training model to obtain a first similarity between each of the image instruction information feature identifier and each of the device behavior feature identifiers, and performing similarity calculation on the voice instruction information feature identifier and the device behavior feature identifier to obtain a second similarity between each of the voice instruction information feature identifier and each of the device behavior feature identifiers; For each target device behavior feature identifier, each corresponding first similarity and each second similarity are integrated to obtain a plurality of integrated similarities; the integrated similarities correspond to integrated feature identifiers, and the integrated feature identifiers include one of the image instruction information feature identifiers and one of the voice instruction information feature identifiers; The integration feature identifier corresponding to the highest integration similarity of the target device behavior feature identifier is used as the target integration feature identifier corresponding to the target device behavior feature identifier; It is determined that the target image instruction information and the target voice instruction information corresponding to the target integration feature identifier have information similarity conditions with the target device behavior corresponding to the target device behavior feature identifier.

2. The method according to claim 1, characterized in that When the instruction information includes image instruction information or voice instruction information, inputting the instruction information set and the device behavior set into an initial pre-training model, and determining the device behavior having information similarity conditions with the instruction information, including: Extracting features of the instruction information and the device behavior through the initial pre-training model to obtain an instruction information feature identifier corresponding to the instruction information and an device behavior feature identifier corresponding to the device behavior; Calculating the similarity between the instruction information feature identifier and the device behavior feature identifier by using the initial pre-training model to obtain the similarity between each instruction information feature identifier and each device behavior feature identifier; The device behavior feature identifier corresponding to the target instruction information feature identifier with the highest similarity is used as the target device behavior feature identifier corresponding to the target instruction information feature identifier; It is determined that the target device behavior corresponding to the target device behavior feature identifier and the target instruction information corresponding to the target instruction information feature identifier have information similarity conditions.

3. The method according to claim 1, characterized in that Extracting features of the voice command information by using the initial pre-trained model to obtain a voice command information feature identifier corresponding to the voice command information includes: Performing voice-to-text conversion on the voice instruction information through the initial pre-trained model to obtain text instruction information; The text instruction information is subjected to feature extraction through the initial pre-training model to obtain a text instruction information feature identifier, and the text instruction information feature identifier is used as the voice instruction information feature identifier.

4. The method according to claim 3, characterized in that: The initial pre-trained model includes an image encoder, a first text encoder, and a second text encoder. The image instruction information, the voice instruction information, and the device behavior are subjected to feature extraction by the initial pre-trained model to obtain an image instruction information feature identifier corresponding to the image instruction information, a voice instruction information feature identifier corresponding to the voice instruction information, and a device behavior feature identifier corresponding to the device behavior, including: Extracting features of the image instruction information by the image encoder to obtain an image instruction information feature identifier corresponding to the image instruction information; Extracting features of the device behavior by using the first text encoder to obtain a device behavior feature identifier corresponding to the device behavior; Performing voice-to-text conversion on the voice instruction information by using the initial pre-trained model to obtain text instruction information; The second text encoder is used to perform feature extraction on the text instruction information to obtain a feature identifier of the voice instruction information.

5. The method according to claim 1, characterized in that The instruction information set at least includes a first instruction information subset stored on the first terminal device, and a second instruction information subset stored on the second terminal device, the first instruction information subset includes a plurality of first instruction information, and the second instruction information subset includes a plurality of second instruction information; The first device behavior corresponding to the first instruction information is determined by a first initial pre-trained model located on the first terminal device, and the second device behavior corresponding to the second instruction information is determined by a second initial pre-trained model located on the second terminal device; The instruction information is used as sample content, and the device behavior corresponding to the instruction information is used as a sample pseudo label, and supervised training is performed on the initial pre-trained model to obtain a target model, including: Acquire a first gradient from the first terminal device and a second gradient from the second terminal device; the first gradient is obtained by supervised training the first initial pre-trained model using the first instruction information and the first device behavior, and the second gradient is obtained by supervised training the second initial pre-trained model using the second instruction information and the second device behavior; determining an integrated gradient according to the first gradient and the second gradient; The integrated gradient is sent to the first terminal device and the second terminal device, so that the first terminal device uses the integrated gradient to fine-tune the first initial pre-trained model to obtain a first target model, and the second terminal device uses the integrated gradient to fine-tune the second initial pre-trained model to obtain a second target model.

6. The method according to claim 5, characterized in that Determining an integrated gradient according to the first gradient and the second gradient includes: A weighted average is performed on the first gradient and the second gradient to obtain the integrated gradient.

7. A model determination device, characterized in that: The device comprises: An acquisition unit, configured to acquire a command information set and a device behavior set; the command information set includes a plurality of command information, the command information includes at least one of image command information and voice command information, and the device behavior set includes a plurality of device behaviors performed by the target device; A determination unit, configured to input the instruction information set and the device behavior set into an initial pre-training model, and determine the device behavior having information similarity conditions to the instruction information; A training unit, configured to use the instruction information as sample content and the device behavior corresponding to the instruction information as sample pseudo labels, to perform supervised training on the initial pre-trained model to obtain a target model; the target model is configured to determine the corresponding target device behavior according to the input target instruction information; When the instruction information includes image instruction information and voice instruction information, the determining unit is used to: Extracting features of the image command information, the voice command information and the device behavior through the initial pre-training model to obtain an image command information feature identifier corresponding to the image command information, a voice command information feature identifier corresponding to the voice command information, and a device behavior feature identifier corresponding to the device behavior; Performing similarity calculation on the image instruction information feature identifier and the device behavior feature identifier through the initial pre-training model to obtain a first similarity between each of the image instruction information feature identifier and each of the device behavior feature identifiers, and performing similarity calculation on the voice instruction information feature identifier and the device behavior feature identifier to obtain a second similarity between each of the voice instruction information feature identifier and each of the device behavior feature identifiers; For each target device behavior feature identifier, each corresponding first similarity and each second similarity are integrated to obtain a plurality of integrated similarities; the integrated similarities correspond to integrated feature identifiers, and the integrated feature identifiers include one of the image instruction information feature identifiers and one of the voice instruction information feature identifiers; The integration feature identifier corresponding to the highest integration similarity of the target device behavior feature identifier is used as the target integration feature identifier corresponding to the target device behavior feature identifier; It is determined that the target image instruction information and the target voice instruction information corresponding to the target integration feature identifier have information similarity conditions with the target device behavior corresponding to the target device behavior feature identifier.

8. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method according to any one of claims 1 to 6 according to the instructions in the program code.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 6.

Images