Intention recognition method, device and electronic device applied to message events

Abstract

The embodiments of the present disclosure disclose an intent recognition method, device and electronic device applied to message events. A specific implementation of the method includes: obtaining a target message event; in response to the message type being a mixed message type, activating a message content encoder set corresponding to the message type in the message content encoding model; in response to successful activation, encoding the message content through the message content encoder set corresponding to the message type; generating a global message content feature according to a message content feature set; extracting associated features from the message content features in the message content feature set according to the feature position of the message content feature in the global message content feature; generating a message event intent for the target message event according to the associated content feature set and the global message content feature; generating intent suggestion information for the message event intent. This implementation effectively realizes intent recognition for multimodal data.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, and electronic device for identifying intent applied to message events. Background Art

[0002] Intent recognition is an important research direction in natural language processing. It aims to understand the intention of users in the process of interaction with the system. It is applied in many fields such as chatbots and voice assistants, and is one of the cornerstone technologies for building intelligent dialogue. At present, the method usually used for intent recognition is to perform intent recognition based on unimodal text modal data.

[0003] However, when the above method is adopted, the following technical problems often occur:

[0004] First, in response to the need for intent recognition of multimodal data, the intent recognition method based on single-modal data cannot effectively extract features, resulting in the inability to effectively recognize intent for multimodal data;

[0005] Second, the corresponding features of multimodal data cannot be effectively associated from the feature perspective, making it impossible to effectively combine multimodal data for comprehensive intent recognition, which in turn leads to inaccurate intent recognition results.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose an intent recognition method, apparatus, and electronic device applied to message events to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide a method for intent recognition applied to message events, the method comprising: acquiring a target message event, wherein the target message event is a message event acquired in real time and to be subjected to intent recognition, the target message event comprising: message content and message type, the message type comprising: audio message type, image message type, text message type and mixed message type; in response to the message type being a mixed message type, activating a message content encoder set corresponding to the message type in a message content encoding model; in response to successful activation, encoding the message content through the message content encoder set corresponding to the message type to obtain a message content feature set; generating a global message content feature based on the message content feature set; extracting associated features from the message content features in the message content feature set based on the feature position of the message content feature in the global message content feature to generate an associated content feature set; generating a message event intent for the target message event based on the associated content feature set and the global message content feature; generating intent suggestion information for the message event intent.

[0010] In a second aspect, some embodiments of the present disclosure provide an intent recognition device applied to message events, the device comprising: an acquisition unit, configured to acquire a target message event, wherein the target message event is a message event acquired in real time and to be subjected to intent recognition, the target message event comprising: message content and message type, the message type comprising: audio message type, image message type, text message type and mixed message type; an activation unit, configured to, in response to the message type being a mixed message type, activate a message content encoder set in a message content encoding model corresponding to the message type; a message content encoding unit, configured to, in response to successful activation, activate a message content encoder set corresponding to the message type through the message content encoding model corresponding to the message type; A message content encoder set is used to encode the message content to obtain a message content feature set; a first generating unit is configured to generate a global message content feature based on the message content feature set; an associated feature extraction unit is configured to extract associated features from the message content features in the message content feature set based on the feature positions of the message content features in the global message content features to generate an associated content feature set; a second generating unit is configured to generate a message event intent for the target message event based on the associated content feature set and the global message content features; and a third generating unit is configured to generate intent suggestion information for the message event intent.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0013] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the intent recognition method applied to message events of some embodiments of the present disclosure, the intent recognition for multimodal data is effectively realized. Specifically, the reason for the inability to effectively recognize the intent is that the intent recognition method based on single-modal data cannot effectively extract features for the intent recognition requirements of multimodal data, resulting in the inability to effectively recognize the intent for multimodal data. Based on this, the intent recognition method applied to message events of some embodiments of the present disclosure first obtains a target message event, wherein the target message event is a message event to be recognized for intent acquired in real time, and the target message event includes: message content and message type, and the message type includes: audio message type, image message type, text message type and mixed message type. Secondly, in response to the above message type being a mixed message type, the message content encoder set corresponding to the above message type in the message content encoding model is activated. Then, in response to the successful activation, the message content is encoded by the message content encoder set corresponding to the above message type to obtain a message content feature set. In practice, when it is a mixed message type, the target message event is characterized by at least two of audio messages, image messages and text messages, so it is necessary to activate the corresponding content encoding model to perform feature extraction. Further, based on the above-mentioned message content feature set, a global message content feature is generated to embed the features corresponding to different types of messages into the same feature space. In addition, based on the feature position of the message content feature in the above-mentioned global message content feature, the message content features in the above-mentioned message content feature set are extracted for associated features to generate an associated content feature set. In this way, further feature extraction of associated features is performed from the position perspective. In addition, based on the above-mentioned associated content feature set and the above-mentioned global message content feature, a message event intent for the above-mentioned target message event is generated. Finally, intent suggestion information for the above-mentioned message event intent is generated. In this way, intent recognition for multimodal data can be effectively achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flowchart of some embodiments of the method for identifying intent applied to message events according to the present disclosure;

[0016] Figure 2 It is a schematic diagram of the scenario in which the client sends a target message event;

[0017] Figure 3 is a schematic structural diagram of some embodiments of an apparatus for identifying intent applied to message events according to the present disclosure;

[0018] Figure 4 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] refer to Figure 1 , shows a process 100 of some embodiments of the method for identifying intent applied to message events according to the present disclosure. The method for identifying intent applied to message events comprises the following steps:

[0026] Step 101, obtaining a target message event.

[0027] In some embodiments, the execution subject (e.g., computing device) of the intent recognition method applied to message events can obtain the above target message events through wired connection or wireless connection. The above target message events are message events that are obtained in real time and to be recognized for intent. In practice, the target message events can represent real-time messages initiated by users on the client. For example, Figure 2 The shown schematic diagram is a scenario in which a client sends a target message event, wherein the target message event 201 may be “picture + the weather of the place above the day after tomorrow”, wherein, therefore, the message type of the target message event may be a mixed message type, i.e., composed of a text message type and an image message type.

[0028] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future.

[0029] It should be noted that the above-mentioned computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here. It should be understood that the number of computing devices can be any number according to the implementation needs.

[0030] In some optional implementations of some embodiments, the execution subject obtaining the target message event may include the following steps:

[0031] The first step is to establish a communication link with the client in response to monitoring a communication request initiated by the client.

[0032] In practice, the execution subject may set a request forwarding server between the client and the execution subject. Specifically, the request forwarding server may be used to monitor and forward requests, and manage communication links, such as creating communication links. For example, the request forwarding server may use an Nginx server. The communication link may be an end-to-end link between the execution subject and the client. In practice, the request forwarding server may create a Socket connection between the client and the execution subject as the communication link. Specifically, the execution subject may monitor whether the client initiates a communication request by means of port monitoring.

[0033] The second step is to perform link security detection on the above communication link.

[0034] In practice, the execution subject may analyze the network traffic of the communication link to implement link security detection. For example, the link security detection may be performed by detecting whether the network traffic of the communication link per unit time is greater than the warning threshold.

[0035] In the third step, in response to the communication link passing the link security detection, a public key and a private key are generated.

[0036] The public key is used for data encryption, and the private key is used for data decryption. In practice, the execution subject may generate the public key and the private key through an asymmetric encryption algorithm. Specifically, the asymmetric encryption algorithm may be an RSA encryption algorithm.

[0037] The fourth step is to distribute the public key to the client through the communication link.

[0038] The fifth step is to, in response to receiving the encrypted message sent by the client through the communication link, decrypt the encrypted message using the private key to obtain the target message event.

[0039] The encrypted message is obtained by the client through public key encryption.

[0040] Step 102: In response to the message type being a mixed message type, a message content encoder set corresponding to the message type in the message content encoding model is activated.

[0041] In some embodiments, the above-mentioned execution subject may activate a message content encoder set corresponding to the message type in the message content encoding model in response to the message type being a mixed message type. Among them, the message content encoder is a model for encoding message events. In practice, different message contents can be set for encoding models for different message types. For example, for an audio message type, a message content encoder for audio message encoding can be set. For an image message type, a message content encoder for image message encoding can be set. For a text message type, a message content encoder for text message encoding can be set. In practice, when the message type is a mixed message type, it represents a message event that contains at least two message types, so it is necessary to activate multiple message content encoders, that is, activate the above-mentioned message content encoder set. The message content encoding model can be a model for encoding message events of different message types.

[0042] In some optional implementations of some embodiments, in response to the message type being a mixed message type, the execution subject activating a message content encoder set corresponding to the message type in a message content encoding model may include the following steps:

[0043] The first step is to decompose the above message types to generate a set of basic message types.

[0044] The basic message types include at least one of an audio message type, an image message type and a text message type.

[0045] In practice, for example, message types can be stored in the form of a deduplication list, because the basic message types corresponding to the target message event may be repeated. For example, the target message event contains two images of the image message type and one text of the text message type. Therefore, the deduplication list can only store the types of message types that exist in the target message event, and will not cause a large number of repeated message types in the list due to the target message event being too long. Therefore, the above-mentioned execution subject can read the above-mentioned deduplication list to obtain the above-mentioned basic message type set.

[0046] The second step is to generate an activation vector based on the above basic message type set.

[0047] In practice, the vector length of the activation vector is 3. The vector values ​​in the activation vector correspond to the audio message type, the image message type, and the text message type, respectively. In the initial state of the activation vector, the vector values ​​in the activation vector are all 0. The above-mentioned execution subject can update the vector value in the activation vector corresponding to the basic message type in the basic message type set, and specifically can update it to 1 to obtain the above-mentioned activation vector.

[0048] The third step is to load the message content encoder set corresponding to the message type in the message content encoding model according to the activation vector.

[0049] In practice, the execution subject may load the model file of the corresponding message content encoder according to the vector value of 1 in the activation vector, so as to achieve the purpose of activating the message content encoder.

[0050] Step 103, in response to successful activation, the message content is encoded by a message content encoder set corresponding to the message type to obtain a message content feature set.

[0051] In some embodiments, in response to successful activation, the execution subject may encode the message content by using a message content encoder set corresponding to the message type to obtain a message content feature set.

[0052] As an example, the message content encoder set may include: message content encoder A, message content encoder B, and message content encoder C. Among them, message content encoder A is used to encode the message content of the text message type. Message content encoder B is used to encode the message content of the image message type. Message content encoder C is used to encode the message content of the audio message type. Therefore, first, the above-mentioned execution subject may encode the message content of the corresponding text message type in the target message event through the message content encoder A to obtain the message content feature A. The above-mentioned execution subject may encode the message content of the corresponding image message type in the target message event through the message content encoder B to obtain the message content feature B. The above-mentioned execution subject may encode the message content of the corresponding audio message type in the target message event through the message content encoder C to obtain the message content feature C. Then, the message content feature A, the message content feature B, and the message content feature C are used as the above-mentioned message content feature set.

[0053] In some optional implementations of some embodiments, the execution subject encodes the message content by using a message content encoder set corresponding to the message type to obtain a message content feature set, which may include the following steps:

[0054] The first step is to segment the message content according to the basic message type set to generate a target information set.

[0055] The target information in the target information set includes: the segmented message content and content positioning information, and the content positioning information represents the relative position of the segmented message content in the message content. In practice, for each basic message type in the basic message type set, the execution subject can scan the message content in the target message event that matches the basic message type to obtain the segmented message content included in the target information.

[0056] As an example, Figure 2 Taking the target message event 201 as an example, the target message event includes an image and text, so the target information set may include: target information A and target information B. Target information A includes: segmented message content A and content positioning information A. Target information B includes: segmented message content B and content positioning information B. The segmented message content A corresponds to the target message event including an image. The content positioning information A indicates the position of the segmented message content A in the target message event. The segmented message content B corresponds to the target message event including text. The content positioning information B indicates the position of the segmented message content B in the target message event.

[0057] In the second step, for each target information in the target information group set, the following processing steps are performed:

[0058] The first sub-step is, in response to the basic message type corresponding to the above target information being an audio message type, performing a short-time Fourier transform on the segmented message content included in the above target information to generate an audio spectrum diagram.

[0059] In practice, the above-mentioned execution entity can perform short-time Fourier Transform (STFT) on the segmented message content included in the target information to generate an audio spectrum diagram.

[0060] The second sub-step is to perform feature compression on the audio spectrum graph through a feature encoder included in the first message content encoder to generate compressed features.

[0061] The first message content encoder is a message content encoder corresponding to the audio message type in the message content encoder set. The feature encoder includes: 2 2D convolutional layers connected in series.

[0062] The third sub-step is to perform feature contrast learning on the compressed features through the feature contrast learner included in the first message content encoder to generate message content features in the message content feature set corresponding to the target information content.

[0063] Among them, the above-mentioned feature contrast learner includes: a linear mapping layer and at least two feature capturers, and the feature capturer includes: a first normalization layer, a multi-head self-attention mechanism layer, a second normalization layer, a feature convolution block, a third normalization layer and a feedforward network. Specifically, the multi-head self-attention mechanism layer includes: a LayerNorm layer, a multi-head attention mechanism layer based on relative position encoding, and a Dropout layer. The feature convolution block includes: a LayerNorm layer, a Pointwise convolution layer, a GIU activation layer, a 1DDepthwise convolution layer, a BatchNorm layer, a Swish activation layer, a Pointwise convolution layer and a Dropout layer. The feedforward network includes: a LayerNorm layer, a LinearLayer layer, a Switch activation layer, a Dropout layer, a LinearLayer layer and a Dropout layer.

[0064] The fourth sub-step is, in response to the basic message type corresponding to the target information being an image message type, segmenting the segmented message content included in the target information into image blocks to obtain an image block set.

[0065] In practice, the execution subject may perform uniform image block segmentation on the segmented message content included in the target information to obtain the image block set.

[0066] The fifth sub-step is to determine the image block information degree of each image block in the above image block set.

[0067] The image block information degree represents the image content in the image block. For example, when the image block is part of the background image, the corresponding image block information degree is low. When the image block is part of the main body of the picture, the corresponding image block information degree is high.

[0068] In practice, the above-mentioned execution subject can extract image features from image blocks through a feature pyramid network composed of three layers connected in series to obtain image block features, and then input the image block features into a classifier to generate image block information corresponding to the image block.

[0069] In a sixth sub-step, the image block set is binary grouped according to the image block information degrees corresponding to the image blocks to obtain a first image block set and a second image block set.

[0070] The average image block information degree corresponding to the second image block set is less than the average image block information degree corresponding to the first image block set. In practice, the execution subject may set an image block information degree threshold and group the image block sets by binary search to obtain the first image block set and the second image block set.

[0071] The seventh sub-step is to perform image feature encoding on the first image block set and the second image block set by a second message content encoder to generate message content features in the message content feature set corresponding to the target information content.

[0072] The second message content encoder is a message content encoder corresponding to the image message type in the message content encoder set, the second message content encoder performs random image masking on the first image block in the first image block set, and the second message content encoder does not perform image masking on the second image block in the second image block set. The second message content encoder uses BEIT (Pre-Training of Image Transformers) as the basic model. Specifically, based on the BEIT input, a classifier is added, and when the image block is an image block in the second image block set, no image masking is performed.

[0073] The eighth sub-step, in response to the basic message type corresponding to the above-mentioned target information being a text message type, uses a third message content encoder to perform text feature encoding on the segmented message content included in the above-mentioned target information to generate a message content feature in the above-mentioned message content feature set that corresponds to the above-mentioned target information content.

[0074] In practice, the third message content encoder is a message content encoder corresponding to the text message type in the message content encoder set. Specifically, the third message content encoder may adopt a BERT model.

[0075] Step 104: Generate global message content features based on the message content feature set.

[0076] In some embodiments, the execution subject may generate a global message content feature according to a message content feature set. In practice, the execution subject may concatenate message content features in the message content feature set to generate a global message content feature.

[0077] In some optional implementations of some embodiments, the execution subject may generate the global message content feature according to the message content feature set, which may include the following steps:

[0078] In the first step, feature conversion is performed on each message content feature in the message content feature set to generate a converted message content feature, thereby obtaining a converted message content feature set.

[0079] The converted message content features in the converted message content feature set are located in the same feature space. In practice, the execution subject may perform feature conversion on the message content features in the form of dimensional scaling to generate converted message content features.

[0080] In the second step, according to the content positioning information corresponding to the converted message content feature, feature projection is performed on the converted message content features in the converted message content feature set to generate the global message content feature.

[0081] In practice, the execution subject may use the position corresponding to the content location information as the projection position, embed the corresponding converted message content features, and obtain the global message content features.

[0082] Step 105 , extracting associated features from the message content features in the message content feature set according to the feature positions of the message content features in the global message content features, so as to generate an associated content feature set.

[0083] In some embodiments, the execution subject may extract associated features from the message content features in the message content feature set according to feature positions of the message content features in the global message content features to generate an associated content feature set.

[0084] In some optional implementations of some embodiments, the execution subject extracts associated features from the message content features in the message content feature set according to the feature position of the message content features in the global message content features to generate an associated content feature set, which may include the following steps:

[0085] By using the associated feature extraction model, associated features are extracted from the message content features at adjacent feature positions in the message content feature set to generate an associated content feature set.

[0086] Among them, the associated feature extraction model includes: an associated feature extraction sub-model set, parameters are shared between each associated feature extraction sub-model in the above associated feature extraction sub-model set, and the associated feature extraction sub-models in the above associated feature extraction sub-model set are activated accordingly according to the number of associations. In practice, the associated feature extraction sub-model can adopt a convolutional network model. Specifically, the associated content feature set can perform feature association extraction on at least two message content features of different message types at adjacent positions. For example, when the number of message content features of different message types at adjacent positions is 3, the associated feature extraction sub-model set can include: 3 associated feature extraction sub-models to form a three-way convolutional network model to perform associated feature extraction. When the number of message content features of different message types at adjacent positions is 2, the associated feature extraction sub-model set can include: 2 associated feature extraction sub-models to form a two-way convolutional network model to perform associated feature extraction. Through the associated feature extraction method, one is to further compress the features, and the other is to associate the features of adjacent positions from the perspective of feature positions.

[0087] The content of "in some optional implementation methods of some embodiments" in step 103, step 104, and step 105 is an inventive point of the present disclosure, which solves the second technical problem mentioned in the turtle essence technology, namely, "the corresponding features of the multimodal data cannot be effectively associated from the feature perspective, making it impossible to effectively combine the multimodal data for comprehensive intent recognition, which leads to the intention recognition result being not accurate enough". Specifically, first, for multimodal data, the present disclosure encodes the image message type, audio message type and text message type as the basic types. Specifically, for the audio message type, the first step is to compress the feature dimension through the feature encoder. The second step is to capture the features through feature comparison learning to obtain the message content features. For the image message type, the first step is to segment the image blocks to split the image into a "token"-like format. The second step is to determine the image block information degree of each image block in the above-mentioned image block set. The third step is to perform binary grouping on the above-mentioned image block set according to the image block information degree corresponding to the image block, and obtain a first image block set and a second image block set, wherein the mean image block information degree corresponding to the above-mentioned second image block set is less than the mean image block information degree corresponding to the above-mentioned first image block set. The fourth step is to perform image feature encoding on the above-mentioned first image block set and the second image block set through the second message content encoder to generate a message content feature in the above-mentioned message content feature set corresponding to the above-mentioned target information content, wherein the above-mentioned second message content encoder is a message content encoder corresponding to the image message type in the above-mentioned message content encoder set, and the above-mentioned second message content encoder performs random image masking on the first image block in the above-mentioned first image block set, and the above-mentioned second message content encoder does not perform image masking on the second image block in the above-mentioned second image block set. Compared with the conventional form of random masking, the present disclosure adds the determination of image block information degree and the grouping based on image block information degree, so that in the image masking stage, the image blocks with low image block information degree are not masked, so as to reduce the model's attention to the image blocks with low image block information degree, in other words, increase the attention to the image blocks with high image block information degree. Next, the above feature extraction is only feature extraction from the perspective of message type. Based on this, the present disclosure projects the message content features to the same feature space in the form of feature projection, and further extracts features between positional relationships based on relative positional relationships. In this way, the corresponding features of multimodal data are effectively associated, so that multimodal data can be effectively combined for comprehensive intent recognition, thereby improving the accuracy of intent recognition.

[0088] Step 106: Generate a message event intent for the target message event based on the associated content feature set and the global message content feature.

[0089] In some embodiments, the execution subject may generate a message event intent for a target message event based on an associated content feature set and a global message content feature. In practice, the execution subject may input an associated content feature set and a global message content feature into an intent recognition layer to generate a message event intent for a target message event, wherein the intent recognition layer includes: a feature fusion layer and a classifier. The feature fusion layer is used to perform feature fusion on an associated content feature set and a global message content feature. The classifier is used to classify the type of message event intent.

[0090] Step 107: Generate intent suggestion information for the message event intent.

[0091] In some embodiments, the above-mentioned execution subject can generate intent suggestion information for the message event intent. In practice, for example, the above-mentioned execution subject can call the API interface of the generative big model to generate intent suggestion information for the message event intent. For another example, the above-mentioned execution subject can generate intent suggestion information through a Llama model deployed in the above-mentioned execution subject. Specifically, the above-mentioned execution subject can use the associated content feature set and the global message content feature as the input of the Llama model to generate intent suggestion information.

[0092] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the intent recognition method applied to message events of some embodiments of the present disclosure, the intent recognition for multimodal data is effectively realized. Specifically, the reason for the inability to effectively recognize the intent is that the intent recognition method based on single-modal data cannot effectively extract features for the intent recognition requirements of multimodal data, resulting in the inability to effectively recognize the intent for multimodal data. Based on this, the intent recognition method applied to message events of some embodiments of the present disclosure first obtains a target message event, wherein the target message event is a message event to be recognized for intent acquired in real time, and the target message event includes: message content and message type, and the message type includes: audio message type, image message type, text message type and mixed message type. Secondly, in response to the above message type being a mixed message type, the message content encoder set corresponding to the above message type in the message content encoding model is activated. Then, in response to the successful activation, the message content is encoded by the message content encoder set corresponding to the above message type to obtain a message content feature set. In practice, when it is a mixed message type, the target message event is characterized by at least two of audio messages, image messages and text messages, so it is necessary to activate the corresponding content encoding model to perform feature extraction. Further, based on the above-mentioned message content feature set, a global message content feature is generated to embed the features corresponding to different types of messages into the same feature space. In addition, based on the feature position of the message content feature in the above-mentioned global message content feature, the message content features in the above-mentioned message content feature set are extracted for associated features to generate an associated content feature set. In this way, further feature extraction of associated features is performed from the position perspective. In addition, based on the above-mentioned associated content feature set and the above-mentioned global message content feature, a message event intent for the above-mentioned target message event is generated. Finally, intent suggestion information for the above-mentioned message event intent is generated. In this way, intent recognition for multimodal data can be effectively achieved.

[0093] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an apparatus for identifying intent of a message event, and these apparatus embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the intention recognition device applied to message events can be specifically applied to various electronic devices.

[0094] like Figure 3As shown, some embodiments of the intent recognition device 300 applied to message events include: an acquisition unit 301, an activation unit 302, a message content encoding unit 303, a first generation unit 304, an associated feature extraction unit 305, a second generation unit 306 and a third generation unit 307. The acquisition unit 301 is configured to acquire a target message event, wherein the target message event is a message event acquired in real time and to be subjected to intent recognition, and the target message event includes: message content and message type, and the message type includes: audio message type, image message type, text message type and mixed message type; the activation unit 302 is configured to, in response to the message type being a mixed message type, activate a message content encoder set corresponding to the message type in the message content encoding model; the message content encoding unit 303 is configured to, in response to successful activation, encode the message content through the message content encoder set corresponding to the message type. The message content is encoded to obtain a message content feature set; the first generating unit 304 is configured to generate a global message content feature according to the above message content feature set; the associated feature extraction unit 305 is configured to extract associated features of the message content features in the above message content feature set according to the feature position of the message content features in the above global message content features to generate an associated content feature set; the second generating unit 306 is configured to generate a message event intent for the above target message event according to the above associated content feature set and the above global message content features; the third generating unit 307 is configured to generate intent suggestion information for the above message event intent.

[0095] It can be understood that the units recorded in the device 300 for identifying the intent of a message event are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the intention recognition device 300 applied to message events and the units contained therein, and will not be repeated here.

[0096] Reference below Figure 4 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0097] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory 402 or a program loaded from a storage device 408 into a random access memory 403. Various programs and data required for the operation of the electronic device 400 are also stored in the random access memory 403. The processing device 401, the read-only memory 402, and the random access memory 403 are connected to each other via a bus 404. An input / output interface 405 is also connected to the bus 404.

[0098] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0099] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the read-only memory 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0100] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0101] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0102] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains a target message event, wherein the target message event is a message event obtained in real time and to be subjected to intent recognition, and the target message event includes: message content and message type, and the message type includes: audio message type, image message type, text message type and mixed message type; in response to the message type being a mixed message type, activate a message content encoder set corresponding to the message type in a message content encoding model; in response to successful activation, encode the message content through the message content encoder set corresponding to the message type to obtain a message content feature set; generate a global message content feature based on the message content feature set; extract associated features from the message content features in the message content feature set based on the feature position of the message content feature in the global message content feature to generate an associated content feature set; generate a message event intent for the target message event based on the associated content feature set and the global message content feature; generate intent suggestion information for the message event intent.

[0103] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, an activation unit, a message content encoding unit, a first generation unit, an associated feature extraction unit, a second generation unit, and a third generation unit. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances. For example, the message content encoding unit may also be described as "a unit that, in response to successful activation, encodes the above message content through a message content encoder set corresponding to the above message type to obtain a message content feature set".

[0106] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0107] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for identifying intent applied to a message event, comprising: Acquire a target message event, wherein the target message event is a message event acquired in real time and to be subjected to intent recognition, and the target message event includes: message content and message type, and the message type includes: audio message type, image message type, text message type, and mixed message type; In response to the message type being a mixed message type, activating a message content encoder set corresponding to the message type in a message content encoding model; In response to successful activation, encoding the message content by a message content encoder set corresponding to the message type to obtain a message content feature set; Generate a global message content feature according to the message content feature set; Extracting associated features from the message content features in the message content feature set according to the feature positions of the message content features in the global message content features to generate an associated content feature set; Generate a message event intent for the target message event according to the associated content feature set and the global message content feature; Generate intent suggestion information for the message event intent, in, The obtaining target message event includes: In response to monitoring a communication request initiated by a client, establishing a communication link with the client; Performing link security detection on the communication link; In response to the communication link passing the link security detection, generating a public key and a private key, wherein the public key is used for data encryption and the private key is used for data decryption; Distributing the public key to the client via the communication link; In response to receiving an encrypted message sent by the client through the communication link, decrypting the encrypted message by using the private key to obtain the target message event, wherein the encrypted message is obtained by the client through public key encryption, in, In response to the message type being a mixed message type, activating a message content encoder set corresponding to the message type in a message content encoding model, comprising: Decomposing the message type to generate a basic message type set, wherein the basic message type includes: at least one of an audio message type, an image message type, and a text message type; Generate an activation vector according to the basic message type set; According to the activation vector, a message content encoder set corresponding to the message type in the message content encoding model is loaded.

2. The method according to claim 1, wherein: The generating a global message content feature according to the message content feature set includes: Performing feature conversion on each message content feature in the message content feature set to generate a converted message content feature, thereby obtaining a converted message content feature set, wherein each converted message content feature in the converted message content feature set is located in the same feature space; According to the content positioning information corresponding to the converted message content feature, feature projection is performed on the converted message content features in the converted message content feature set to generate the global message content feature.

3. The method according to claim 2, wherein: The extracting associated features of the message content features in the message content feature set according to the feature position of the message content feature in the global message content feature to generate an associated content feature set includes: Through the associated feature extraction model, associated features are extracted from the message content features of adjacent feature positions in the message content feature set to generate an associated content feature set, wherein the associated feature extraction model includes: an associated feature extraction sub-model set, parameters are shared between the associated feature extraction sub-models in the associated feature extraction sub-model set, and the associated feature extraction sub-models in the associated feature extraction sub-model set are activated accordingly according to the number of associations.

4. The method according to claim 3, wherein: The message content is encoded by the message content encoder set corresponding to the message type to obtain a message content feature set, including: According to the basic message type set, the message content is segmented to generate a target information set, wherein the target information in the target information set includes: segmented message content and content positioning information, wherein the content positioning information represents a relative position of the segmented message content in the message content; For each target information in the target information group set, the following processing steps are performed: In response to the basic message type corresponding to the target information being an audio message type, performing a short-time Fourier transform on the segmented message content included in the target information to generate an audio spectrogram; The audio spectrogram is feature compressed by a feature encoder included in a first message content encoder to generate compressed features, wherein the first message content encoder is a message content encoder corresponding to the audio message type in the message content encoder set, and the feature encoder includes: 2 2D convolutional layers connected in series; The feature contrast learner included in the first message content encoder performs feature contrast learning on the compressed features to generate message content features in the message content feature set that correspond to the target information, wherein the feature contrast learner includes: a linear mapping layer and at least two feature capturers, and the feature capturer includes: a first normalization layer, a multi-head self-attention mechanism layer, a second normalization layer, a feature convolution block, a third normalization layer and a feedforward network.

5. The method according to claim 4, wherein: After the feature contrast learner included in the first message content encoder performs feature contrast learning on the compressed features to generate a message content feature in the message content feature set corresponding to the target information, the method further includes: In response to the basic message type corresponding to the target information being an image message type, segmenting the segmented message content included in the target information into image blocks to obtain an image block set; Determining an image block information degree of each image block in the image block set; According to the image block information degrees corresponding to the image blocks, the image block set is binary grouped to obtain a first image block set and a second image block set, wherein the mean image block information degree corresponding to the second image block set is less than the mean image block information degree corresponding to the first image block set; By means of a second message content encoder, image feature encoding is performed on the first image block set and the second image block set to generate a message content feature in the message content feature set corresponding to the target information, wherein the second message content encoder is a message content encoder corresponding to the image message type in the message content encoder set, the second message content encoder performs a random image mask on the first image block in the first image block set, and the second message content encoder does not perform an image mask on the second image block in the second image block set; In response to the basic message type corresponding to the target information being a text message type, a third message content encoder is used to perform text feature encoding on the segmented message content included in the target information to generate a message content feature in the message content feature set that corresponds to the target information, wherein the third message content encoder is a message content encoder in the message content encoder set that corresponds to the text message type.

6. An intention recognition device applied to a message event, comprising: an acquisition unit, configured to acquire a target message event, wherein the target message event is a message event acquired in real time and to be subjected to intent recognition, and the target message event includes: message content and message type, and the message type includes: audio message type, image message type, text message type, and mixed message type; an activation unit, configured to activate a message content encoder set corresponding to the message type in a message content encoding model in response to the message type being a mixed message type; a message content encoding unit configured to, in response to successful activation, encode the message content by a message content encoder set corresponding to the message type to obtain a message content feature set; A first generating unit is configured to generate a global message content feature according to the message content feature set; a correlation feature extraction unit configured to extract correlation features from the message content features in the message content feature set according to feature positions of the message content features in the global message content features, so as to generate a correlation content feature set; A second generating unit is configured to generate a message event intent for the target message event according to the associated content feature set and the global message content feature; A third generating unit is configured to generate intention suggestion information for the message event intention, in, The obtaining target message event includes: In response to monitoring a communication request initiated by a client, establishing a communication link with the client; Performing link security detection on the communication link; In response to the communication link passing the link security detection, generating a public key and a private key, wherein the public key is used for data encryption and the private key is used for data decryption; Distributing the public key to the client via the communication link; In response to receiving an encrypted message sent by the client through the communication link, decrypting the encrypted message by using the private key to obtain the target message event, wherein the encrypted message is obtained by the client through public key encryption, in, In response to the message type being a mixed message type, activating a message content encoder set corresponding to the message type in a message content encoding model, comprising: Decomposing the message type to generate a basic message type set, wherein the basic message type includes: at least one of an audio message type, an image message type, and a text message type; Generate an activation vector according to the basic message type set; According to the activation vector, a message content encoder set corresponding to the message type in the message content encoding model is loaded.

7. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images