User intention recognition method and related equipment

Abstract

The embodiment of the invention provides a user intention recognition method and related equipment, and the method comprises the steps: analyzing the description features of image data and the content features of text data based on the sequence of the image data in received multi-modal data prior to the text data; and based on the description features, the content features and the similarity parameters between the image data and the text data, determining a correlation result of the image data and the text data by using a classification model, and if the correlation result indicates that the image data and the text data are related data, determining a comprehensive intention of the image data and the text data as a user intention. By using the method, the identification accuracy of the user intention in the multi-modal interaction scene can be improved.

Description

Technical Field

[0001] This application belongs to the field of terminal technology and relates to image processing technology, and in particular to a user intent recognition method and related equipment. Background Technology

[0002] Different electronic devices or software products offer voice assistant services to quickly respond to user needs, retrieve relevant information, or invoke related functions. Typically, user intent is identified by analyzing their text messages, enabling corresponding actions or services to be performed. However, in multimodal interaction environments, the same input text may convey different intents depending on the accompanying voice, images, or video information, leading to reduced accuracy in user intent recognition and impacting user experience. Summary of the Invention

[0003] In view of the above, it is necessary to provide a user intent recognition method and related equipment that can solve the problem of accurately recognizing user intent in multimodal interaction environments.

[0004] In a first aspect, this application provides a user intent recognition method, the method comprising: receiving multimodal data input by a user, the multimodal data including image data and text data; when the image data is received before the text data, parsing the image data to obtain descriptive features and parsing the text data to obtain content features; determining a similarity parameter between the image data and the text data; obtaining a correlation result between the image data and the text data using a preset classification model based on the descriptive features, the content features, and the similarity parameter; if the correlation result indicates that the image data and the text data are related data, determining the user intent based on the comprehensive intent of the image data and the text data.

[0005] The above technical solution analyzes the descriptive features of the image data and the content features of the text data based on the order of image data preceding text data in the received multimodal data. Based on the descriptive features, content features, and similarity parameters between the image and text data, a classification model is used to determine the relevance between the image and text data. If the relevance result indicates that the image and text data are related, the combined intent of the image and text data can be determined as the user intent. This approach improves the accuracy of user intent recognition in multimodal interaction scenarios by performing content analysis and relevance judgment on image and text data.

[0006] In one possible implementation, the method further includes: if the text data is received before the image data, and the text data contains preset keywords, determining the user intent based on the comprehensive intent.

[0007] The above technical solutions can more effectively associate images with text content based on keywords in text data, thereby accurately understanding and responding to user needs.

[0008] In one possible implementation, the method further includes: if the text data is received before the image data, and if the text data does not contain preset keywords, determining the user intent based on the text data.

[0009] In one possible implementation, the method further includes: based on the user intent, providing query results or invoking applications, services, or functions related to the user intent; if it is not possible to provide corresponding query results or invoke related applications, services, or functions based on the user intent, determining an updated user intent based on the combined intent of the image data and the text data.

[0010] With the above technical solution, when text data is received first and image data is received later, if the text data does not contain preset keywords, the user's intent is determined based on the text data; if the query results or corresponding functions cannot be directly obtained based solely on the text data, the intent of both will be comprehensively analyzed in conjunction with the subsequently received image data to update and accurately determine the user's final needs, thereby providing more accurate services or responses.

[0011] In one possible implementation, the image data includes images or video frames extracted from videos; the text data includes text information or text information recognized from speech data.

[0012] The above technical solutions enable the reception of multimodal data input by users through diverse data reception methods, thereby improving the user's interactive experience.

[0013] In one possible implementation, the similarity parameter includes a first parameter, and the method for determining the first parameter includes: determining the cosine similarity between the descriptive feature and the content feature, and determining the first parameter based on the cosine similarity.

[0014] By employing the aforementioned technical solution, and calculating the cosine similarity between the descriptive features of image data and the content features of text data, the directional difference between image data and text data can be determined without considering its magnitude, thereby accurately measuring the degree of similarity between image data and text data in the feature space. Using cosine similarity as a similarity parameter effectively reflects the correlation between images and text, providing a scientific basis for subsequent judgments on whether image data and text data are related, and further improving the accuracy and efficiency of user intent recognition.

[0015] In one possible implementation, the similarity parameter includes a second parameter, and the method for determining the second parameter includes: parsing the image data to obtain image description text, generating prompt information based on the image description text, the prompt information being used to prompt a preset large model to generate query keywords based on the image description text; determining the distribution probability of the keywords in the text data in the query keywords, and determining the second parameter based on the distribution probability.

[0016] The above technical solutions can leverage the generation capabilities of pre-set large models and the ability to deeply understand language to improve the accuracy and efficiency of image and text data correlation assessment, providing strong support for the accurate recognition of user intent.

[0017] In one possible implementation, the method for obtaining the comprehensive intent of the image data and the text data includes: based on the image data and the text data, using a preset multimodal intent recognition model to obtain the comprehensive intent, wherein the comprehensive intent represents the intent to use the text data as a query statement corresponding to the image data.

[0018] The above technical solution can determine the user's intent by combining the image data and text data based on a large model when the image data and text data are related, thereby improving the accuracy of user intent recognition.

[0019] Secondly, this application provides a user intent recognition method, the method comprising: receiving multimodal data input by a user, the multimodal data including first data and second data; parsing the first data to obtain descriptive features, and parsing the second data to obtain content features; determining a similarity parameter between the first data and the second data; obtaining a correlation result between the first data and the second data using a preset classification model based on the descriptive features, the content features, and the similarity parameter; if the correlation result indicates that the first data and the second data are related data, determining the user intent based on the comprehensive intent of the first data and the second data.

[0020] The above technical solution allows for the determination of the correlation between the first and second data based on their descriptive features, content features, and similarity parameters. If the correlation results indicate that the first and second data are related, the combined intent of the first and second data can be determined as the user's intent. This approach improves the accuracy of user intent recognition in multimodal interaction scenarios by performing content analysis and correlation judgment on the first and second data.

[0021] In one possible implementation, the method further includes: if the correlation result indicates that the first data and the second data are unrelated data, determining the user intent based on the intent corresponding to the first data or the intent corresponding to the second data.

[0022] Through the above technical solution, when multimodal data is determined to be irrelevant, user intent can be flexibly determined based on data priority. When text data has a higher priority, the text data in the second set of data is directly used to obtain the user intent through the intent recognition model. However, when image data has a higher priority, such as in product search scenarios on online shopping platforms, image data, due to its rich product feature information, more accurately reflects user needs. Therefore, the image data (or processed image description text) in the first set of data can be directly input into the model to identify user intent. This data priority-based selection strategy ensures efficient and accurate understanding and response to user needs in different scenarios.

[0023] In one possible implementation, parsing the first data to obtain descriptive features includes: obtaining image description text of the image data in the first data through a preset visual language model; and performing feature extraction on the image description text to obtain the descriptive features of the first data.

[0024] In one possible implementation, feature extraction of the image description text includes: text encoding of the image description text.

[0025] In one possible implementation, parsing the second data to obtain content features includes: extracting features from the text data in the second data to obtain the content features of the second data.

[0026] The above technical solution can be used to analyze the first data and the second data to obtain the descriptive features of the first data and the content features of the second data, providing a data basis for determining whether text data and image data are related data.

[0027] In one possible implementation, the similarity parameter includes a first parameter, and the method for determining the first parameter includes: determining the cosine similarity between the descriptive feature and the content feature, and determining the first parameter based on the cosine similarity.

[0028] By employing the aforementioned technical solution, and calculating the cosine similarity between the descriptive features of the first data and the content features of the second data, the directional difference between the first and second data can be determined without considering its magnitude, thereby accurately measuring the degree of similarity between the first and second data in the feature space. Using cosine similarity as a similarity parameter effectively reflects the correlation between the first and second data, providing a scientific basis for subsequent judgments on whether the first and second data are related, and further improving the accuracy and efficiency of user intent recognition.

[0029] In one possible implementation, the similarity parameter includes a second parameter, and the method for determining the second parameter includes: parsing the first data to obtain image description text, generating prompt information based on the image description text, the prompt information being used to prompt a preset large model to generate query keywords based on the image description text; determining the distribution probability of the keywords in the second data in the query keywords, and determining the second parameter based on the distribution probability.

[0030] The above technical solutions can leverage the generation capabilities of pre-set large models and the ability to deeply understand language to improve the accuracy and efficiency of multimodal data correlation assessment, providing strong support for the accurate identification of user intent.

[0031] In one possible implementation, the classification model includes a binary classification model, and the training method of the binary classification model includes: acquiring training data, the training data including content features of first historical data, descriptive features of second historical data, similarity parameters and labels between the first historical data and the second historical data, the labels representing whether the first historical data and the second historical data are related data; and training the binary classification model using the training data based on a supervised training method until the prediction accuracy of the binary classification model is greater than a preset accuracy threshold.

[0032] The above technical solution utilizes a pre-defined binary classification model, combining descriptive features of image data, content features of text data, and similarity parameters between them, to accurately determine the relevance between images and text data. This binary classification model undergoes rigorous supervised training, with training data encompassing the features, similarities, and labels of historical images and text, ensuring the model can accurately distinguish between relevant and irrelevant data. By setting a high accuracy threshold and continuously optimizing model parameters, the prediction accuracy of the classification model can be improved, providing strong support for the correlation analysis of image and text data.

[0033] In one possible implementation, the method for obtaining the comprehensive intent corresponding to the first data and the second data includes: obtaining the comprehensive intent based on the first data and the second data using a preset multimodal intent recognition model.

[0034] The above technical solution can determine user intent based on the comprehensive intent of multimodal data output by a large model when multimodal data is relevant data, thereby improving the accuracy of user intent recognition.

[0035] Thirdly, embodiments of this application provide an electronic device, the electronic device including a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device performs the above-described user intent recognition method.

[0036] Fourthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to input multimodal data into the chip system and execute the user intent recognition method described above.

[0037] Fifthly, embodiments of this application provide a computer storage medium storing program instructions that, when executed on an electronic device, cause the processor of the electronic device to perform the aforementioned user intent recognition method.

[0038] Furthermore, the technical effects brought about by the third to fifth aspects can be found in the descriptions of the methods in the above-mentioned method section, and will not be repeated here. Attached Figure Description

[0039] Figure 1(a)-Figure 1(b) This is an example diagram of a multimodal application scenario provided in an embodiment of this application.

[0040] Figure 2 This is an example diagram of a UX interface provided in an embodiment of this application.

[0041] Figure 3 This is an example diagram illustrating the multimodal recognition principle of a related technology provided in an embodiment of this application.

[0042] Figure 4 This is a software architecture diagram of an electronic device provided in an embodiment of this application.

[0043] Figure 5 This is a flowchart of a user intent recognition method provided in an embodiment of this application.

[0044] Figure 6 This is a flowchart of a user intent recognition method provided in another embodiment of this application.

[0045] Figure 7This is a flowchart of a user intent recognition method provided in another embodiment of this application.

[0046] Figure 8 This is a flowchart of a user intent recognition method provided in another embodiment of this application.

[0047] Figure 9 A hardware architecture diagram of an electronic device provided in one embodiment of this application.

[0048] Figure 10 An architecture diagram of a chip system provided in one embodiment of this application. Detailed Implementation

[0049] In one embodiment of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in one embodiment of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to limit the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. Where there is no conflict, the following embodiments and features described herein can be combined with each other.

[0051] With the rapid development of technology, various electronic devices (such as mobile phones) and software products have integrated voice assistant services. These services aim to respond quickly to user needs in an intelligent and user-friendly way, assisting users in querying information or accessing relevant functions, thereby improving user convenience and efficiency in daily life and work. Related user intent recognition methods typically rely on analyzing the text information input by the user, using natural language processing techniques and machine learning algorithms to interpret the user's intent, and then performing corresponding operations or providing related services.

[0052] However, in increasingly rich multimodal interaction environments, user input is no longer limited to a single text format, but increasingly incorporates information from multiple modalities such as voice, images, and video. While this multimodal interaction provides users with a more natural and intuitive way of expression, it also presents unprecedented challenges to the accurate identification of user intent. In one example, the same input text, with the addition of different modal information, may express completely different intents, making it difficult for text-based user intent recognition methods to accurately identify user intent.

[0053] Referring to Figure 1(a), which illustrates a multimodal application scenario provided in an embodiment of this application, a user inputs the text "Where is this?" and an image in the graphical user interface of a mobile phone. The user's true intention could be the location or city shown in the image. However, if the user's intention is identified solely based on the text data, the identified intention might be "to query the user's current location," such as the location or city corresponding to the phone's location information. Similarly, in another example, as shown in Figure 1(b), a user first inputs an image and then the text "What's the weather like here?" The user's true intention could be the weather conditions of the area shown in the image. However, if the user's intention is identified solely based on the text data, the identified intention might be "to query the weather conditions of the user's current location," such as the weather conditions of the location or city corresponding to the phone's location information. Clearly, it is difficult to identify the user's true intention based solely on text data.

[0054] To more accurately identify user intent, one approach used in related technologies is based on User Experience (UX) design methods. This involves targeted user experience design tailored to the specific needs, user characteristics, and business scenarios of a particular vertical (e.g., a specific industry or field). For example... Figure 2As shown, UX provides a user interface that allows users to access and interact with specific vertical domains by selecting controls displayed on the interface. However, UX design typically requires significant manpower and cannot be quickly scaled, and it is difficult to apply to use cases that require domain segmentation. For example, if the first sentence of a user's data is a question asking what is included in the image data, and the second sentence is a command to open the phone's alarm clock settings interface, UX cannot handle this kind of domain segmentation use case.

[0055] Another approach involves training a large multimodal model to improve its intent recognition accuracy. However, this method is still immature in multimodal recognition technology, and in practical applications, it suffers from low recognition accuracy and poor model robustness. This is mainly due to the complexity and diversity of multimodal data, which poses significant challenges to model training and optimization. Furthermore, processing large amounts of data from different modalities and performing complex model training and inference requires substantial computational resources and time, resulting in a significant cost burden. Directly performing large-scale multimodal intent recognition places high demands on the model and is currently difficult to apply directly to real-world scenarios.

[0056] Although manufacturers have developed related multimodal interaction functions, such as integrating multimodal interaction technology into products, allowing users to interact with devices through voice, gestures, and other methods, thereby improving user experience and product competitiveness, the methods used are not directly based on large multimodal models for intent recognition. Their recognition principles can be found in [reference needed]. Figure 3 As shown.

[0057] The combination of image modality injection and query intent recognition faces challenges from the complexity of image information and the diversity of user queries, making it difficult for the system to accurately determine whether the image content is relevant to the query, and the output intent recognition results are still not accurate enough.

[0058] Furthermore, multimodal interaction environments also present issues of conflict and redundancy between modal information. Information from different modalities may contradict each other, leading to ambiguity in the recognition of user intent; simultaneously, information from some modalities may not be substantially helpful in recognizing user intent, resulting in wasted resources and reduced computational efficiency. These problems further increase the difficulty of user intent recognition and affect user experience satisfaction.

[0059] To address the aforementioned issues, this application provides a user intent recognition method that parses received multimodal data and determines the user intent based on the comprehensive analysis of the multimodal data, thereby improving the accuracy of user intent recognition in multimodal interaction scenarios.

[0060] User intent recognition methods can be applied to various electronic devices, such as mobile phones, tablets, wearable devices, camera devices, computers, and self-moving devices. Electronic devices include application processors, which run the operating system. The following section will discuss... Figure 4 The software structure of an electronic device is illustrated by example.

[0061] See Figure 4 The diagram shown is a software architecture diagram of an electronic device provided in an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. For example, the Android system, from top to bottom, consists of the application layer 101, framework layer 102, Android runtime and system libraries 103, hardware abstraction layer 104, kernel layer 105, and hardware layer 106.

[0062] Application layer 101 may include a series of application packages. For example, application packages may include applications such as camera, gallery, calendar, calling, map, navigation, WLAN, Bluetooth, music, video, SMS, device control services, etc.

[0063] The framework layer 102 provides an Application Programming Interface (API) and programming framework for applications in the application layer. The application framework layer includes predefined functions. For example, it may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0064] The window manager manages window programs. It can obtain screen size, determine the presence of a status bar, lock the screen, and capture screenshots. The content provider stores and retrieves data, making it accessible to applications. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, and phone books. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon can include views for displaying text and views for displaying images. The phone manager provides communication functionality for electronic devices, such as managing call status (including connection and disconnection). The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager allows applications to display notifications in the status bar, conveying informational messages that disappear automatically after a short pause without user interaction. For example, the notification manager is used to notify of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the system's top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, causing electronic devices to vibrate, and flashing indicator lights.

[0065] The Android Runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0066] Application layer 101 and framework layer 102 run in a virtual machine. The virtual machine executes the Java files of the application layer and framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0067] System library 103 may include multiple functional modules. For example, a surface manager, media libraries, 3D graphics processing libraries (e.g., Open GLES), 2D graphics engines (e.g., SGL), etc.

[0068] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D Graphics Engine is the drawing engine for 2D graphics.

[0069] Hardware Abstraction Layer 104 runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to the upper layers.

[0070] Kernel layer 105 is the layer between hardware and software. Kernel layer 105 contains at least the display driver, touch driver, audio driver, and sensor driver.

[0071] The kernel layer 105 is the core of the operating system for electronic devices. It is the first layer of software extension based on the hardware, providing the most basic functions of the operating system. It is the foundation for the operation of the operating system, responsible for managing system processes, memory, device drivers, files, and network systems, and determining the system's performance and stability. For example, the kernel layer can determine the timing of an application's operation on a certain part of the hardware.

[0072] Kernel layer 105 includes hardware-dependent programs such as interrupt handlers and device drivers, as well as basic, common, and frequently running modules such as clock management and process scheduling modules, and critical data structures. The kernel layer can be located within the processor or embedded in internal memory.

[0073] Hardware layer 106 includes the hardware of electronic devices, such as displays, buttons, cameras, etc.

[0074] See Figure 5 The diagram shown is a flowchart of a user intent recognition method provided in an embodiment of this application. The user intent recognition method can be applied to various electronic devices and specifically includes the following process.

[0075] S301 receives multimodal data input by the user.

[0076] In some embodiments of this application, in a multimodal interaction scenario, multimodal data may include, but is not limited to, image data and text data, wherein image data includes images or video frames extracted from videos; and text data includes text information or text information recognized from speech data.

[0077] In some embodiments of this application, the electronic device can receive photos, images, etc., directly uploaded or captured by the user as image data, and can also extract single or multiple video frames as image data from user-uploaded or real-time recorded videos. The electronic device can provide a graphical user interface (GUI) to receive text typed by the user as text information, and can also use speech recognition technology to recognize user-inputted voice data to obtain text information. The source of image data and text data is not limited; it can be various types of data stored, received, or input in the electronic device, or it can be various types of data directly downloaded from the Internet.

[0078] Through the above embodiments, multimodal data input by users can be received through diverse data reception methods, thereby improving the user's interactive experience.

[0079] S302, when image data is received before text data, the image data is parsed to obtain descriptive features, and the text data is parsed to obtain content features.

[0080] In some embodiments of this application, the order in which users input data from different modalities in a multimodal interaction scenario may affect the accuracy of user intent recognition. The electronic device can determine the reception time of each piece of data in the received multimodal data, thereby determining the reception order of the data from different modalities based on the reception time. Based on the reception order, different scenarios can be determined, and then a method corresponding to the current scenario can be used to recognize the user intent, thereby improving the accuracy of user intent recognition based on the current scenario.

[0081] In one example, the electronic device receives two pieces of multimodal data: image data A and text data B, with image data A being received before text data B. The electronic device can determine that the current scenario is one where image data was received before text data (hereinafter referred to as Scenario 1), and use the method corresponding to Scenario 1 to determine the user's intent in the current scenario.

[0082] In some embodiments of this application, in scenario one, the user's true intent can typically include: a combined intent of image data and text data, or an intent corresponding to the text data. The combined intent indicates the intent to use text data as a query statement corresponding to the image data. For example, the text data can be used to characterize a question posed by the user to the image data, or an information query based on the image data, or a call to related functions, services, applications, etc.

[0083] In some embodiments of this application, since text data is the easiest data to parse to obtain intent, scenario one can be further determined and classified by determining whether the text data contains preset keywords related to image data. The preset keywords may include keywords used to indicate image data received prior to the text data; for example, preset keywords may include, but are not limited to, "above," "picture," "image," "sheet," etc.

[0084] In some embodiments of this application, scenario one may further include the following situation (a): the text data does not contain preset keywords. If the text data does not contain preset keywords, the user intent can be determined based on the judgment result of whether the image data and text data are related data. For example, if it is determined that the image data and text data are related data, the user intent is determined based on the comprehensive intent. Alternatively, if it is determined that the image data and text data are irrelevant data, the user intent is determined based on the text data.

[0085] In one example, the multimodal data includes image data A and text data B received sequentially. Text data B is "What is the answer to the question?". By performing pre-defined keyword recognition on text data B, it is determined that text data B does not contain the pre-defined keyword. Therefore, this can be identified as situation (a), and the user's intent can be determined based on the judgment result of whether the image data and text data are related data.

[0086] In some embodiments of this application, it can be determined whether image data and text data are related data based on the descriptive features of image data and the content features of text data. Specifically, the image description text of the image data can be obtained through a preset visual language model; feature extraction is then performed on the image description text to obtain the descriptive features of the image data.

[0087] In some embodiments of this application, the visual language model can be a pre-trained large-scale visual-language model (VLM) capable of simultaneously processing and understanding both visual (image) and linguistic (text) modalities. For example, the visual language model can include, but is not limited to, image description models (e.g., GPT (Generative Pre-trained Transformer)) and visual question answering models (e.g., Intern GPT, Det GPT). In one example, prompt words can be set for the visual language model, which can be used to prompt the visual language model to perform corresponding functions. For example, the prompt word could be: "generate image description text for image data."

[0088] In some embodiments of this application, a preset text encoder can be used to extract features from the image description text to obtain descriptive features of the image data. Feature extraction from the image description text may include text encoding of the image description text. For example, the embedding vector of the text-encoded image description text can be used as the descriptive features of the image data.

[0089] In some embodiments of this application, obtaining the content features of text data includes: performing feature extraction on the text data to obtain the content features of the text data. Feature extraction on the text data may include: text encoding of the text data. For example, the embedding vector of the text data obtained from text encoding is used as the content feature of the text data. The embedding vector of the text data and the embedding vector of the image description text can be vector features mapped to the same space, thereby enabling correlation analysis in subsequent processes.

[0090] Through the above embodiments, by identifying preset keywords in text data, it is possible to determine the combined intent of text data and image data containing preset keywords as the user intent. If the text data does not contain preset keywords, the determination of whether to use the intent of the text data or the combined intent as the user intent is based on the judgment result of whether the text data and image data are related data. This can effectively improve the accuracy and flexibility of user intent recognition when image data is received before text data.

[0091] S303, determine the similarity parameter between image data and text data.

[0092] In some embodiments of this application, the similarity parameter includes a first parameter, and the method for determining the first parameter includes: determining the cosine similarity between the descriptive feature and the content feature, and determining the first parameter based on the cosine similarity. For example, if the cosine similarity is 0.7, then the first parameter can be 0.7.

[0093] By calculating the cosine similarity between the descriptive features of image data and the content features of text data, the directional difference between the two data can be determined regardless of its magnitude, thus accurately measuring the degree of similarity between them in the feature space. Using cosine similarity as a similarity parameter can effectively reflect the correlation between images and text, providing a scientific basis for subsequent judgments on whether image and text data are related, and further improving the accuracy and efficiency of user intent recognition.

[0094] In some embodiments of this application, the similarity parameter includes a second parameter, and the method for determining the second parameter includes: parsing image data to obtain image description text, generating prompt information based on the image description text, the prompt information being used to prompt a preset large model to generate query keywords based on the image description text; determining the distribution probability of keywords in the text data in the query keywords, and determining the second parameter based on the distribution probability.

[0095] In some embodiments of this application, the pre-set large model includes, but is not limited to, a generative pre-trained (GPT) model. This pre-set large model can generate query keywords highly relevant to the image description text based on prompt information. These query keywords can summarize the main content or features of the image, providing an important reference for judging the relevance between text data and image data. By calculating the distribution probability of keywords in the text data among the query keywords, the degree of association between text data and image data can be quantified, thereby determining the second parameter.

[0096] In one example, the multimodal data includes an image (C) containing a math problem and a text (D) titled "Solve this math problem and tell me the answer." To determine the similarity parameters between the image and text data, a pre-defined visual language model can be used to parse the image data C, obtaining image description text, such as: "An image containing a math problem with a quadratic equation, where the math problem is: ***." Based on the image description text, a prompt is generated, for example, "Please generate relevant query keywords based on the following image description text: 'An image containing a math problem with a quadratic equation, where the math problem is: ***'." A pre-defined large model generates query keywords based on the prompt information; for example, query keywords can be a series of keywords such as: "quadratic equation," "mathematics," "problem," "mathematical question," "solve," "answer," "analysis," etc. To determine the probability distribution of keywords in the text data among the query keywords, keywords can be extracted from the text data D, such as: "solution," "mathematical question," "answer," etc. Calculate the probability distribution of these keywords among the previously generated query keywords. For example, "math problem" is a directly matching keyword with a probability distribution of 1 (or close to 1, which can be set according to actual needs); "answer" is a directly matching keyword with a probability distribution of 1; while "solution" does not directly match the query keywords, it is similar to keywords such as "solve" and "analysis" in the query keywords, so a certain probability distribution can be obtained through semantic similarity calculation (e.g., 0.97). When determining the second parameter based on the probability distribution, it can be determined based on the average of all probability distributions of all keywords in the text data. For example, the second parameter = (1 + 1 + 0.91) / 3 = 0.97.

[0097] Through the above embodiments, the generation capabilities of the preset large model and the ability to deeply understand language can be utilized to improve the accuracy and efficiency of image and text data correlation assessment based on image description text, thus providing support for the accurate recognition of user intent.

[0098] In some other embodiments of this application, the similarity parameter may further include a third parameter, which can be determined based on the following formula: P(W_1,W_2)=P(W_1)×P(W_2|W_1), where P(W_1) represents the frequency or probability of the descriptive feature W_1 of the image data appearing in the overall dataset. In practical applications, this probability can be preset as a constant or a known value, as it does not depend on the user query. P(W_2|W_1) represents the conditional probability that the content feature W_2 of the text data is related to the descriptive feature W_1 given the descriptive feature W_1; the higher the conditional probability, the stronger the association between the content feature and the descriptive feature. P(W_1,W_2) represents the joint probability that the descriptive feature W_1 and the content feature W_2 appear simultaneously. This joint probability can be used as the third parameter to quantify the association between the image data and the text data as a whole. In one example, prior data can be used to statistically analyze different types of content features W_2 and descriptive features W_1, and determine the range of the condition P(W_2|W_1) between different types of content features W_2 and descriptive features W_1. After determining the types of content features of image data and descriptive features of text data in the received multimodal data, the similarity parameter between image data and text data is determined based on the range of P(W_2|W_1) corresponding to that type.

[0099] Through the above embodiments, in order to quantify the correlation between image data and text data as a whole, a conditional probability-based model is employed to statistically analyze the relationship between different types of content features W_2 and descriptive features W_1 using prior data, and to determine the range of their conditional probabilities P(W_2|W_1). When multimodal data is received, the types of content features of image data and descriptive features of text data are determined. Then, based on the conditional probability ranges corresponding to these types, the similarity parameters between image data and text data are calculated. In this way, the correlation between images and text can be effectively evaluated, providing strong support for the accurate recognition of subsequent user intent.

[0100] S304. Based on the descriptive features, content features, and similarity parameters, a preset classification model is used to obtain the correlation results between image data and text data.

[0101] In some embodiments of this application, the classification model includes a binary classification model. The training method of the binary classification model includes: acquiring training data, which includes content features of historical image data, descriptive features of historical text data, similarity parameters and labels between historical image data and historical text data, where the labels represent whether historical image data and historical text data are related data; and training the binary classification model using the training data based on a supervised training method until the prediction accuracy of the binary classification model is greater than a preset accuracy threshold.

[0102] In one example, the training data includes content features of historical image data and descriptive features of corresponding historical text data. Each pair of data has a similarity parameter and a specific label. The similarity parameter accurately reflects the degree of similarity between the historical image data and the historical text data; the label is a binary identifier used to indicate whether there is a correlation between the historical image data and the historical text data in each pair. For example, if the historical image data and the historical text data are related, the label is 1; if they are not related, the label is 0.

[0103] Supervised training methods can be used to train binary classification models. Guided by known data labels, the model can repeatedly learn and internalize the correspondence between data features and labels, gradually improving its predictive ability. To ensure the model's accuracy and reliability, an accuracy threshold (e.g., 0.98) can be set to measure model performance, requiring the model's prediction accuracy to reach or exceed this threshold during training. Therefore, during training, the model's predictive performance can be continuously monitored, and necessary model parameter adjustments and optimizations can be made based on the actual situation. If the model's prediction accuracy exceeds the accuracy threshold, the model training can be considered complete, and the model has acquired the required classification ability and stability.

[0104] In one example, a binary classification model may include an input layer, a feature fusion layer, a hidden layer, and an output layer. The input layer receives descriptive features from image data, content features from text data, and a similarity parameter. The feature fusion layer fuses the descriptive and content features to obtain a fusion vector; for example, this can be achieved through concatenation, weighted summation, or other fusion strategies. The feature fusion layer also integrates the fusion vector with the similarity parameter so that the resulting integrated vector contains similarity information between the descriptive and content features. The hidden layer may include multiple neurons for performing nonlinear transformations and feature extraction on the integrated vector, thereby learning complex features capable of distinguishing between relevant and irrelevant data. The output layer includes one neuron for outputting the model's prediction result, which can be a probability value representing the probability that the input data belongs to the positive class (relevant data). A threshold (e.g., 0.5) can be set to convert the probability value into a binary classification label; for example, values ​​greater than the threshold are converted to binary label 1, or values ​​less than or equal to the threshold are converted to binary label 0.

[0105] In one example, the multimodal data includes an image data C containing a math problem and a text data D "Solve this math problem and tell me what the answer is." The image description text is: "An image containing a math problem with a quadratic equation, where the math problem is: ***." The similarity parameters between image data C and text data D include a first parameter of 0.7 and a second parameter of 0.97. The output of the binary classification model is 1, indicating that image data C and text data D are related data.

[0106] Through the above embodiments, a pre-defined binary classification model can be used to accurately determine the correlation between image and text data by combining the descriptive features of image data, the content features of text data, and the similarity parameters between them. This binary classification model undergoes rigorous supervised training, and its training data covers the features, similarities, and labels of historical images and text, ensuring that the model can accurately distinguish between relevant and irrelevant data. By setting a high accuracy threshold and continuously optimizing the model parameters, the prediction accuracy of the classification model can be improved, providing strong support for the correlation analysis of image and text data.

[0107] S305, if the correlation results indicate that the image data and text data are related data, determine the user intent based on the combined intent of the image data and text data.

[0108] In some embodiments of this application, if the correlation result indicates that the image data and the text data are related data, a comprehensive intent can be obtained based on the image data and the text data using a preset multimodal intent recognition model. The comprehensive intent represents the intent to use the text data as a query statement for the corresponding image data.

[0109] In one example, image description text can be obtained from image data using a pre-defined visual language model. This image description text and the text data are then input into a multimodal intent recognition model. The multimodal intent recognition model then uses the text data as a query statement for the image description text and outputs the corresponding comprehensive intent. The multimodal intent recognition model can be a pre-defined deep learning-based artificial neural network model, including but not limited to combinations of one or more of the following models: Generative Pre-Trained (GPT) models, Transformer models, Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, etc.

[0110] In one example, the multimodal data includes image data C containing a math problem and text data D: "Solve this math problem and tell me the answer." The image description text is: "An image containing a math problem with a quadratic equation, where the math problem is: ***." The relevance results indicate that image data C and text data D are relevant data, which can prompt the multimodal intent recognition model to use the text data as the query statement for the image description text and output the corresponding comprehensive intent. For example, the comprehensive intent could be: "Solve the quadratic equation math problem in image data C and output the answer."

[0111] In another example, the multimodal data includes a building image and a text message "I want to go here". After using the methods of the above embodiments, the building image and the text message are determined to be related data. Based on the building image and the text message, the corresponding comprehensive intent is determined. For example, the comprehensive intent means to determine the target location based on the specific location identified in the image, to open the navigation software and provide navigation services to reach the target location.

[0112] The two examples above are merely illustrative. In actual applications, the user intent obtained can include not only the results of the query but also the invocation of relevant applications, services, or functions to directly provide relevant services to the user. For example, calling a video playback software to play a video or dialing a phone number identified from an image.

[0113] Through the above embodiments, when image data and text data are related data, the user intent can be determined based on the comprehensive intent of the image data and text data output by the large model, thereby improving the accuracy of user intent recognition.

[0114] The user intent recognition method provided in this application can analyze the descriptive features of the image data and the content features of the text data based on the order of image data preceding text data in received multimodal data. Based on the descriptive features, content features, and similarity parameters between the image data and text data, a classification model is used to determine the correlation result between the image data and text data. If the correlation result indicates that the image data and text data are related, the combined intent of the image data and text data can be determined as the user intent. This method improves the accuracy of user intent recognition in multimodal interaction scenarios by performing content analysis and correlation judgment on image data and text data.

[0115] In some embodiments of this application, if the correlation result indicates that the image data and text data are irrelevant, the user intent can be determined based on the intent corresponding to the text data. This method is applicable when the priority of text data is higher than that of image data, allowing the text data to be directly input into a preset intent recognition model to obtain the intent corresponding to the text data as the user intent. In one example, the modal data includes an image data C containing a math problem and a text data E "Continue playing music for me." The correlation result indicates that the image data and text data are irrelevant, and the user intent can be determined based on the text data E.

[0116] In another example of this application, the user's true intent in Scenario 1 may also include the intent corresponding to the image data. For example, if the correlation result indicates that the image data and text data are irrelevant, the user's intent can be determined based on the intent corresponding to the image data. This method can be applied to situations where the priority of image data is higher than that of text data, thus allowing the image data (or image description text) to be directly input into a preset intent recognition model to obtain the intent corresponding to the image data as the user's intent.

[0117] In one example, if the current application environment is an online shopping platform, users can search for similar or related products by uploading images. In this scenario, the image data uploaded by the user (e.g., a photo of a shoe) better expresses their needs than the provided text data (e.g., "Find me the same style"). Therefore, image data takes precedence over text data because it contains enough information to identify the characteristics of the product (such as style, color, etc.), thus enabling a more accurate inference of the user's intent. Therefore, the platform can directly input the image data (or processed image description text, if available) into a pre-defined intent recognition model to obtain the intent corresponding to the image data as the user's intent.

[0118] Through the above embodiments, when image data and text data are determined to be irrelevant, user intent can be flexibly determined based on data priority. When text data has a higher priority, the user intent is directly obtained through the intent recognition model using the text data. However, when image data has a higher priority, such as in product search scenarios on online shopping platforms, image data, due to its rich product feature information, more accurately reflects user needs. Therefore, image data (or processed image description text) can be directly input into the model to identify user intent. This data priority-based selection strategy ensures efficient and accurate understanding and response to user needs in different scenarios. Furthermore, in addition to using priority as a basis for determining the recognition order, intent recognition can be performed separately on image data and text data. Then, the relevance of the two recognition results can be evaluated. If the two recognition results are irrelevant, one can be selected for determination. Alternatively, a choice can be provided for the user to further confirm their intent; for example, a prompt can be output asking the user which intent to process next, or both recognition results can be output. In practical applications, the examples described above are not the only possibilities.

[0119] In some other embodiments of this application, the user's true intent in Scenario 1 may also include the user intent corresponding to the image data. For example, if the image data contains text data, and a clear user intent can be directly obtained after parsing the text data, the intent of the text data in the image data can be directly obtained as the user intent.

[0120] In one example, the multimodal data includes an image data F containing a math problem and a text data G that says "I don't know". The image data F contains the text "Please use the ** solution method to solve the following math problem". Since the text data in the image data already clearly reveals the intent, there's no need to perform further parsing on the text data G; the intent corresponding to the text in the image data F can be directly obtained as the user's intent.

[0121] Through the above embodiments, the accuracy and efficiency of intent recognition can be improved in scenarios where the text in the image data clearly expresses the user's needs.

[0122] In some embodiments of this application, scenario one may further include the following situation (b): the text data contains preset keywords. If the text data contains preset keywords, the user's intent can be determined based on the combined intent of the image data and the text data.

[0123] In one example, the multimodal data includes image data A and text data H, where text data H is "What is the answer to the question in the picture above?" By performing preset keyword recognition on text data H, it is determined that text data H contains preset keywords such as "above", "picture", and "image". This can be determined as situation (b), where text data H is the query statement for image data A, and the combined intent of the image data and text data is taken as the user intent.

[0124] Through the above embodiments, images can be more effectively associated with text content based on keywords in text data, thereby accurately understanding and responding to user needs.

[0125] In some embodiments of this application, the multimodal data received by the electronic device includes two data sets: text data I and image data J, wherein the text data I is received earlier than the image data J. The electronic device can determine that the current scenario is one in which the text data is received before the image data (hereinafter referred to as Scenario Two), and use the method corresponding to Scenario Two to determine the user intent of the current scenario.

[0126] In some embodiments of this application, similar to scenario one, scenario two may further include the following situation (a): the text data does not contain preset keywords. If the text data does not contain preset keywords, and the priority of the text data is higher than the priority of the image data, the user intent is determined based on the text data. Alternatively, if the text data does not contain preset keywords, and the priority of the image data is higher than the priority of the text data, the user intent is determined based on the image data.

[0127] In some embodiments of this application, if the user intent is determined based on text data, but it is impossible to provide the user with corresponding query results or invoke corresponding applications, services, or functions based on the determined user intent, an updated user intent can be determined based on the combined intent of image data and text data. In one example, if text data I is "What color is this ball?", and it is impossible to provide corresponding query results based on the user intent determined by text data I, an updated user intent can be determined based on the combined intent of text data I and subsequently received image data J. For example, the updated user intent could be "What color is the ball in image data J?".

[0128] In another example, if the user's intent is determined based on text data to beautify an image using an application XX (e.g., application XX is an application for editing and beautifying images), but the electronic device does not have application XX installed, making it impossible to edit or beautify the image entered by the user, in some embodiments of this application, the updated user intent can be determined based on the combined intent of image data and text data. In other embodiments of this application, when the corresponding application, service, or function cannot be invoked according to the user's intent, the electronic device can output a prompt message. The prompt message can prompt the user to download the relevant application from the application market to fulfill the user's intent, or it can prompt the user that the current device cannot invoke the corresponding function or service, and ask the user to continue to enter further information. Based on the further information, the updated user intent is re-determined, and corresponding operations are performed according to the updated user intent, including at least providing feedback on query results and invoking application services or functions related to the user's intent.

[0129] Based on the above embodiments, when an electronic device receives text data first and then image data, it will be identified as a specific situation (Situation Two), and the user's intent will be processed according to the rules under this situation. If the text data does not contain preset keywords, the user's intent will be determined based on the priority of the text and image data; if the query results or corresponding functions cannot be directly obtained based solely on the text data, the intent of both will be comprehensively analyzed in conjunction with the subsequently received image data to update and accurately determine the user's final needs, thereby providing a more accurate service or response.

[0130] In some embodiments of this application, similar to scenario one, scenario two may further include the following situation (b): the text data contains preset keywords. If the text data contains preset keywords, the user intent is determined based on the combined intent of the text image data and the image data. The embodiment of scenario (b) in scenario one can be referred to, and will not be described again.

[0131] Through the above embodiments, images can be more effectively associated with text content based on keywords in text data, thereby accurately understanding and responding to user needs.

[0132] refer to Figure 6 The diagram shown is a flowchart of a user intent recognition method provided in another embodiment of this application.

[0133] S401 receives multimodal data input by the user.

[0134] In some embodiments of this application, in a multimodal interaction scenario, the multimodal data may include first data and second data, wherein the first data includes images or video frames extracted from videos; and the second data includes text information or text information recognized from speech data.

[0135] In some embodiments of this application, the method for an electronic device to receive first data and second data can be referred to the above description of step S301.

[0136] Through the above embodiments, multimodal data input by users can be received through diverse data reception methods, thereby improving the user's interactive experience.

[0137] S402, parse the first data to obtain descriptive features, and parse the second data to obtain content features.

[0138] In some embodiments of this application, it can be determined whether the first data and the second data are related data based on the descriptive features of the first data and the content features of the second data, thereby determining the user's intent based on the determination result of whether they are related data. Specifically, image description text of the image data in the first data can be obtained through a preset visual language model; feature extraction is performed on the image description text to obtain the descriptive features of the first data.

[0139] In some embodiments of this application, the visual language model can be referenced to the description above regarding step S302.

[0140] In some embodiments of this application, a preset text encoder can be used to extract features from the image description text to obtain the descriptive features of the first data. Feature extraction from the image description text may include text encoding of the image description text. For example, the embedding vector of the text-encoded image description text can be used as the descriptive features of the first data.

[0141] In some embodiments of this application, obtaining the content features of the second data includes: extracting features from the text data in the second data to obtain the content features of the text data. Extracting features from the text data may include: text encoding the text data. For example, the embedding vector of the text data obtained from the text encoding is used as the content feature of the second data. The embedding vector of the text data and the embedding vector of the image description text can be vector features mapped to the same space, thereby enabling correlation analysis in subsequent processes.

[0142] Through the above embodiments, the first data and the second data can be parsed to obtain the descriptive features of the first data and the content features of the second data, providing a data basis for determining whether text data and image data are related data.

[0143] S403, determine the similarity parameter between the first data and the second data.

[0144] In some embodiments of this application, the method for determining the similarity parameter between the first data and the second data can refer to the description of step S303 above, wherein the image data in step S303 can be replaced with the first data, and the text data in step S303 can be replaced with the second data. In one example, the similarity parameter between the first data and the second data may also include a first parameter and a second parameter. In another example, the similarity parameter between the first data and the second data may also include a third parameter.

[0145] The above embodiments enable the effective evaluation of the correlation between multimodal data based on similarity parameters, providing strong support for the accurate identification of user intent in the future.

[0146] S404. Based on the descriptive features, content features, and similarity parameters, a preset classification model is used to obtain the correlation results between the first data and the second data.

[0147] In some embodiments of this application, the classification model includes a binary classification model. The training method of the binary classification model includes: acquiring training data, which includes content features of first historical data, descriptive features of second historical data, similarity parameters and labels between the first historical data and the second historical data, and the labels represent whether the first historical data and the second historical data are related data; and training the binary classification model using the training data based on a supervised training method until the prediction accuracy of the binary classification model is greater than a preset accuracy threshold.

[0148] In one example, the training data includes content features of first historical data and descriptive features of corresponding second historical data. Each pair of data has a similarity parameter and a specific label. The similarity parameter accurately reflects the degree of similarity between the first and second historical data; the label can be a binary identifier used to indicate whether there is a correlation between the first and second historical data in each pair. For example, if the first and second historical data are related, the label is 1; if they are not related, the label is 0.

[0149] In one example, the training method, model structure, and model application of the binary classification model can be found in the description of step S304 above.

[0150] In one example, the multimodal data includes a first data point C containing a math problem and a second data point D that reads "Solve this math problem and tell me the answer." The image description text is: "An image containing a math problem with a quadratic equation, where the math problem is: ***." The similarity parameters between the first data point C and the second data point D include a first parameter of 0.7 and a second parameter of 0.97. The output of the binary classification model is 1, indicating that the first data point C and the second data point D are related data.

[0151] Through the above embodiments, a pre-defined binary classification model can be used to accurately determine the relevance of multimodal data by combining the descriptive features of the first data, the content features of the second data, and the similarity parameters between them. This binary classification model undergoes rigorous supervised training, and its training data covers the features, similarities, and labels of both first and second historical data, ensuring that the model can accurately distinguish between relevant and irrelevant data. By setting a high accuracy threshold and continuously optimizing the model parameters, the prediction accuracy of the classification model can be improved, providing strong support for the correlation analysis of multimodal data.

[0152] S405, if the correlation result indicates that the first data and the second data are related data, determine the user intent based on the combined intent of the first data and the second data.

[0153] In some embodiments of this application, if the correlation result indicates that the first data and the second data are related data, a comprehensive intent can be obtained based on the first data and the second data using a preset multimodal intent recognition model.

[0154] In one example, image description text can be obtained from the image data in the first set of data using a pre-defined visual language model. This image description text and the text data are then input into a multimodal intent recognition model. The multimodal intent recognition model then uses the text data from the second set of data as the query statement for the image description text and outputs the corresponding comprehensive intent. The multimodal intent recognition model can be a pre-defined deep learning-based artificial neural network model, including but not limited to combinations of one or more of the following models: Generative Pre-Trained (GPT) models, Transformer models, Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, etc.

[0155] In one example, the multimodal data includes a first data point C containing a math problem and a second data point D that reads "Solve this math problem and tell me the answer." The image description text is: "An image containing a math problem with a quadratic equation, where the math problem is: ***." The relevance results indicate that the first data point C and the second data point D are relevant data, which can prompt the multimodal intent recognition model to treat the text data as the query statement of the image description text and output the corresponding comprehensive intent. For example, the comprehensive intent could be: "Solve the quadratic equation math problem in image data C and output the answer."

[0156] Through the above embodiments, when the multimodal data is relevant data, the user intent can be determined based on the comprehensive intent of the multimodal data output by the large model, thereby improving the accuracy of user intent recognition.

[0157] The user intent recognition method provided in this application can determine the correlation result between the first data and the second data based on the descriptive features of the first data, the content features of the second data, and the similarity parameter between the first data and the second data using a classification model. If the correlation result indicates that the first data and the second data are related, the combined intent of the first data and the second data can be determined as the user intent. This method can improve the accuracy of user intent recognition in multimodal interaction scenarios by performing content analysis and correlation judgment on the first data and the second data.

[0158] In some embodiments of this application, if the correlation result indicates that the first data and the second data are irrelevant, the user intent can be determined based on the intent corresponding to the text data in the second data. This method is applicable when the priority of text data is higher than that of image data, so the text data can be directly input into a preset intent recognition model to obtain the intent corresponding to the text data as the user intent. In one example, the multimodal data includes a first data C containing a math problem and a second data E "Continue playing music for me". The correlation result indicates that the first data and the second data are irrelevant, and the user intent can be determined based on the second data E.

[0159] In another example of this application, the user's true intent may also include the intent corresponding to the first data. For example, if the correlation result indicates that the first data and the second data are unrelated, the user's intent can be determined based on the intent corresponding to the image data in the first data. This method can be applied to situations where image data has a higher priority than text data, thus allowing image data (or image description text) to be directly input into a preset intent recognition model to obtain the intent corresponding to the image data as the user's intent.

[0160] In one example, if the current application environment is an online shopping platform, users can search for similar or related products by uploading images. In this scenario, the image data uploaded by the user (e.g., a photo of a shoe) better expresses their needs than the provided text data (e.g., "Find me the same style"). Therefore, image data takes precedence over text data because it contains enough information to identify the characteristics of the product (such as style, color, etc.), thus enabling a more accurate inference of the user's intent. Therefore, the platform can directly input the first data (or processed image description text, if available) into a pre-defined intent recognition model to obtain the intent corresponding to the first data as the user's intent.

[0161] Through the above embodiments, when multimodal data is determined to be irrelevant, user intent can be flexibly determined based on data priority. When text data has a higher priority, the text data in the second set of data is directly used to obtain the user intent through the intent recognition model. However, when image data has a higher priority, such as in product search scenarios on online shopping platforms, image data, due to its rich product feature information, more accurately reflects user needs. Therefore, the image data (or processed image description text) in the first set of data can be directly input into the model to identify user intent. This data priority-based selection strategy ensures efficient and accurate understanding and response to user needs in different scenarios.

[0162] In some other embodiments of this application, the user's true intent may also include the user intent corresponding to the first data. For example, if the image data in the first data contains text data, and a clear intent can be directly obtained after parsing the text data, the intent of the text data in the first data can be directly obtained as the user intent.

[0163] In one example, the multimodal data includes a first data F containing a math problem and a text data G that says "I don't know". The first data F is an image containing the text "Please use ** to solve the following math problem". In this case, since the text data in the image data already provides a clear intent, there is no need to perform further parsing on the text data G in the second data; the intent corresponding to the text in the first data F can be directly obtained as the user intent.

[0164] Through the above embodiments, the accuracy and efficiency of intent recognition can be improved in scenarios where the text in the first data is sufficient to clearly express the user's needs.

[0165] In some embodiments of this application, if the second data contains preset keywords, the user intent can be determined based on the comprehensive intent. In one example, the multimodal data includes image data A in the first data and second data H, where the second data H is "What is the answer to the question in the picture above?". By performing preset keyword recognition on the second data H, it is determined that the second data H contains preset keywords such as "above", "picture", and "image". It can be determined that the second data H is the query statement of the first data A, and the comprehensive intent is taken as the user intent.

[0166] Through the above embodiments, multimodal data content can be more effectively linked based on keywords in the second data, thereby accurately understanding and responding to user needs.

[0167] In some embodiments of this application, if it is impossible to provide corresponding query results or invoke corresponding functions based on the user intent determined by the second data, an updated user intent is determined based on the combined intent of the first data and the second data. In one example, if the second data I is "What is the color of this ball?", and it is impossible to provide corresponding query results based on the user intent determined by the second data I, an updated user intent can be determined based on the combined intent of the second data I and the image data J in the received first data. For example, the updated user intent is "What is the color of the ball in image data J?".

[0168] Through the above embodiments, if the text data does not contain preset keywords, the user's intent is determined based on the priority of the text and image data; if the query results or corresponding functions cannot be directly obtained based solely on the text data, the received image data will be combined to comprehensively analyze the intent of both in order to update and accurately determine the user's final needs, thereby providing more accurate services or responses.

[0169] refer to Figure 7 The diagram shown is a flowchart of a user intent recognition method provided in another embodiment of this application. It can receive multimodal data of user data, including image data and text data. It identifies whether the text data contains preset keywords. Preset keywords can include keywords used to indicate image data; for example, preset keywords may include, but are not limited to, "image," "picture," "image," "sheet," etc.

[0170] If the text data includes preset keywords, the user's intent can be determined based on the combined intent of the text and image data. If the text data does not contain preset keywords, the receiving order of the image and text data is determined.

[0171] If text data is received before image data, the user intent is determined based on the text data. If the user intent determined by the text data can be used to provide corresponding query results or invoke corresponding functions, then the corresponding query results or functions are provided based on the user intent. If the user intent cannot be used to provide corresponding query results or invoke corresponding functions based on the text data, then an updated user intent is determined based on the combined intent of the image data and the text data.

[0172] If image data is received before text data, determine whether the image data contains text data. If the text data contained in the image data clearly indicates the user's intent, determine the user's intent based on the text data in the image data. If the image data does not contain text data, or if the text data contained in the image data does not indicate the user's intent, determine whether the image data and text data in the multimodal data are related data. For example, refer to the embodiments in steps S302 to S304 to determine whether the image data and text data are related data.

[0173] If the image and text data in the multimodal data are related, the user intent is determined based on the combined intent of the image and text data. If the image and text data in the multimodal data are not related, the higher-priority data between the image and text data is identified, and the user intent is determined based on the higher-priority data.

[0174] Through the above embodiments, by receiving multimodal data containing images and text, the system first checks whether the text contains preset image indicator keywords. If it does, the user intent is determined by combining both; if not, the system intelligently judges and prioritizes data with clearly defined intent (such as clear text in the first received text or image) based on the data reception order and content relevance to determine the user intent; when a single data type cannot clearly define the intent, the system then combines both or updates the user intent based on data priority, thereby flexibly and accurately meeting the user's query or function call needs.

[0175] refer to Figure 8The diagram shows a flowchart of a user intent recognition method provided in another embodiment of this application. The method involves receiving image data, determining image description text, and text encoding the image description text to obtain an image description embedding. It also involves receiving text data, text encoding the text data to obtain a text content embedding. Based on the image description embedding and the text content embedding, similarity features (e.g., cosine similarity) and perplexity features (e.g., the aforementioned third parameter) are determined. The image description embedding, text content embedding, similarity features, and perplexity features are input into a classification model to obtain a judgment result on whether the image data and text data are related data. If the image data and text data are related data, the user intent is determined based on the combined intent of the image data and text data; if the image data and text data are not related data, the user intent is determined based on the text data.

[0176] Through the above embodiments, multiple features can be concatenated and processed by a binary classification model to accurately determine whether image data and text data are related, thereby improving the accuracy of user intent recognition. Furthermore, the parallel strategy of image description embedding generation, text content embedding generation, and intent recognition model intent recognition is essentially consistent with the efficiency of intent-centric model recognition methods in related technologies, enabling rapid and accurate recognition of user intent in multimodal interactive environments.

[0177] This application also provides an electronic device 100, see reference. Figure 9 As shown, the electronic device 100 can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. The embodiments of this application do not impose special restrictions on the specific type of electronic device 100.

[0178] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, Universal Serial Bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0179] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0180] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0181] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0182] The processor 110 may also include a memory for storing instructions and data. In one embodiment of this application, the memory in the processor 110 is a cache memory. The memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instructions or data again, it can directly retrieve them from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0183] In one embodiment of this application, the processor 110 may include one or more interfaces. These interfaces may include an Inter-integrated Circuit (I2C) interface, an Inter-integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0184] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In one embodiment of this application, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0185] The I2S interface can be used for audio communication. In one embodiment of this application, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to realize communication between the processor 110 and the audio module 170. In one embodiment of this application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to realize the function of answering phone calls through a Bluetooth headset.

[0186] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In one embodiment of this application, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In another embodiment of this application, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0187] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In one embodiment of this application, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In one embodiment of this application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback via Bluetooth headphones.

[0188] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a Camera Serial Interface (CSI) and a Display Serial Interface (DSI). In one embodiment of this application, the processor 110 and the camera 193 communicate via the CSI interface to realize the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the electronic device 100.

[0189] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In one embodiment of this application, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0190] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, MicroUSB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. Furthermore, the interface can be used to connect other electronic devices 100, such as AR devices.

[0191] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0192] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0193] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0194] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0195] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0196] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In one embodiment of this application, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In another embodiment of this application, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0197] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In one embodiment of this application, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules.

[0198] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including Wireless Local Area Networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0199] In one embodiment of this application, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. Wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the Beidou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0200] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor that recognizes user intent and connects to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0201] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light-Emitting Diode (AMOLED), a Flexible Light-Emitting Diode (FLED), a Minied, Microled, Micro-OLED, or a Quantum Dot Light-Emitting Diode (QLED), etc. In one embodiment of this application, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0202] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0203] The ISP is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, etc. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one embodiment of this application, the ISP can be set in the camera 193.

[0204] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment of this application, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0205] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0206] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0207] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, information prompts, speech recognition, and text understanding.

[0208] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0209] Random access memory can include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), double data rate synchronous dynamic random-access memory (DDRSDRAM, for example, fifth-generation DDRSDRAM is generally called DDR5SDRAM), etc.

[0210] Non-volatile memory can include disk storage devices and flash memory.

[0211] Flash memory can be classified according to its operating principle, including NORFLASH, NANDFLASH, 3D NANDFLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multi-media card (eMMC), etc.

[0212] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0213] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0214] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0215] Internal memory 121 or external memory interface 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by processor 110. The one or more computer programs include multiple instructions, which, when executed by processor 110, can implement the screen display detection method executed on electronic device 100 in the above embodiments, so as to realize the screen display detection function of electronic device 100.

[0216] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0217] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In one embodiment of this application, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0218] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0219] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0220] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0221] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0222] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0223] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0224] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0225] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In one embodiment of this application, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100. This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on the electronic device 100, the electronic device 100 performs the aforementioned related method steps to implement the methods in the above embodiments.

[0226] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the methods described in the above embodiments.

[0227] See Figure 10 As shown, this application embodiment also provides a chip system 1200, which is applied to an electronic device 100. The chip system includes one or more processors 1210. The processors 1210 are used to call computer instructions to cause the electronic device 100 to input an image to be processed into the chip system. The chip system parses the multimodal data based on the method provided in this application embodiment and outputs the user intent.

[0228] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the apparatus to perform the methods in the above-described method embodiments.

[0229] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0230] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0231] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0232] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0233] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0234] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0235] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A user intent recognition method, applied to electronic devices, characterized in that, The method includes: Receive multimodal data input by the user, the multimodal data including image data and text data; In the case where the image data is received before the text data, the image data is parsed to obtain descriptive features, and the text data is parsed to obtain content features; Determine the similarity parameter between the image data and the text data; Based on the descriptive features, the content features, and the similarity parameters, a preset classification model is used to obtain the correlation results between the image data and the text data; If the correlation result indicates that the image data and the text data are related, the user intent is determined based on the combined intent of the image data and the text data.

2. The user intent recognition method as described in claim 1, characterized in that, The method further includes: If the text data is received before the image data, and the text data contains preset keywords, the user intent is determined based on the comprehensive intent.

3. The user intent recognition method as described in claim 1, characterized in that, The method further includes: If the text data is received before the image data, and the text data does not contain preset keywords, the user intent is determined based on the text data.

4. The user intent recognition method as described in claim 3, characterized in that, The method further includes: Based on the user intent, provide query results or invoke applications, services or functions related to the user intent; If the user cannot provide the corresponding query results or invoke the relevant application, service, or function based on the user's intent, an updated user intent is determined based on the combined intent of the image data and the text data.

5. The user intent recognition method as described in claim 1, characterized in that, The image data includes images or video frames extracted from videos; the text data includes text information or text information recognized from speech data.

6. The user intent recognition method as described in claim 1, characterized in that, The similarity parameter includes a first parameter, and the method for determining the first parameter includes: Determine the cosine similarity between the descriptive feature and the content feature, and determine the first parameter based on the cosine similarity.

7. The user intent recognition method as described in claim 1, characterized in that, The similarity parameter includes a second parameter, and the method for determining the second parameter includes: The image data is parsed to obtain image description text, and prompt information is generated based on the image description text. The prompt information is used to prompt a preset large model to generate query keywords based on the image description text. Determine the distribution probability of keywords in the text data among the query keywords, and determine the second parameter based on the distribution probability.

8. The user intent recognition method as described in claim 1, characterized in that, The method for obtaining the combined intent of the image data and the text data includes: Based on the image data and the text data, a preset multimodal intent recognition model is used to obtain the comprehensive intent, which represents the intent to use the text data as a query statement corresponding to the image data.

9. A user intent recognition method, applied to electronic devices, characterized in that, The method includes: Receive multimodal data input by the user, wherein the multimodal data includes first data and second data; The first data is parsed to obtain descriptive features, and the second data is parsed to obtain content features; Determine the similarity parameter between the first data and the second data; Based on the descriptive features, the content features, and the similarity parameters, a preset classification model is used to obtain the correlation results between the first data and the second data; If the correlation result indicates that the first data and the second data are related data, the user intent is determined based on the combined intent of the first data and the second data.

10. The user intent recognition method as described in claim 9, characterized in that, The method further includes: If the correlation result indicates that the first data and the second data are unrelated, the user intent is determined based on the intent corresponding to the first data or the intent corresponding to the second data.

11. The user intent recognition method as described in claim 9, characterized in that, The descriptive features obtained by parsing the first data include: Image description text is obtained from the image data in the first data using a preset visual language model; Feature extraction is performed on the image description text to obtain the description features of the first data.

12. The user intent recognition method as described in claim 11, characterized in that, The feature extraction of the image description text includes: text encoding of the image description text.

13. The user intent recognition method as described in claim 9, characterized in that, The content features obtained by parsing the second data include: Feature extraction is performed on the text data in the second data to obtain the content features of the second data.

14. The user intent recognition method as described in claim 9, characterized in that, The similarity parameter includes a first parameter, and the method for determining the first parameter includes: Determine the cosine similarity between the descriptive feature and the content feature, and determine the first parameter based on the cosine similarity.

15. The user intent recognition method as described in claim 9 or 11, characterized in that, The similarity parameter includes a second parameter, and the method for determining the second parameter includes: The first data is parsed to obtain image description text, and prompt information is generated based on the image description text. The prompt information is used to prompt the preset large model to generate query keywords based on the image description text. Determine the distribution probability of the keywords in the second data among the query keywords, and determine the second parameter based on the distribution probability.

16. The user intent recognition method as described in claim 9, characterized in that, The classification model includes a binary classification model, and the training method for the binary classification model includes: Acquire training data, which includes content features of first historical data, descriptive features of second historical data, similarity parameters and labels between the first historical data and the second historical data, and the labels represent whether the first historical data and the second historical data are related data. The binary classification model is trained using the training data based on a supervised training method until the prediction accuracy of the binary classification model is greater than a preset accuracy threshold.

17. The user intent recognition method as described in claim 9, characterized in that, The method for obtaining the combined intent corresponding to the first data and the second data includes: Based on the first data and the second data, the comprehensive intent is obtained using a preset multimodal intent recognition model.

18. An electronic device, characterized in that, The electronic device includes a memory and a processor: The memory is used to store program instructions; The processor is configured to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, cause the electronic device to perform the user intent recognition method as described in any one of claims 1 to 17.

19. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to input multimodal data into the chip system and execute the user intent recognition method as described in any one of claims 1 to 17.

20. A computer storage medium, characterized in that, The computer storage medium stores program instructions that, when executed on the electronic device, cause the processor of the electronic device to perform the user intent recognition method as described in any one of claims 1 to 17.

Images